Addition of `ins_serial_no` requirement of `validate_glatos_receivers` negates use of `hfx_deployments.csv` as test data for read_otn_deployments`

[mhpob]

The addition of ins_serial_no as a required column in validate_glatos_receivers seems to have broken test-read_otn_deployments (commit 5f6b727). Seems to run through glatos_check_col_names, also introduced in 5f6b727.

The test data set, hfx_deployments.csv, does not contain an ins_serial_no column.

[chrisholbrook]

I am not aware of any glatos function that requires ins_serial_no, so perhaps this requirement should be relaxed; but it also seems like a reasonable requirement for an object about devices. @jdpye, is hfx_deployments.csv still an active file structure in OTN?

[jdpye]

Yep, it's sourced from our instrument metadata shortform: https://members.oceantrack.org/data/data-collection/otn-instrument-deployment-short-form.xlsx

So most if not all OTN projects will have a file like this. I don't have the ability to build this at present using our GeoServer, but i could try and build a layer for this in the near future, so we could default to a web-facing example of this data.

[jdpye]

on further review, the existence of columns like off_set in this example means it was likely to be sourced from our longer-form metadata sheets, where we track things like how far up the riser is from the bottom mooring.

I still think we should accept ins_serial_no and agree to looking into whether we can soften its requirement, but this HFX example is even more than our basic deployment metadata requires.

[jdpye]

further-further review - the true source looks to be our geoserver layer otn.stations_receivers. We're looking into if we can add ins_serial_no to that layer. https://members.oceantrack.org/geoserver/otn/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=otn:stations_receivers&outputFormat=csv

[mhpob]

Seems like two separate issues now: a fix of the OTN geoserver to include receiver number and a fix of the hfx_deployments.csv in glatos. The former seems more complicated due to the various permissions involved.

Is it correct, then, to say that GLATOS deployment data must include ins_serial_no, but there is no such requirement for OTN deployment data? If so, would it be faster to bake that distinction into read_otn_deployments by checking if the column exists and adding a dummy column if not before it gets validated rather than restructuring how OTN reports data?

If that's wrong and OTN usually reports a serial number that should be checked, hfx_detections has a collectornumber column that could be used to create a serial number and tack it onto hfx_deployments. The script to do this could be documented in the instdata and reverted/altered if something better comes along.

[jdpye]

Instrument serial number is marked 'required' on our deployment sheets, we usually follow up or pull from manufacturer specifications if we don't get them from the data submitters. So we will be in good shape to require this, just have to express it in the datasets we're pushing out into the world via GeoServer.

https://ocean-tracking-network.github.io/node-manager-training/07_deploy_metadata/index.html

[mhpob]

@jdpye it seems where we left this was to wait for OTN to express in_serial_no on the geoserver. Has this been done?