Dataloader move to universal from gpii-ops/gpii-dataloader

[stepanstipl]

This PR moves dataloader from original gpii-ops/gpii-dataloader repo into universal.

Main reasons behind this:

• Get rid of the duplication of Dockerfile and necessity to maintain extra image and pipeline
• Get rid of cloning the repo and installing npm modules when the container is executed (this would be slow, is prone to errors and would raise question of versioning)
• Data loader will always have the new universal code (as it will be running in the universal image)
• Data loader will get triggered every time there's a change to universal (as the universal image will get updated)

Most of the code was in universal already, and gpii-ops/gpii-dataloader contained basically only shell wrapper and the docker image. Work on new dataloader (GPII-3138 and #626) requires changes to the existing dataloader wrapper and therefore presented a good opportunity for the move.

This is a continuation of gpii-ops/gpii-dataloader#6 PR.

Main changes:

• Import deleteAndLoadSnapsets.sh and DataLoader.md
• Add dataLoader dependencies to docker image (jq and curl)
• Update data loader script to reflect new location in universal
• Minor formatting changes
• Add DB connectivity check to dataloader
• Update vagrantCloudBasedContainers.sh to work with new dl location
• Update DataLoader docs to reflect current state

I have tested this locally and plan to create corresponding PR and test for gpii-infra changes required.

[gpii-bot]

Could one of the admins verify that these changes are reasonable to test? If so, please reply with "ok to test".

[cindyli]

ok to test

[gpii-bot]

CI job failed: https://ci.gpii.net/job/universal-tests/1212/

[gpii-bot]

CI job passed: https://ci.gpii.net/job/universal-tests/1214/

[mrtyler]

LGTM

[mrtyler]

Old text was correct.

[stepanstipl]

Thanks for catching this, not sure how it got in and I didn't notice, reverted back to original text.

[gpii-bot]

CI job passed: https://ci.gpii.net/job/universal-tests/1226/

[gpii-bot]

CI job failed: https://ci.gpii.net/job/universal-tests/1229/

[stepanstipl]

I'm closing this in favor of #692 (via klown#3) as we agreed with @klown

[gpii-bot]

CI job passed: https://ci.gpii.net/job/universal-tests/1231/