Skip to content

Deployment notes

Jump to bottom
Sundareswar Pullela edited this page Apr 18, 2025 · 6 revisions

RTX-KG2c dev/ops notes

To start a KG2 Plover instance when ITRB's KG2 Plover is down

Start the kg2cplover.rtx.ai ec2 instance and run the following (should take about 10 minutes):

ssh [email protected]
cd PloverDB/
sudo docker start plovercontainer

This should take about 10 minutes to complete; while you're waiting, you can check the logs with sudo docker logs plovercontainer, or by going to https://kg2cplover.rtx.ai:9990/logs in your browser. You can check which KG2 version Plover is running by going to https://kg2cplover.rtx.ai:9990/code_version in your browser.

If the above sudo docker start command gave some sort of error, instead try doing a rebuild (should take about 1 hour):

screen
cd PloverDB/
bash -x run.sh

When it's ready, the last few lines of the log should look something like this:

2022-03-02 00:25:58,807 INFO: Indexes are fully loaded! Took 5.27 minutes.
WSGI app 0 (mountpoint='') ready in 317 seconds on interpreter 0x pid: 11 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 11)
spawned uWSGI worker 1 (pid: 14, cores: 1)
spawned uWSGI worker 2 (pid: 15, cores: 1)
running "unix_signal:15 gracefully_kill_them_all" (master-start)...

While you're waiting for the above command to finish, point the ARAX code to this Plover instead of the ITRB Plover(s):

  1. Change the necessary KG2 URL(s) under the servers slot in RTX/code/UI/OpenAPI/specifications/export/KG2/1.5.0/openapi.yaml to this Plover endpoint (i.e., https://kg2cplover.rtx.ai:9990).
    1. The maturity level(s) you should edit here depends on which ITRB server is down; usually it's CI, in which case you would change the development and staging maturity level server urls to https://kg2cplover.rtx.ai:9990.
  2. Push that change to master
  3. Roll master out to the relevant ARAX endpoints on arax.ncats.io (can ask Eric Deutsch to do this)
    1. This step is necessary because ARAX pulls the KG2 URL to use directly from its local copy of this file
  4. Ask Eric Deutsch to trigger a refresh of our KG2 SmartAPI registration, so that it pulls the change you made to KG2 openapi.yaml
    1. This makes it so that other reasoning agents will use the Plover you just started (https://kg2cplover.rtx.ai:9990) when querying KG2

At this point, once Plover has finished starting up and loading indexes, ARAX should successfully get back answers from KG2 when answering queries.

You can test this by running the 'Example 1' acetaminophen query in the ARAX UI, and verifying in the KP table that KG2 successfully returned answers. Note that arax.ncats.io is CI/staging maturity, and the other ARAX endpoints (e.g., arax.ncats.io/beta) are dev maturity (all ARAX endpoints query the KG2 endpoint that has the same maturity).

To build Plover from a new KG2 version

(Note: This example is for KG2.10.0, but the steps should be analogous for future KG2 versions.)

Create a new branch in the PloverDB repo for this KG2 version

We'll name ours kg2.10.0c for this example:

git checkout -b kg2.10.0c

Update the default KG2 version in our new branch

The KG2c files that Plover needs are automatically uploaded to kg2webhost.rtx.ai during the KG2c build; all you need to do is tell Plover to use the new versions of those files, with these steps:

  1. In app/config.json, change
    • "nodes_file": "https://kg2webhost.rtx.ai/kg2c-2.10.0-v1.0-nodes.jsonl.gz" to "https://kg2webhost.rtx.ai/kg2c-2.10.1-v1.0-nodes.jsonl.gz", or whatever exactly the new KG2c JSON Lines nodes file is called on kg2webhost.rtx.ai.
    • "edges_file": "https://kg2webhost.rtx.ai/kg2c-2.10.0-v1.0-edges.jsonl.gz" to "https://kg2webhost.rtx.ai/kg2c-2.10.1-v1.0-edges.jsonl.gz", or whatever exactly the new KG2c JSON Lines edges file is called on kg2webhost.rtx.ai.
    • Update the biolink version to match the version of the above kg2c files.
  2. Commit and push this change to your branch
  3. Make any other Plover code changes that this new KG2 version necessitates in your branch (usually only needed if KG2's core schema changed)

Start Plover on a host instance

Then pick an EC2 instance to serve this new Plover from. Generally we use kg2cplover.rtx.ai, but if that instance is already serving a different version of Plover that needs to remain live (i.e., it is being called by one of the RTX-KG2 instances), then you can use kg2cplover3.rtx.ai. Note that you can tell which Plover a given KG2 instance is using by going to its /code_version endpoint (e.g., https://kg2cplover.rtx.ai:9990/code_version or https://kg2cplover3.rtx.ai:9990/code_version.

We usually deploy an updated PloverDB into one of our team's self-hosted PloverDB instances in EC2 first, and then once we have a completely working RTX-KG2 KP and ARAX based on the updated (self-hosted) PloverDB, we will eventually merge the updated PloverDB code into main which will trigger auto-deployment into ITRB CI.

Start the Plover EC2 instance (this example uses kg2cplover.rtx.ai) and run the following (with your branch name/version number subbed in):

ssh [email protected]
cd PloverDB/
git fetch
git checkout kg2.10.0c
screen
bash -x run.sh

The build should take around 1 hour to finish.

After it's done, verify the new Plover service is working by running the test suite against it. From your own machine (assuming you have cloned the PloverDB repo and done pip install -r requirements.txt):

cd PloverDB/
pytest -v test/test_kg2c.py --endpoint https://kg2cplover.rtx.ai:9990

(NOTE: If you loaded Plover onto the kg2cplover3.rtx.ai instance, use this endpoint URL instead: https://kg2cplover3.rtx.ai:9990)

Note that sometimes tests need to be updated due to changes in the new KG2 version, though the majority of tests should pass. For any failing tests, ensure they're failing due to expected topological changes in the new KG2 version; if so, tweak them to get them passing again (via adjusting pinned curies, predicates, or whatever makes sense).

When we're ready for the ITRB CI Plover instance to be running this new KG2 version, merge your branch into main. This should automatically deploy to the ITRB CI Plover (allow about an hour for it to rebuild). Ping Kanna and/or Pouyan in Slack to update the ITRB Test and Prod Plovers.

About kg2webhost.rtx.ai

The kg2webhost.rtx.ai system is a t2.micro instance (Ubuntu 20.04 AMI) running in the us-east-1 AWS region (Virginia), with 200 GiB of EBS storage. Currently, Amy, Sundar, Adil, and Steve have RSA public keys installed to be able to ssh in. It has nginx installed and the DocumentRoot directory is /var/www/kg2webhost (owned by user ubuntu). AWS CLI is installed and configured for user ubuntu in the directory /home/ubuntu/venv/bin/aws (note, that AWS CLI installation is configured to have default region of us-west-2, since that is where our main KG2 S3 bucket is located). Nginx is configured with HTTPS in this instance, with the SSL certificate being managed by certbot. The crontab for renewing the cert is located in /etc/cron.d/certbot. Currently, this nginx webserver is only used for hosting the KG2c JSON Lines files so that the PloverDB plover.py module can curl in the file at app start-up.

Multiomics dev/ops notes

The Multiomics KPs are hosted in a single PloverDB application on the multiomics.rtx.ai instance (for 'development' maturity), using the multiomics branch in the PloverDB repo. Commits to the multiomics branch are automatically deployed to the ITRB CI Multiomics Plover.

To restart the Multiomics Plover

If you want to restart the Multiomics Plover service, say after the multiomics.rtx.ai EC2 instance has been rebooted, log into the host instance with:

ssh [email protected]

And then run the following commands:

cd PloverDB
git checkout multiomics
git pull origin multiomics
bash -x run.sh

This will build and start the Plover service, which should be ready to use within about five minutes.

You can verify that the service is running by going to https://multiomics.rtx.ai:9990/code_version and https://multiomics.rtx.ai:9990/logs in your browser.

On multiomics.rtx.ai, the built-in Plover remote rebuild server should always be running as well. You can start that with these commands:

cd PloverDB
screen
pyenv activate plover
fastapi run rebuild_main.py

To test the Multiomics Plover

We have a very small pytest suite you can use to test the Multiomics Plover service. From your own machine (assuming you have cloned the PloverDB repo and done pip install -r requirements.txt):

cd PloverDB/test/
git checkout multiomics
git pull origin multiomics
pytest -v --endpoint https://multiomics.rtx.ai:9990

Branching strategy and ITRB deployments

Currently the RTX-KG2 Plover is built from (or in a branch off of) main, and the Multiomics Plover is built from the multiomics branch.

Importantly, for both the KG2 Plover and the Multiomics Plover, ITRB is set up so that the CI instance tracks the relevant branch (i.e., main and multiomics, respectively); to deploy to Test/Prod, we request a deployment from ITRB (in slack), who will then roll out the current CI build to the Test/Prod ITRB instances (meaning, Plover does not have separate branches for test/prod deployments).

When doing dev work on PloverDB, I always make changes in the dev branch, and then merge dev into main when I'm ready to deploy those changes to the RTX-KG2 Plover, and merge dev into multiomics when I'm ready to deploy those changes to the Multiomics Plover. Always be sure to do full test builds (using bash -x run.sh) and run pytests prior to pushing those merges, as both of those branches auto-deploy to ITRB CI instances.

Note that, ideally, KG2 would auto-deploy from a branch off of main called kg2c (instead of main, and then dev work could happen directly in main), but with ITRB's absence in the latest phase of Translator we haven't been able to configure this. So the KG2 ITRB CI Plover still currently tracks main.

Note that there are a few files that exist in main that do not exist in the multiomics branch and vice versa: If you do dev work in the dev branch and only merge it as I outlined above, you shouldn't run into merge conflicts with these files. But if you use another branching strategy you may run into merge conflicts; basically just be sure to keep the KG2 and Multiomics JSON config files separate (i.e., config.json is the KG2 config file and only belongs in main (and dev, which was created off of main); the multiomics config files (config_ctkp.json, config_dakp.json, etc.) only belong in the multiomics branch. The same applies to the pytest files for each Plover - /test/test_kg2c.py belongs only in main and the Multiomics test files (/test/test_ctkp.py, /test/test_dakp.py, etc.) only belong in the multiomics branch. Again, if you follow the branching strategy laid out above, you shouldn't run into merge conflicts with these (they've already been sorted out).

Dev work flow

When doing dev work I run PloverDB in Docker, using its run.sh script (i.e., bash -x run.sh). I will often put test JSON Lines nodes/edges files into PloverDB/app/ locally and then list those files' names under the nodes_file and edges_file slots in the config file; this spares Plover from downloading graph files, which saves some time, and also lets you use whatever test files you'd like.

See the README for some notes on debugging.

Clone this wiki locally