Upgrade to Biolink 2.1 (KG2.7.1) - due Aug. 11

[amykglen]

associated code changes should go in the kg2integration branch

first update NodeSynonymizer:

• change SRI NodeNormalizer URL to Biolink 2.1 endpoint
• specify version=2.1.0 in Biolink Lookup tool URL
• adjust NodeSynonymizer conflations
• adjust any other mentions of ChemicalSubstance
• try doing a synonymizer build

once we have what seems to be a good synonymizer:

• update conflations hard-coded in KG2c build (record_kg2c_meta_info.py)
• build a new KG2c (put it at http://kg2-7-1c.rtx.ai:7474/browser/) @acevedol
• load it into plover (available at http://kg2-7-1cplover.rtx.ai:9990)

rebuild or edit and put copies in /translator/data/orangeboard/databases/KG2.7.1 on arax.ncats.io:

• configv2.json (should point to the new KG2c/plover)
  * note: saved this as config_local.json, since we want it to be used over configv2.json during testing
• KG2c meta knowledge graph
• NodeSynonymizer
• KG2c sqlite
• NGD database
• COHD database @chunyuma
• refreshed DTD @chunyuma
• DTD model @chunyuma
• DTD database @chunyuma
• 'slim' databases (used for Travis) @chunyuma / @finnagin

note: As databases are rebuilt, the new copy of config_local.json will need to be updated to point to their new paths.

update ARAX codebase:

• make updates to the ARAX codebase:
  • change usages of biolink:ChemicalSubstance to biolink:ChemicalEntity (lots of tests use this)
    • Expand/Resultify @amykglen
    • CHP/DTD/COHD Expand @chunyuma
    • Overlay @finnagin ?
    • Other?
  • update Expand's hard-coded conflations
  • specify version 2.1.0 in every place Expand grabs the Biolink model
• test everything together (entire ARAX pytest suite should pass when using the new config_local.json - must locally set force_local=True in ARAX_expander.py to avoid using the old KG2 API)

other things:

• update the test triples that go in some NCATS repo @finnagin
• update Biolink version (2.1.0) and KG2 version (2.7.1) in openapi yaml @edeutsch ?
• update SmartAPI registration for ARAX
• after main roll-out is complete, rename config_local.json to config_local.json_FROZEN_DO-NOT-EDIT-FURTHER (any remaining edits to the config file, such as when the DTD build is complete, should be made directly to the master configv2.json on araxconfig.rtx.ai)

[amykglen]

well the synonymizer build appeared to go well - artifacts are uploaded to arax.ncats.io at /data/orangeboard/databases/KG2.7.1/synonymizer.

only 56 problems listed in Problems.tsv (down from 76 for KG2.6.7.1).

going to do some spot-checking tomorrow to make sure things look ok.

[edeutsch]

terrific!

[amykglen]

things seem good in the synonymizer based on some spot-checking... although wondering about one thing - is it normal for the SRI_normalizer_category and other SRI fields here to be null?

python3 node_synonymizer.py --lookup CHEMBL.COMPOUND:CHEMBL112

...
    "id": {
      "SRI_normalizer_category": null,
      "SRI_normalizer_curie": null,
      "SRI_normalizer_name": null,
      "category": "biolink:SmallMolecule",
      "identifier": "CHEMBL.COMPOUND:CHEMBL112",
      "name": "ACETAMINOPHEN"
    },
...

[edeutsch]

It could happen if the concept is not known to the SRI Node Normalizer, but that is not the case here. Looks like a bug. What happens if you run this:

python3 sri_node_normalizer.py -c CHEMBL.COMPOUND:CHEMBL112

in the location where the SRI NN and NodeSyn was built?

[amykglen]

it returns this:

==========================================================
Native SRI Node Normalizer results:
{
  "CHEMBL.COMPOUND:CHEMBL112": {
    "equivalent_identifiers": [
      {
        "identifier": "PUBCHEM.COMPOUND:1983",
        "label": "Acetaminophen"
      },
      {
        "identifier": "CHEMBL.COMPOUND:CHEMBL112",
        "label": "ACETAMINOPHEN"
      },
      {
        "identifier": "UNII:362O9ITL9D",
        "label": "ACETAMINOPHEN"
      },
      {
        "identifier": "CHEBI:46195",
        "label": "paracetamol"
      },
      {
        "identifier": "DRUGBANK:DB00316"
      },
      {
        "identifier": "MESH:D000082",
        "label": "Acetaminophen"
      },
      {
        "identifier": "CAS:103-90-2"
      },
      {
        "identifier": "CAS:360769-21-7"
      },
      {
        "identifier": "DrugCentral:52",
        "label": "paracetamol"
      },
      {
        "identifier": "GTOPDB:5239",
        "label": "paracetamol"
      },
      {
        "identifier": "HMDB:HMDB0001859",
        "label": "Acetaminophen"
      },
      {
        "identifier": "KEGG.COMPOUND:C06804",
        "label": "Acetaminophen"
      },
      {
        "identifier": "INCHIKEY:RZVAJINKPMORJF-UHFFFAOYSA-N"
      }
    ],
    "id": {
      "identifier": "PUBCHEM.COMPOUND:1983",
      "label": "Acetaminophen"
    },
    "type": [
      "biolink:SmallMolecule",
      "biolink:MolecularEntity",
      "biolink:ChemicalEntity",
      "biolink:PhysicalEssence",
      "biolink:NamedThing",
      "biolink:Entity",
      "biolink:PhysicalEssenceOrOccurrent"
    ]
  }
}
==========================================================
Local more compact and useful formatting:
{
  "curie": "CHEMBL.COMPOUND:CHEMBL112",
  "equivalent_identifiers": [
    {
      "identifier": "PUBCHEM.COMPOUND:1983",
      "label": "Acetaminophen"
    },
    {
      "identifier": "CHEMBL.COMPOUND:CHEMBL112",
      "label": "ACETAMINOPHEN"
    },
    {
      "identifier": "UNII:362O9ITL9D",
      "label": "ACETAMINOPHEN"
    },
    {
      "identifier": "CHEBI:46195",
      "label": "paracetamol"
    },
    {
      "identifier": "DRUGBANK:DB00316"
    },
    {
      "identifier": "MESH:D000082",
      "label": "Acetaminophen"
    },
    {
      "identifier": "CAS:103-90-2"
    },
    {
      "identifier": "CAS:360769-21-7"
    },
    {
      "identifier": "DrugCentral:52",
      "label": "paracetamol"
    },
    {
      "identifier": "GTOPDB:5239",
      "label": "paracetamol"
    },
    {
      "identifier": "HMDB:HMDB0001859",
      "label": "Acetaminophen"
    },
    {
      "identifier": "KEGG.COMPOUND:C06804",
      "label": "Acetaminophen"
    },
    {
      "identifier": "INCHIKEY:RZVAJINKPMORJF-UHFFFAOYSA-N"
    }
  ],
  "equivalent_names": [
    "Acetaminophen",
    "ACETAMINOPHEN",
    "ACETAMINOPHEN",
    "paracetamol",
    "Acetaminophen",
    "paracetamol",
    "paracetamol",
    "Acetaminophen",
    "Acetaminophen"
  ],
  "preferred_curie": "PUBCHEM.COMPOUND:1983",
  "preferred_curie_name": "Acetaminophen",
  "status": "OK",
  "type": "biolink:SmallMolecule"
}

[edeutsch]

hmm, okay, definitely seems like a bug. do you use those fields for anything?

[amykglen]

no, I don't use them. so unless it's indicative of a larger issue, not a problem for the KG2c build.