Improvement Feedback Summary from UAB Team - Meeting on September 17

[chunyuma]

From the UAB meeting:

1. We should be sure to include qualifiers
2. For TN's, perhaps consider asking the LLM to also say if the abstract at least mentioned the source and target nodes of the triple
3. Consider asking the LLM to check if the source/target nodes, but the predicate is opposite

They might have a handful of known TN PMID/triple pairs
Andy wouldn't mind going through a hundred or so candidate TN PMID/triple pairs if presented in the form of: (abstract, SemMedDB associated sentence, triple)

[chunyuma]

@dkoslicki, I have a few questions below from the UAB feedback and would appreciate your thoughts.

1. I noticed that Kaiwen was doing something similar to what we did although we are not exactly the same. Should we keep working on this by ourselves or collaborating with UAB team?
2. Kaiwen mentioned a good idea today. Instead of providing a confidence generated from LLM, should we also let AI categorize the response to something like “strongly supported”, “associated” or “not supported”?

[dkoslicki]

Collaborating with the UAB team is good, but they have concerns since their Kaiwen's work on this is for a different project, so they're unsure if we can directly use it. But we can certainly talk with them, learn the best practices, and implement those.

I too like the multi-category response; gives the LLM more options when uncertainty is high. Please do follow up with her via Slack. I recall the categories being more like: "direct evidence for support", "opposite assertion", "related, but not directly", "not supported", and "I don't know", or something like that. It would be good to hear more directly from her