Action Items

[dchiu911]

• Can you please list the OTTA IDs used in the paper along with their final pathology and share them with Martin Koebel.
• Can you please add predictive entropy to the algorithms to encope the algorithm’s uncertainty
• Can you please update the app to include a prediction of SPOT, and new PROTYPE that does Histotype and Molecular Subtype?
• On that note, Susan mentioned that there is an issue with NOTCH3 not being recognized, can you please have a look?
• Can you predict when some genes are missing? Susan says they were able to run SPOT with missing genes
• Susan is sending over genes for LGSC to have a look at whether they can be helpful to include.

[dchiu911]

Please see Figure 4.13 for predictive entropy of top 3 workflows in confirmation set.

In my opinion, I don't think entropy is an effective metric to compare across methods, but is fine for comparing gene sets used:

• Sequential: since the predictions all come from binary classification models, the points all lie along the deterministic relationship between predicted probability and predictive entropy in the case of 2 classes. This is why we don't see any jitter in the points. We do not aggregate probabilities across the sequence since the probabilities are not conditional on previous steps, and take only the probabilities belonging to the selected class (e.g. if predicted class is "ENOC", we take the probabilities from the 3rd algorithm in the sequence to calculate entropy). This ensures our inputs still sum to a probability of 1 representing all possibilities.
• SMOTE-Random Forest: entropy is more effective here for comparing between the full set vs. optimal set of genes used.
• Two-Step: the first step is binary and the second step is multinomial, so that is why the HGSC predictions all fall along the binary relationship, while the other 4 classes are subject to variability among 4 predicted probabilities

The interpretation of entropy is also not consistent across methods. The sequential method will tend to have lower entropy than the SMOTE-rf because it has only 2 vs. 5 possible classes. A random chance binary model would have theoretical maximum entropy of 1 (base log 2) while a random chance 5-class model would have a maximum of ~2.32. In other words, if these two different methods have the same entropy value it doesn't mean they have the same level of uncertainty.