Model Performance Evaluation
4,795 drugs · 824,249 DDI pairs · DrugBank v5.1
Why cold-start evaluation is what matters here
The GNN's real-world purpose is to predict interactions for drugs with no documented
DrugBank pairs — newly approved drugs, rare compounds, or combinations simply never
studied together. In a standard 80/20 edge split every test drug still appears in the training
graph, so graph heuristics exploit dense connectivity (average degree ~344) and score ~0.97 AUC
— not because they generalise, but because the graph is so dense that almost any two test drugs
share common neighbours.In cold-start evaluation, 10 % of drugs are held out entirely — they have zero training edges. Common Neighbours, Adamic-Adar and Jaccard score exactly 0 for every cold pair (AUC = 0.50, pure chance — analytically guaranteed). Logistic Regression survives because it uses node features (CYP450, ATC codes, physicochemical properties) that exist independently of graph topology. The GNN's advantage is to combine those features with partial graph context from nearby warm drugs — AUROC 0.9175 vs LR's 0.8974.
Cold-Start Evaluation — The Real Benchmark
10 % of drugs held out completely (zero training edges) · simulates newly approved drugs entering the formulary
Held-out drugs: 284 / 2,847 (10 %)
Cold pairs: 158,642 pos + 158,642 neg
Seed: 42
Pos:neg: 1:1
| Model | Category | Cold-Start AUC-ROC | Performance | Avg Precision |
|---|---|---|---|---|
| Graph Heuristics (all) Common Neighbors · Adamic-Adar · Jaccard |
graph heuristic | 0.5000 (collapses) | 0.5000 | |
| Logistic Regression node features only (CYP450, ATC, physicochemical) |
non-graph ML | 0.8974 | 0.9032 | |
| GNN — HeteroGraphSAGE + NCN features + graph context · this model |
GNN | 0.9175 | 0.8824 |
Graph heuristics evaluate to exactly 0 for cold pairs — cold drugs have no training neighbours by definition, so Common Neighbors = Adamic-Adar = Jaccard = 0 for every pair. AUC = 0.50 is analytically guaranteed.
GNN advantage over LR: AUROC +0.0201 · AUPR −0.0208 (the AUPR gap reflects the difficulty of ranking novel interactions without prior graph context). GNN wins on AUROC, which is the primary metric for ranking novel pairs.
GNN advantage over LR: AUROC +0.0201 · AUPR −0.0208 (the AUPR gap reflects the difficulty of ranking novel interactions without prior graph context). GNN wins on AUROC, which is the primary metric for ranking novel pairs.
Warm evaluation — for reference only
Standard 80/20 Edge Split (Transductive)
All test drugs appear in the training graph · 80 % train / 20 % test · 1:1 pos:neg ·
20 % masked positives (nnPU) removed before split
⚠ Context only — high heuristic scores reflect graph density (avg degree ~344), not generalisation
| Model | Category | Warm AUC-ROC | Bar | Avg Precision |
|---|---|---|---|---|
| Feature Cosine Similarity | no-graph baseline | 0.6041 | 0.5952 | |
| Degree Product | graph heuristic | 0.9534 | 0.9383 | |
| Logistic Regression | non-graph ML | 0.9570 | 0.9474 | |
| Common Neighbors | graph heuristic | 0.9738 | 0.9648 | |
| Adamic-Adar | graph heuristic | 0.9748 | 0.9660 | |
| Jaccard | graph heuristic | 0.9763 | 0.9669 | |
| Homo GraphSAGE V1 | GNN | 0.9615 | 0.9450 | |
| Hetero GraphSAGE V1 no CN pooling |
GNN | 0.9726 | 0.9586 | |
| Hetero GraphSAGE + NCN ★ this model |
GNN | 0.9738 | 0.9589 |
Graph heuristics score ≥0.97 here because DrugBank is extremely dense (avg degree ~344).
Topology-based methods are competitive on warm eval because nearly every drug pair shares
common neighbours — this is a dataset property, not generalisation.
Feature Cosine (no graph, raw drug features only) scores 0.60 — confirming the graph structure
is what drives heuristic performance, not the features alone.
GNN model evolution
Architecture Progression — Warm AUROC / AUPR
Each version adds one structural improvement over the previous
| Version | Key addition | Warm AUROC | Warm AUPR | Δ AUROC |
|---|---|---|---|---|
| Homo GraphSAGE V1 | Homogeneous DDI graph only | 0.9615 | 0.9450 | — |
| Hetero GraphSAGE V1 | + drug→protein edges (hetero graph) | 0.9726 | 0.9586 | +0.0111 |
| Hetero + NCN (ours) ★ | + CN pooling of shared DDI neighbours & shared protein targets | 0.9738 | 0.9589 | +0.0012 |
The heterogeneous graph (drug-protein edges) adds +0.0111 AUROC over the homogeneous baseline.
CN pooling (NCN decoder) adds a further +0.0012 — the incremental gain is modest on warm eval
but the decoder provides richer mechanistic context for cold-start pairs where common neighbours
from shared protein targets still exist.
Prior work comparison
Published Benchmarks vs This Work
Note: datasets and evaluation protocols differ — direct numerical comparison is indicative only
| Model | Dataset scale | AUROC | AUPR | Notes |
|---|---|---|---|---|
| LLM-DDI | ~4,000 drugs | 0.9571 | 0.7346 | LLM text encoding; low AUPR |
| ACDGNN | ~500–1,500 drugs | 0.9835 | 0.9881 | Confirmed negatives; smaller, denser subset |
| Ours (Hetero + NCN) ★ | 4,795 drugs · PU learning | 0.9738 | 0.9589 | No confirmed negatives; largest scale |
ACDGNN operates on a smaller, confirmed-negative subset (~500–1,500 drugs) which makes
the task easier than our setting. LLM-DDI has very low AUPR (0.7346) — we exceed it by
+0.2243. Our model operates at full scale (4,795 drugs) under PU learning (no confirmed negatives), which
is structurally harder. Feature Cosine similarity alone scores 0.60 AUROC — confirming the +0.37 AUROC
contribution of graph structure.
Error analysis — threshold 0.43
Confusion Matrix + Degree-Based Error Analysis
Test set · 131,944 pairs (65,972 pos + 65,972 neg) · threshold 0.43 · warm evaluation
Confusion Matrix
55,406
True Positives
3,055
False Positives
10,566
False Negatives
62,917
True Negatives
Precision: 0.9477
Recall: 0.8398
Avg Combined Degree by Error Type
| Error type | Avg degree | Relative |
|---|---|---|
| True Positives | 2,292 | |
| False Negatives | 2,168 | |
| False Positives | 1,851 |
False Negatives have slightly lower avg degree than True Positives (2,168 vs 2,292).
High-degree drugs are also missed, indicating the model struggles with specific interaction patterns
rather than purely data-sparse drugs.
False Positives have the lowest avg degree (1,851) — the model over-predicts for moderately connected drugs that share structural or topological similarity without documented DDIs. This suggests pharmacological complexity (multiple CYP450 targets, transporters) is harder to predict than data sparsity. The threshold 0.43 was calibrated on a held-out sample; clinical deployment would require prospective recalibration.
False Positives have the lowest avg degree (1,851) — the model over-predicts for moderately connected drugs that share structural or topological similarity without documented DDIs. This suggests pharmacological complexity (multiple CYP450 targets, transporters) is harder to predict than data sparsity. The threshold 0.43 was calibrated on a held-out sample; clinical deployment would require prospective recalibration.
Evaluation Notes
- Cold-start split: 284 / 2,847 drugs (10 %) selected at the drug level (not edge level). All edges involving cold drugs removed from training. 158,642 positive pairs + 158,642 sampled negatives.
- Warm split: PyG
RandomLinkSplit— 80 % train / 10 % val / 10 % test edges. 20 % of DDI edges masked before split (nnPU masked positives). 1:1 positive:negative ratio. - nnPU prior: π = 824,249 / (4795 × 4794 / 2) ≈ 0.0717. Logistic regression trained on up to 200,000 subsampled pairs.
- GNN training: 500 epochs maximum · early stopping patience 30 · best val AUROC 0.9738 reached at convergence · 1,532,353 parameters total.
- Data: DrugBank Full Database v5.1 · 4,795 FDA-approved drugs · 824,249 undirected DDI pairs · 2,708 human proteins · 21,574 drug→protein edges.