How records are matched, and where the comparison is limited.
Ground Truth brings four records to the same mutation coordinate: protein structure and confidence, an AlphaMissense result, a clinical classification, and an independent laboratory measurement where matched data is available. Five proteins anchor the initial study, while the explorer can analyze other supported genes on demand.
AlphaMissense was calibrated on ClinVar. So grading it against ClinVar partly grades it on its own answer key. The test and the calibration data overlap. Independent lab experiments, called deep mutational scanning, provide a separate comparison. For the two proteins below, the laboratory comparison is lower:
The difference is about 0.05 to 0.12 for these proteins. It is real, but modest.
Data sources
Each public data source is reconciled to the UniProt canonical sequence so the records line up residue-for-residue.
| Source | Role |
|---|---|
| UniProt | canonical sequence + features |
| AlphaFold DB (v6) | structure, pLDDT, AlphaMissense scores |
| ClinVar (NCBI E-utilities) | clinical verdict + review status (ground truth) |
| CFTR2 | expert clinical panel for CFTR; supplies the benign variants ClinVar lacks |
| MAVEdb (DMS) | independent laboratory experiments; the separate comparison |
The pipeline
- 01Resolve. gene → canonical UniProt accession (one coordinate system for all layers).
- 02Pull. AlphaFold model + per-residue pLDDT; AlphaMissense scores for every substitution; ClinVar verdicts via NCBI E-utilities.
- 03Match. each clinical variant to the AI score at the exact (position, amino-acid change).
- 04Check. a three-way wild-type assertion: clinical = canonical sequence = AM table. Anything that disagrees is dropped and logged, never guessed.
- 05Resolve verdict. expert panel > ClinVar ≥2★, collapsed to one binary verdict; conflicts excluded.
- 06Compare & test. confusion matrix, balanced metrics, and the structural error-enrichment test.
Quality control
- ·Missense only: single-residue substitutions, which are the variants AlphaMissense scores.
- ·Reviewed clinical classifications: ClinVar review status ≥ 2★, with conflicts excluded and expert-panel calls treated as authoritative.
- ·Both classes required: a protein needs ≥5 confirmed pathogenic and benign variants, or its metrics are flagged unreliable. CFTR was supplemented by CFTR2. PAH still fails this rule.
- ·Wild-type consistency: every dropped variant is logged with its reason.
The numbers, defined
- ·Balanced accuracy / MCC: reported as the headline because ClinVar skews pathogenic, so raw accuracy flatters the model.
- ·ROC-AUC: threshold-free; how well the AI’s continuous score ranks pathogenic above benign.
- ·Fisher’s exact + Mantel-Haenszel OR: tests whether errors concentrate in low-pLDDT residues while controlling for class composition.
Per-protein bookkeeping
| Protein | kept | dropped | path / benign | reliable? |
|---|---|---|---|---|
| TP53 | 270 | 99 | 196 / 74 | yes |
| BRCA1 | 332 | 249 | 152 / 180 | yes |
| CFTR | 204 | 103 | 184 / 20 | yes |
| HBB | 44 | 83 | 38 / 6 | yes |
| PAH | 414 | 78 | 413 / 1 | no |
Limitations (read these)
AlphaMissense was calibrated on ClinVar, so grading it against ClinVar partly grades it on its own answer key. We added independent lab experiments (DMS) as the honest check: vs experiment AlphaMissense scores ~0.87 (TP53) and ~0.91 (BRCA1), below the 0.96 to 0.99 it scores against ClinVar. The premium is real but modest for well-measured proteins.
ClinVar skews pathogenic; per-protein error counts are small, so per-protein enrichment tests are underpowered. Balanced metrics and the pooled, class-stratified read are primary.
BRCA1’s number depends heavily on which functional assay you trust. A noisy one gave 0.65, while a clean depletion assay 0.91; we use the clean one and avoid ACMG-derived scoresets (which fold in clinical priors). PAH’s functional assay barely correlates with AlphaMissense and is treated as inconclusive, not a verdict; PAH’s clinical-benign gap is also unresolved (PAHvdb is scrape-only).
Pick any variant in the protein viewer and follow its ClinVar link. The TP53 hotspot R175H and the HBB sickle variant E7V are useful records to inspect first.