Circularity in Variant Prediction

Zetobit · Bioinformatics Insight Series Prediction & Evidence

Bioinformatics Insight Series

Circularity in Variant Prediction

Why a Reported Accuracy Partly Measures Memorization of the Answer Key

Kanna Nandakumar, PhD · Zetobit, LLC

A missense variant-effect predictor arrives with a number attached: an AUC around 0.9, benchmarked against a large set of pathogenic and benign variants. The number is computed correctly. The question is what it is a number about — and for this class of tool, a meaningful share of it is a measurement of overlap between the benchmark and the training data rather than of predictive skill.

This is a different problem from the one covered in the splice-predictor piece. That one concerned what a score means once you have it: a SpliceAI delta is the probability that splicing is altered, not a statement about the resulting transcript, and the threshold at which it becomes evidence is not a property of the tool. This piece is one layer up. It is about the evaluation — whether the reported performance was ever a test of generalisation at all.

Two kinds of leakage, named in 2015

The canonical treatment is a 2015 study that evaluated ten prediction tools across five variant datasets and identified two distinct forms of what the authors called circularity: the phenomenon that predictors are evaluated on variants or proteins used to train them¹.

Type 1 is ordinary data leakage — the same variant in both the training and evaluation sets. It is a familiar machine-learning failure with a familiar remedy, and it is the one everybody thinks of.

Type 2 is the interesting one, and it has no equivalent in most machine-learning settings. It arises because variant databases label genes almost uniformly: in practice, all the recorded variants in a given gene tend to carry the same label. A model that has seen any variant from a gene can predict the rest of that gene correctly without knowing anything about the substitution in front of it.

The magnitude is the part that startles. In one independent evaluation dataset the authors constructed, more than 98% of proteins — 4,425 of 4,490 — contained variants of a single class only, either all pathogenic or all neutral. Nearly 95% of variants sat in those single-class proteins, and just 0.7% of all variants were in proteins with a roughly balanced mix of pathogenic and neutral entries¹.

On a benchmark where 95% of variants sit in genes that are entirely pathogenic or entirely benign, guessing the gene's majority label is a very strong strategy — and it requires knowing nothing about the variant.

The authors tested exactly that. They built a “protein majority vote” predictor that scores a variant purely by the pathogenic-to-neutral ratio of other variants in the same protein, ignoring the substitution entirely. It systematically outperformed FatHMM-W, the best-performing real tool in that comparison¹.

That result is the whole argument in one experiment. A method that is definitionally incapable of distinguishing a pathogenic variant from a benign one in the same gene — and incapable of saying anything about a gene not yet in the database — beat the field on the standard benchmark.

Where the inflation shows up

The 2015 study is a decade old, and the tools it evaluated have largely been superseded. What has not been superseded is the structure of the problem, so it is worth being precise about how the inflation manifested.

On the two benchmarks that overlapped training data, the four best-performing methods were all wholly or partly trained on those benchmarks. Several tools that beat CADD and SIFT on those datasets did not do so on the independent ones — the signature of Type 1 circularity¹.

FatHMM-W topped four of the five datasets, including the independent ones where Type 1 leakage had been removed. Its weighting scheme upweights substitutions in proteins and domains that already contain a high fraction of pathogenic variants — which is Type 2 circularity implemented as a feature. A plain logistic regression on those two weighting terms alone reproduced roughly its full performance¹. And when performance was broken out by protein composition, FatHMM-W was excellent on single-class proteins and lost most of its advantage on mixed ones, where it was outperformed by nearly every other tool¹.

Mixed proteins are, of course, the clinical situation. A gene arrives with some variants known pathogenic and some known benign, and the question is which category a new one falls into.

Figure 1. The three leakage paths. The first two are the named Type 1 and Type 2; the third was anticipated in the same 2015 paper and has since become concrete through the ACMG/AMP computational-evidence criteria.

Why the labels are shaped this way

It would be easy to read the single-class-gene statistic as sloppy curation. It is not. It is a faithful record of how the field works.

A gene enters a disease database when someone finds a pathogenic variant in it. That discovery draws further sequencing of the same gene in the same patient population, which yields further pathogenic variants. Benign variants, meanwhile, arrive from a different pipeline entirely — population frequency catalogues, where a variant is called benign largely because it is common. The two label classes are collected by different processes from different populations, and they land in different genes¹.

So the gene identity carries real signal about the label. It just isn't the signal anyone wants a variant-effect predictor to be using, because it cannot generalise to a gene the database has not yet catalogued — which is precisely the case where a clinician needs help.

The loop that closes: predictions become labels

The 2015 paper closed with a warning about a third form of circularity it could see coming: variants newly entered into databases may be annotated using existing prediction tools, so those tools will appear to perform well on “new” data while in fact only recovering labels they themselves supplied¹.

That is now the documented state of clinical practice. The ACMG/AMP framework includes computational evidence as criteria PP3 (pathogenic) and BP4 (benign). Examining 727 missense variants curated by ClinGen expert panels, one study found PP3 or BP4 had been applied to 55% of them. Removing those criteria and reclassifying changed the outcome for 15% of the variants where they had been used — moving 14% from pathogenic to likely pathogenic, 24% from likely pathogenic to VUS, and 64% from likely benign to VUS².

Read that carefully. In more than half of expert-curated missense classifications, predictor output is one of the inputs. For a meaningful fraction, it is load-bearing — remove it and the classification moves. Those classifications then flow into ClinVar, where they become the labels for the next generation of benchmarks and the next generation of training sets.

A tool evaluated against that corpus is being asked to agree with a body of evidence it helped create. Some agreement is guaranteed by construction, and nothing in an AUC distinguishes that portion from genuine skill.

What the field has done about it

The honest part of this story is that the problem is well recognised by the people who build and calibrate these tools, and the responses have been substantive.

Independent functional data. The most direct fix is to abandon clinical labels for benchmarking and use deep mutational scanning instead — experimental measurements of protein function generated with no reference to any database. Benchmarking 55 predictors against DMS data from 26 human proteins “while introducing minimal data circularity,” one group found the top ranks dominated by unsupervised methods, with a protein language model first overall³. That is a telling result: methods that never saw a pathogenicity label do best on the benchmark that cannot be gamed by having seen one.

The same work notes the strong showing of a recent supervised tool as evidence that developers are now taking circularity seriously³ — and a follow-up expanded to 97 predictors across 36 proteins found that agreement with functional assays tracks clinical classification performance, which supports DMS as a valid independent yardstick⁴.

Calibration against carefully assembled sets. The ClinGen Sequence Variant Interpretation Working Group calibrated thirteen missense tools by mapping scores to local posterior probabilities and hence to ACMG/AMP evidence strengths, using data sets deliberately assembled to avoid the circularity problem⁵. The outcome matters for anyone using these scores clinically: most tools reached only supporting evidence at the newly established thresholds, several reached moderate, and four — BayesDel, MutPred2, REVEL and VEST4 — could reach strong for pathogenicity and moderate for benignity at particular score intervals⁵⁶.

In the Bayesian point system, supporting is ±1 point and strong is ±4, on a scale where pathogenic classification requires 10⁶. That is the practical translation of a decade of circularity work: a tool reporting an AUC of 0.9 contributes, at best, a modest fraction of the evidence needed to classify a variant.

What the benchmark numbers do and do not establish

Benchmark	What it controls	What still leaks
Random split of a clinical set	Nothing	Type 1, Type 2, and label self-reference
Held-out variants not in training	Type 1	Type 2 and label self-reference
Gene-disjoint split	Type 1 and Type 2	Label self-reference; still uses clinical labels
Curated set excluding PP3/BP4	All three, if training data is also disjoint	Curation errors; limited to resolved variants
Deep mutational scanning	All three	Assay–disease mismatch: function is not pathogenicity

The last row is not a free lunch. DMS measures a protein property in an assay, and discordance between functional scores and clinical pathogenicity is well documented⁷ — one benchmarking study found some DMS datasets classify clinical variants exceptionally well and others poorly³. It trades a circularity problem for a construct-validity problem, which is a good trade but a trade.

The newest tools change the shape, not the question

Models like AlphaMissense are trained without known variant classifications, using population frequency as a weak label alongside structural context — which structurally removes the Type 1 and Type 2 paths through the training data.

But note what they are still evaluated against. The AlphaMissense paper reports ClinVar-based comparisons across ACMG genes, a developmental-disorder benchmark, and cancer hotspots⁸. Clinical labels remain the yardstick, and a review of the field is explicit about the residual risks: expert-assigned labels may contain errors, and tools that replicate those errors get credit for them; clinical benchmarks include only resolved variants, which are not representative of the ones arriving as VUS; and tools trained on clinical data can show inflated performance through information leakage⁹.

There is also a subtler substitution. Using allele frequency as a proxy label imports a different assumption — that common means benign — which is largely true and not universally so, and which ties the model's notion of benign to whichever populations are well represented in the frequency database. The circularity is broken; a dependency on the reference population takes its place.

One more caveat worth carrying: a 2024 extension of the ClinGen calibration to newer tools states plainly that no additional filtering of the evaluation data against each tool's training data was performed⁶. Even careful, circularity-aware work sometimes cannot exclude it, because training sets are not always public.

What to do about it

1. Treat a published AUC as a claim about a benchmark, not about your gene. Ask which dataset produced it and whether that dataset overlapped the training data. If the training set isn't public, Type 1 circularity cannot be excluded — and the 2015 authors argued on exactly that basis that training data should be published¹.
2. Ask for performance on mixed genes specifically. Accuracy on genes containing both pathogenic and benign variants is the number that corresponds to the clinical question. Overall accuracy is dominated by genes where the answer is already known¹.
3. Split by gene, not by variant, in any internal validation you run. A variant-level split leaves Type 2 completely intact.
4. Prefer calibrated thresholds to developer defaults. A tool's own recommended cut-off was not chosen to correspond to an evidence strength; the ClinGen thresholds were⁵. This is the same lesson as the splice-predictor piece, arrived at from the benchmark side.
5. Use one predictor, chosen in advance. The calibration work recommends a single tool fixed per laboratory or per gene rather than a consensus of several⁵, because tools sharing training data and features are not independent evidence.
6. Never count a predictor twice. If a tool's output already contributed to a ClinVar classification you are citing as supporting evidence, that classification is not independent corroboration of the tool.
7. Where functional data exist, weight them accordingly — and check what the assay measured, since a functional readout and a clinical phenotype are different endpoints⁷.
8. Record the tool, version and score with the classification, so a future reanalysis can tell which part of the evidence was computational and reweight it when calibration changes.
9. Be most sceptical exactly where the tool is most useful. Novel genes and mixed genes are where circularity inflated the benchmark and where the model has the least to go on. A high score on a gene with no prior disease association is the case the reported accuracy least describes.

The shape of the error

The published GIAB piece made a neighbouring argument: a 99.5% F1 is measured inside high-confidence regions, so the number is accurate about the territory where calling is easy. The mechanism there is restriction — the benchmark excludes the hard cases.

Here the mechanism is contamination. The hard cases are present in the benchmark; the answers have simply leaked into the model, through the same variant, through the same gene, and lately through the labels themselves. Both produce a number that is correct and that describes something other than what the reader assumes.

None of this means the tools are bad. The unsupervised and structure-based models perform well on benchmarks that cannot be gamed, and the calibration work has given the field defensible evidence weights for the first time. What it means is narrower and more durable: a reported accuracy for a variant-effect predictor is a joint property of the model and the answer key, and the answer key was assembled by a research community that studies some genes and not others, and that has been using these same tools to assign labels for a decade.

The score is real. What it is a score on is the thing to establish before quoting it.

References

1. Grimm DG, Azencott C-A, Aicheler F, et al. The evaluation of tools used to predict the impact of missense variants is hindered by two types of circularity. Human Mutation 2015;36(5):513–523. onlinelibrary.wiley.com/doi/10.1002/humu.22768
2. Ghosh R, Harrison SM, Rehm HL, Plon SE, Biesecker LG, on behalf of the ClinGen Sequence Variant Interpretation Working Group. Evaluating the impact of in silico predictors on clinical variant classification. Genetics in Medicine 2022;24(1):51–59. gimjournal.org — S1098-3600(21)05398-3
3. Livesey BJ, Marsh JA. Updated benchmarking of variant effect predictors using deep mutational scanning. Molecular Systems Biology 2023;19(8):e11474. ncbi.nlm.nih.gov/pmc/articles/PMC10407742
4. Livesey BJ, Marsh JA. Variant effect predictor correlation with functional assays is reflective of clinical classification performance. bioRxiv 2024.05.12.593741. Preprint at the time of writing; verify the published version. biorxiv.org — 2024.05.12.593741
5. Pejaver V, Byrne AB, Feng B-J, et al., ClinGen Sequence Variant Interpretation Working Group. Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations for PP3/BP4 criteria. American Journal of Human Genetics 2022;109(12):2163–2177. ncbi.nlm.nih.gov/pmc/articles/PMC9748256
6. Calibration of additional computational tools expands ClinGen recommendation options for variant classification with PP3/BP4 criteria. Genetics in Medicine 2025. sciencedirect.com — S1098360025000498
7. Livesey BJ, Marsh JA. Interpreting protein variant effects with computational predictors and deep mutational scanning. Disease Models & Mechanisms 2022;15(6):dmm049510. journals.biologists.com — dmm049510
8. Cheng J, Novati G, Pan J, et al. Accurate proteome-wide missense variant effect prediction with AlphaMissense. Science 2023;381:eadg7492. science.org/doi/10.1126/science.adg7492
9. Making sense of missense: challenges and opportunities in variant pathogenicity prediction. Disease Models & Mechanisms 2024;17(12):dmm052218. journals.biologists.com — dmm052218

Zetobit, LLC · Bioinformatics consulting · Lexington, KY · zetobit.com