Benchmarking Against Truth Sets: Why 99.5% F1 Is Measured Where Calling Is Easy

Zetobit.

Bioinformatics Insight Series

Benchmarking Against Truth Sets: Why 99.5% F1 Is Measured Where Calling Is Easy

A GIAB benchmark reports accuracy inside high-confidence regions — and those regions are defined, in part, by excluding the places where methods disagree. The score is real and the resource is excellent. It is simply computed on the portion of the genome that was tractable enough to establish truth in.

By Kanna Nandakumar, PhD · Zetobit

Validating a variant calling pipeline has a standard form. Run it on HG002, compare against the Genome in a Bottle benchmark with hap.py, report precision, recall and F1. The numbers come back around 99.5% for SNVs, somewhat lower for indels, and the pipeline is documented as accurate.

That workflow is correct, and GIAB is one of the most valuable resources the field has produced. The question worth asking is narrower: accurate where?

A benchmark comparison is restricted to a BED file of high-confidence regions. Variants your pipeline calls outside that file are not counted as false positives. Variants it misses outside that file are not counted as false negatives. They are excluded from the arithmetic entirely — which means the headline number is silent about them in both directions.

How the high-confidence region gets defined

The exclusion is not arbitrary, and understanding its logic is the whole point.

GIAB's integration approach combines results from different sequencing technologies and analysis methods, using expert-driven heuristics and features of the mapped reads to determine where each method should be trusted. Critically, it excludes regions where all methods may have systematic errors, or locations where methods produce different variants or genotypes with no evidence of bias or error.

Read that carefully, because it explains the structure of the problem. The benchmark region is defined partly by consensus among callers. Where methods agree, truth can be established. Where they systematically disagree — or might all be systematically wrong together — the region is set aside, because a benchmark you cannot trust is worse than no benchmark.

That is exactly the right decision for building a truth set. It also means the excluded regions are, by construction, enriched for the places where variant calling is hardest. The benchmark and the difficulty are defined by the same property.

The published exclusion criteria make this concrete. Among them: regions covered by segmental duplications greater than 10 kb with more than five copies at over 99% similarity; L1H LINEs over 500 bp; regions 15 kb either side of reference N's; long tandem repeats and low-complexity sequence; and regions difficult to map from short reads.

The regions excluded from the benchmark and the regions where calling fails are, largely, the same regions — because both are defined by the same difficulty.

The excluded fraction is not small, and it is not random

The scale has improved substantially and is worth stating precisely. The v3.3.2 benchmark covered about 85% of GRCh38 and primarily used short-read technologies, excluding most segmental duplications. The v4.2.1 benchmark added long and linked reads to reach 92% of the autosomal GRCh38 assembly, adding more than 300,000 SNVs and 50,000 indels, and including 16% more exonic variants — many in challenging, clinically relevant genes not covered previously, such as PMS2.

So roughly 8% of the autosomes sits outside even the current benchmark, and the sex chromosomes have their own separate treatment. A recent complete diploid assembly of HG002 quantified the gap directly: beyond the sex chromosomes, it contains an additional 701.4 Mb — 11.7% — of high-confidence autosomal sequence absent from the v4.2.1 variant benchmark, mostly satellite repeats but also hundreds of megabases of typically segmentally duplicated sequence.

The single most useful number in this literature is what happened when the benchmark expanded. Moving from v3.3.2 to v4.2.1 identified eight times more false negatives in a short-read variant call set than the previous benchmark had. The call set did not change. The pipeline did not change. The region where accuracy was measured changed, and eight times as many missed variants became visible.

Figure 1. Expanding the benchmark from v3.3.2 to v4.2.1 revealed eight times more false negatives in the same short-read call set. This is the clearest available demonstration that a benchmark score characterizes a pipeline and a region jointly. A pipeline reported as highly accurate against a narrower benchmark has not been shown to be accurate in the territory that benchmark omitted.

Difficulty concentrates in the clinically important places

If the excluded regions were biologically inert, none of this would matter much. They are not.

GIAB assembled a list of 4,697 autosomal medically relevant genes and found that 395 of them showed less than 90% inclusion in the v4.2.1 benchmark. A curated benchmark was then built for 273 of those genes, adding over 17,000 SNVs, 3,600 indels and 200 SVs. Even after that effort, another 122 autosomal genes covered under 90% by v4.2.1 remain excluded from the CMRG benchmark.

These genes are qualitatively harder, not marginally so: for 99% of the newly covered genes, at least 15% of the gene region is either challenging to sequence or contains challenging variants in HG002.

And measuring performance there changes the picture substantially. For indels longer than 15 bp, recall for HiFi-DeepVariant fell from 99.2% against v4.2.1 to 82.0% against the CMRG benchmark, with precision falling from 99.9% to 96.0%. That is a strong method on high-quality long reads, and its recall dropped seventeen points when the evaluation moved into difficult genes.

The starkest finding in that work concerns the reference itself. False duplications in GRCh37 or GRCh38 caused reference-specific missed variants in medically relevant genes including CBS, CRYAA and KCNE1 — and masking those false duplications improved variant recall from 8% to 100%.

Eight percent recall in a medically relevant gene, invisible to a standard benchmark comparison because the gene sat outside the high-confidence region. That is the failure mode this article is about, in one number.

The benchmark is one sample, on one reference

Three further scope limits are worth holding in view, none of them criticisms of GIAB.

It is a small number of genomes. HG002 and its parents, HG001, and a handful of others. A pipeline tuned to perform well on HG002 has been tuned on one person's variant spectrum. Population-specific variation, and regions where another individual's haplotypes diverge more from the reference, are not represented.

It is a germline benchmark. Somatic calling faces low variant allele fractions, tumor purity, subclonality and FFPE artifacts — none of which a germline truth set exercises. A germline F1 says nothing about somatic sensitivity at 3% VAF.

It reflects the reference build's errors. The false-duplication findings show that benchmark and reference limitations interact: a region can be excluded, or a variant missed, because of an assembly artifact rather than anything about the sequencing.

Table 1. What a headline GIAB F1 score does and does not characterize. None of these are defects in the benchmark; they are scope boundaries that belong in the reported result.

Dimension	What the score covers	What it does not
Genomic territory	High-confidence regions, ~92% of autosomes in v4.2.1	Segmental duplications, satellites, long repeats, hard genes
Variant class	SNVs and indels under ~50 bp	Structural variants, CNVs, repeat expansions
Sample diversity	A small set of deeply characterized genomes	Population-specific variation; divergent haplotypes
Application	Germline calling at normal zygosities	Somatic, low-VAF, mosaic, FFPE, liquid biopsy
Reference	Performance given a specific assembly	Errors caused by that assembly's own artifacts

What to report instead of one number

Report stratified performance, not a global F1

GIAB and GA4GH publish stratification BED files precisely for this, and hap.py consumes them directly. Performance in low-mappability regions, segmental duplications, homopolymers, tandem repeats and extreme-GC sequence tells you where a pipeline actually stands. A single global number averages the easy majority over the hard minority and is dominated by the former.

Quote the benchmark version and region with the score

"99.5% F1" is incomplete. "99.5% F1 for SNVs against HG002 v4.2.1 high-confidence regions, GRCh38" is a claim someone can evaluate. Given the 8× false-negative shift between versions, the version is part of the measurement.

Benchmark the genes your assay is for

If a panel targets genes with known segmental duplication problems — PMS2, SMN1, CYP2D6, HBA1/HBA2 — a genome-wide F1 is close to irrelevant to its clinical performance. The CMRG benchmark exists for this purpose, and it is the correct comparator for a clinically focused assay.

Track the non-assessed fraction as a metric

How much of your assay's target territory falls outside the benchmark region is a number you can compute once and report. It is the size of the blind spot, and it belongs in a validation document next to the F1.

Use orthogonal methods where the benchmark cannot reach

For regions no truth set covers, benchmarking is not available and confirmation has to come from elsewhere — long reads, targeted assays such as MLPA for known-difficult loci, or orthogonal chemistry. The absence of a benchmark is not evidence of accuracy.

The short version

A benchmark F1 characterizes a pipeline and a region jointly, and the region was defined partly by excluding where methods disagree — which is where calling is hardest. Expanding the benchmark from v3.3.2 to v4.2.1 found eight times more false negatives in an unchanged call set. Report stratified performance and the benchmark version, and treat the non-assessed fraction of your target as a metric rather than an absence.

A resource doing exactly what it says

It would be a serious misreading to treat any of this as a criticism of Genome in a Bottle. The consortium documents its exclusions in detail, publishes the stratification files that make honest evaluation possible, states benchmark coverage explicitly in every release, and has spent years methodically expanding into the hard regions — the CMRG effort exists specifically because its authors identified the gap and set out to close it. The benchmark is transparent about being a benchmark of a defined region.

The gap opens downstream, when a stratified, carefully scoped evaluation is compressed into a single number in a validation summary or a marketing page. At that point the region silently disappears and the score reads as a property of the pipeline alone.

Nothing about a 99.5% F1 is false. It just describes the part of the genome where establishing truth was possible — and the reason truth was hard to establish in the rest is the same reason your pipeline will struggle there.

Zetobit builds and validates CAP/CLIA-compliant NGS pipelines, including stratified benchmarking against GIAB and CMRG truth sets with explicit reporting of non-assessed target territory. If you are validating a clinical assay or interpreting a vendor's accuracy claim, we're happy to talk.

References

1. Wagner J, Olson ND, Harris L, et al. Benchmarking challenging small variants with linked and long reads. Cell Genomics 2(5):100128 (2022). doi:10.1016/j.xgen.2022.100128.
2. Wagner J, Olson ND, Harris L, et al. Curated variation benchmarks for challenging medically relevant autosomal genes. Nature Biotechnology 40:672–680 (2022). doi:10.1038/s41587-021-01158-1.
3. Krusche P, Trigg L, Boutros PC, et al. Best practices for benchmarking germline small-variant calls in human genomes. Nature Biotechnology 37:555–560 (2019). doi:10.1038/s41587-019-0054-x.
4. Global Alliance for Genomics and Health Benchmarking Team. GIAB high-confidence set documentation. GA4GH benchmarking-tools repository, accessed July 2026.
5. Olson ND, Wagner J, McDaniel J, et al. precisionFDA Truth Challenge V2: Calling variants from short and long reads in difficult-to-map regions. Cell Genomics 2(5):100129 (2022). doi:10.1016/j.xgen.2022.100129.
6. A complete diploid human genome benchmark for personalized genomics. bioRxiv preprint (2025). doi:10.1101/2025.09.21.677443. Preprint — check for the peer-reviewed version before citing.
7. Rhodes J, et al. Systematic benchmark of state-of-the-art variant calling pipelines identifies major factors affecting accuracy of coding sequence variant discovery. BMC Genomics 23:126 (2022). doi:10.1186/s12864-022-08365-3.
8. Zook JM, McDaniel J, Olson ND, et al. An open resource for accurately benchmarking small variant and reference calls. Nature Biotechnology 37:561–566 (2019). doi:10.1038/s41587-019-0074-6.