Variant Calling in Segmental Duplications and Paralogous Regions: Why High-Homology Genes Defeat Standard Short-Read Pipelines

Zetobit · Bioinformatics Insight Series

Variant Calling in Segmental Duplications and Paralogous Regions: Why High-Homology Genes Defeat Standard Short-Read Pipelines

When two stretches of the genome are nearly identical, a short read cannot say which one it came from. The aligner knows this, flags its uncertainty — and the variant caller quietly throws the evidence away.

By Kanna Nandakumar, PhD Zetobit LLC

A short-read variant pipeline reports a clean, high-confidence genotype for most of the exome. Then it reaches a gene like PMS2, SMN1, or CYP2D6 and goes silent — not because there is no variant, but because it cannot decide where the reads belong. The reads pile up, the coverage looks adequate, and the caller emits nothing, or worse, emits something that came from a different locus entirely.

This is not a coverage problem and it is not a base-quality problem. It is a mapping problem, and it is structural. Roughly 5% of the human genome on GRCh38 — and closer to 7% on the complete T2T-CHM13 assembly — falls in segmental duplications, also called low-copy repeats: blocks longer than a kilobase that share 90% or greater identity with another location.¹ Duplications at least 10 kb long with 98%+ identity alone cover roughly 3% of the genome and overlap more than 800 protein-coding genes.² Several of those genes are clinically actionable. This piece is about why the standard pipeline fails them, and what "handling" these regions actually requires.

Mapping quality encodes a choice the caller can't see

When an aligner places a read, it assigns a mapping quality (MAPQ): a phred-scaled estimate of the probability that the placement is wrong. The critical detail is what happens when a read fits two locations equally well. If a read maps to a duplicated region and its paralog just as well, the aligner cannot prefer one over the other, so it assigns MAPQ 0 — the explicit signal that this placement is a coin flip.³

Every downstream caller is built to treat that signal as poison. GATK, Mutect2, and their peers discard or heavily downweight low-MAPQ reads by default, because in unique regions a low-MAPQ read genuinely is an alignment error and keeping it would inflate false positives.¹ The logic is sound almost everywhere. But inside a segmental duplication, MAPQ 0 does not mean "bad read" — it means "real read from an ambiguous origin." The filter that protects the rest of the genome is precisely the filter that blinds the pipeline here.

The consequence shows up starkly in benchmarking. Two state-of-the-art callers agreed on 99.7% of SNVs inside the Genome in a Bottle high-confidence regions, but only 76.5% outside them — and those excluded regions are, disproportionately, the low-copy repeats the benchmark historically had to leave out because no one could call them reliably.¹

Reads from a gene and its high-identity paralog map equally well to both loci. The aligner encodes that ambiguity as MAPQ 0; the caller, by default, drops those reads — producing either a dropout or a call whose true locus of origin is unknown.

The clinical cost, in specific genes

The abstraction becomes concrete fast. PMS2, a mismatch-repair gene whose loss causes Lynch syndrome, has a pseudogene, PMS2CL, that is roughly 98% identical to PMS2 across exons 11–15.⁴ Short reads from this region cannot be assigned unambiguously by standard pipelines, so a variant may be called that actually resides in the pseudogene, or a true pathogenic variant may be missed.⁵ Worse, sequence exchange and gene conversion between the two loci are frequent enough that even the handful of reference-genome bases that differ between them cannot be reliably attributed to one copy or the other.⁶ The reference itself does not encode a stable answer.

The same structural trap recurs across the clinically important genome: SMN1 versus SMN2 in spinal muscular atrophy, CYP2D6 versus CYP2D7 in pharmacogenomics, U2AF1 versus its near-exact copy U2AF1L5 in myeloid malignancy — where users report entire genes yielding only MAPQ-0 reads and callers returning empty output.⁷ A pipeline that is validated as "highly sensitive" on genome-wide benchmarks can be silently blind at exactly these loci, because the benchmarks that certified it excluded them.

The failure is invisible by construction. A dropout in a paralogous gene does not raise an error. Coverage plots look normal, because the reads are present — they are simply unusable to the caller. Unless a pipeline explicitly reports callability per region, a paralog dropout is indistinguishable from a confidently wild-type result.

Why the obvious fixes don't work

"Just keep the MAPQ-0 reads." Disabling the mapping-quality filter re-admits the ambiguous reads, but it does not tell the caller which locus they belong to. A standard diploid caller then merges evidence from two loci into one, and reports a genotype that is a blend of gene and pseudogene — a variant that exists in neither as called.

"Mask the paralog." Masking one copy before alignment forces all reads onto the surviving copy. This concentrates coverage, but a variant that is actually private to the masked pseudogene now appears to sit in the functional gene. You have traded a dropout for a false positive, which in a clinical report is the more dangerous error.

"Trust the reference differences." Paralogous sequence variants (PSVs) — the fixed base differences between copies — are the natural anchor for telling reads apart. But the frequency of any given PSV varies widely across populations and, at loci with active gene conversion, across individuals.⁶ A PSV that discriminates the copies in one sample may be uninformative in the next.

What actually works

The methods that succeed share one premise: a duplicated region cannot be called one copy at a time. The reads from all homologous copies have to be analyzed jointly, at the correct combined ploidy, rather than forced through a per-locus diploid model.¹ In practice that takes a few forms:

• Higher-ploidy joint calling. Align permissively to a single representative copy and call with the ploidy expected across all copies — for a gene plus one near-identical pseudogene, a tetraploid model — so the caller expects and can partition four alleles instead of two.⁸
• PSV-aware read assignment. Use gene-specific invariant positions, catalogued from population data, to sort reads to their locus before calling — the strategy behind refined PMS2 pipelines that restrict exon 11 to reads intersecting gene-specific positions.⁵
• Equivalent-capture reflex workflows. Design probes to enrich gene and pseudogene with equal efficiency, resolve what short reads can (over 90% of samples in one PMS2 workflow), and reflex the remainder to long-range PCR for definitive localization.⁹
• Long reads. Reads spanning kilobases can carry enough flanking unique sequence — or enough PSVs on a single molecule — to map unambiguously where short reads never could, resolving many of these loci outright.²

None of these is a drop-in flag. Each requires knowing, in advance, which regions of your panel or exome are paralogous, and treating them as a distinct analytical problem with its own model, its own truth set, and its own reporting. The single most valuable thing a clinical pipeline can do is the cheapest: maintain an explicit list of low-mappability regions and declare non-callability rather than emit a confident wild-type call it cannot justify. A stated "cannot resolve — reflex required" is a correct answer. A silent dropout dressed as a negative result is not.

The takeaway

Segmental duplications are not an exotic edge case; they overlap hundreds of genes, several of them the exact ones a hereditary-cancer or pharmacogenomic panel exists to interrogate. The standard short-read pipeline does not fail loudly here. It fails by design and in silence, because the filters that make it accurate everywhere else are the filters that make it blind in these regions. Knowing where those regions are — and refusing to report through them without a model built for them — is the difference between a pipeline that looks validated and one that is.

References

1. Bansal V, et al. A multilocus approach for accurate variant calling in low-copy repeats using whole-genome sequencing. Bioinformatics. 2023;39(Suppl 1):i279. PMC10311303
2. Bhattacharya S, Bansal V. Sensitive alignment using paralogous sequence variants improves long-read mapping and variant calling in segmental duplications. bioRxiv. 2020. 10.1101/2020.07.15.202929
3. Zero mapping quality and multi-mapping reads (BWA behaviour). Biostars discussion. biostars.org/p/60665
4. P711: PMS2 in somatic oncology testing — real or pseudogene interference? Genetics in Medicine Open. 2024. gimopen.org
5. Open-source bioinformatic pipeline to improve PMS2 genetic testing using short-read NGS data. J Mol Diagn. 2024. jmdjournal.org
6. Bouras A, et al. PMS2 or PMS2CL? Characterization of variants detected in the 3′ of the PMS2 gene. Genes Chromosomes Cancer. 2024. gcc.23193
7. How to keep MAPQ on multiple-mapping reads (U2AF1 / U2AF1L5). GATK community. gatk.broadinstitute.org
8. Equivalent hybrid capture reflex workflow for the 3′ exons of PMS2. Genet Med / bioRxiv 379693. PMC6162901
9. SDrecall: a sensitive approach for variant detection in segmental duplications. Genome Biology. 2026. s13059-025-03928-5

Zetobit LLC · Bioinformatics Insight Series · Lexington, KY · zetobit.com