“We Ran the Standard Pipeline”: Four Questions That Make That Sentence Mean Something

Zetobit Reading the Report

Reading the Report

“We Ran the Standard Pipeline”: Four Questions That Make That Sentence Mean Something

The phrase describes conformity to something nobody has named, at a version nobody has stated. Four short questions turn it into a claim you can act on.

Kanna Nandakumar, PhD  ·  Zetobit

It arrives in an email from the sequencing group, in the methods paragraph at the back of a report, on the second slide of a vendor’s deck. We ran the standard pipeline. It is offered as reassurance, and it usually lands as one, because it sounds like a statement about quality. It is not. It is a statement about conformity — and it does not say to what.

This is not a complaint about analysts. The phrase is honest shorthand, and the person who says it can almost always answer in detail if asked. The trouble is that nobody asks, because the sentence sounds finished. Four questions finish it, and none of them takes more than a line to answer.

01Standard according to whom?

At least three different things travel under that one word. The first is the laboratory’s own written, version-controlled, signed-off procedure. The second is a published community workflow — a widely used toolkit’s recommended configuration, or a shared open-source workflow release. The third is habit: the settings that have been in place since the last person who understood them moved on.

The gap between the first two is wider than the shared vocabulary suggests. The maintainers of the most widely cited “best practices” workflow in human sequencing are unusually direct about this in their own documentation: the canonical version is built for human genome research and tuned to the instrumentation and throughput needs of one specific sequencing facility, and a workflow that has been meaningfully adapted for another use case should not be called by that name at all.¹ Meanwhile, the very first of the seventeen consensus recommendations issued jointly by the Association for Molecular Pathology and the College of American Pathologists is that a clinical laboratory must validate its own bioinformatics pipeline, and the ninth states plainly that the pipeline is part of the test procedure.²

So we followed best practices and our pipeline is validated for this test are two different sentences, and only one of them is about your specimen. Both are perfectly respectable answers. They are just not interchangeable, and the shorthand hides which one you were given.

02Standard as of when?

A pipeline is a dated object. Reporting standards already concede this at a much smaller scale: the joint AMP/ASCO/CAP guidance asks that every reported variant carry a transcript accession together with its version number — NM_000546.5, not NM_000546 — because without the version the description cannot be resolved unambiguously.³ The ACMG secondary-findings gene list is versioned for the same reason, and has moved from v3.0 to v3.3 in four years.⁴ If a version number is load-bearing for one transcript and one gene list, it is load-bearing for the workflow that produced the entire document. The same AMP/CAP recommendations close by requiring supplemental validation whenever a significant change is made to any component — a rule that only means anything if the laboratory can tell you which version ran.²

Two dates matter here, and they are not the same date. One is the version of the code. The other is when the reference genome build and the annotation and population resources were last refreshed. The second moves much faster than the first, is far more often left unrecorded, and is usually the one that explains why the same sample analyzed twelve months apart does not produce the same page.

A methods paragraph of this kind is not defective — it is typical, and it is compatible with excellent work. But the single phrase carrying the analytical claim refers to four objects that the document never names. Each has a short, concrete answer that the issuing laboratory already holds.

03Validated on what — and does my sample look like that?

The seventh AMP/CAP recommendation is the one clinicians should know by heart: validation of the pipeline must be appropriate and applicable to the intended clinical use, the specimen types, and the variant types of the test.² “Standard” is therefore always standard within an envelope. Outside that envelope the pipeline still runs — software does not decline — and it still emits a document that looks exactly like the ones inside it.

The practical form of the question is short. Was degraded or fixed tissue in the validation set, or only high-quality fresh material? Was the assay validated with a matched normal, or on tumor tissue alone? Down to what allele fraction? Were copy-number changes, structural rearrangements, and repeat expansions in scope — or is their absence from your report a silence rather than a negative result? An answer of we run thousands of these a month is a statement about volume. It is not a statement about validation, and the two are easy to mistake for each other in conversation.

04What did it decide before I saw it?

Every pipeline is a stack of defaults, and a default is a decision someone made once, for a general case, and stopped thinking about. The guidance treats them as decisions: any method that alters or filters sequence reads before interpretation must itself be validated and documented, precisely because filtering governs what reaches a human being at all.²

How much room is left inside the word “standard”? Considerably more than it implies. When one group built seventy germline pipelines from seven aligners and ten variant callers and ran all of them over the same two genomes, average agreement across pipelines was 58.1% for single-nucleotide variants and 34.1% for insertions and deletions in the European sample, and 40.1% and 25.0% in the African one. Agreement was worst exactly where clinical interest is highest: rare variants were called concordantly at roughly a third to a half the rate of common ones, and variants predicted to have severe functional consequences agreed less often than benign ones.⁵ Annotation is no tighter. Given identical variants, two standard transcript sets processed through the same software agreed on only 44% of putative loss-of-function calls, and two widely used annotation tools running on the same transcripts agreed on 65%.⁶ Every one of those configurations was somebody’s standard pipeline.

That spread is not evidence that anyone did anything wrong. It is evidence that the phrase, unqualified, does not narrow the space of possible results nearly as much as its confidence suggests — and that it narrows it least for the findings that made you order the test.

The research version of the same four questions

Outside the clinical laboratory there is often no validated procedure to point at, and that is entirely legitimate; a core facility or academic collaborator is not operating under accreditation standards. The four questions do not disappear, they change form. Which workflow, at which release tag. Which genome build and which annotation release. Which parameters differ from the shipped defaults. What was filtered out before the table reached you. A collaborator who answers all four in two lines of an email has given you far more than a laboratory that answers none of them across a full page of methods.

None of this is an argument against standard pipelines. Choosing a well-tested default over a bespoke configuration is usually the right call, and a laboratory that has frozen, documented, and validated its workflow is doing the thing you want it to do. The point is narrower: the sentence, by itself, is underdetermined. It asserts a relationship to a reference nobody has named, at a version nobody has stated, validated for a use nobody has confirmed resembles yours. The answers exist. They are short. They are simply not in the sentence — and the most reliable way for them to reach the person who has to act on the result is for that person to ask.

Three questions to ask

1. Which version of the pipeline produced this result, and on what date were its reference genome build and annotation resources last refreshed?
2. What specimen types and classes of variant were in the validation set for this assay — and is mine among them?
3. What was filtered out before this document reached me, and is there anything in what was removed that I would have wanted to see?

References

1. Broad Institute. About the GATK Best Practices. GATK Documentation. gatk.broadinstitute.org
2. Roy S, Coldren C, Karunamurthy A, et al. Standards and Guidelines for Validating Next-Generation Sequencing Bioinformatics Pipelines: A Joint Recommendation of the Association for Molecular Pathology and the College of American Pathologists. J Mol Diagn. 2018;20(1):4–27. doi:10.1016/j.jmoldx.2017.11.003
3. Li MM, Datto M, Duncavage EJ, et al. Standards and Guidelines for the Interpretation and Reporting of Sequence Variants in Cancer: A Joint Consensus Recommendation of the Association for Molecular Pathology, American Society of Clinical Oncology, and College of American Pathologists. J Mol Diagn. 2017;19(1):4–23. doi:10.1016/j.jmoldx.2016.10.002
4. ACMG Secondary Findings Working Group. ACMG SF v3.3 list for reporting of secondary findings in clinical exome and genome sequencing: A policy statement of the American College of Medical Genetics and Genomics. Genet Med. 2025. gimjournal.org
5. Hwang KB, Lee IH, Li H, et al. Comparative analysis of whole-genome sequencing pipelines to minimize false negative findings. Sci Rep. 2019;9:3219. doi:10.1038/s41598-019-39108-2
6. McCarthy DJ, Humburg P, Kanapin A, et al. Choice of transcripts and software has a large effect on variant annotation. Genome Med. 2014;6(3):26. doi:10.1186/gm543

Reading the Report is a Zetobit series for people who receive genomic and multi-omic results rather than produce them. Each piece takes the document as given and asks what it establishes, what it leaves open, and what to ask next.