The Pipeline Ran. Did It Run the Same?

Zetobit Bioinformatics Insight Series

↻

Insight No. 13 · Reproducibility

The Pipeline Ran. Did It Run the Same?

Nextflow gives you the machinery for reproducible workflows. It doesn't hand you reproducibility for free — and the gap between the two is where clinical pipelines quietly drift.

Zetobit LLC Workflow Engineering ~7 min read

A workflow that finishes without error feels like a success. But "it completed" and "it would complete identically next quarter, on another cluster, for an auditor" are different claims — and only the second one matters when the output feeds a clinical report.

Nextflow has become the default substrate for that kind of work, and for good reason. It separates workflow logic from execution, so the same pipeline description runs on a laptop, an HPC scheduler, or AWS Batch without edits. It resumes failed runs from the last cached step. And through the nf-core community it comes with a large catalog of peer-reviewed, versioned pipelines. As of early 2026 that catalog spans well over a hundred pipelines across genomics, transcriptomics, proteomics, and imaging.

All of that enables reproducibility. None of it guarantees it. The distinction is the whole subject of this issue.

Three things people mean by "reproducible"

The word gets used for three quite different guarantees, and a pipeline can satisfy one while failing another:

1

Re-runnable The same command launches the same workflow again without hand-holding. Nextflow's -resume and its work-directory cache handle this well.

2

Portable The workflow produces the same result on a different machine or executor. This depends almost entirely on how the software environment is pinned — not on Nextflow itself.

3

Bit-for-bit deterministic Every run yields byte-identical output. This is the hardest tier, and some tools make it impossible without extra constraints (thread counts, seeds, sort orders).

Most teams believe they're operating at tier three and are actually somewhere in tier one or two. The failures are rarely dramatic. A variant caller shifts a borderline allele fraction by a hair. A count matrix reorders. A dependency updated silently between runs. Nothing crashes — the numbers just aren't the ones from last time.

Where the drift comes from

Four sources account for most of the irreproducibility we see in practice, and none of them are Nextflow's fault:

• Floating container tags. Referencing an image as :latest or even :1.2 means the bytes behind that name can change under you. A tag is a label, not a fingerprint.
• Unpinned pipeline versions. Running nextflow run nf-core/rnaseq without -r pulls whatever the default branch is that day.
• Unpinned Nextflow itself. The runtime has semantics that evolve across releases; a workflow validated on one version isn't guaranteed to behave identically on the next.
• Non-deterministic tools. Multi-threaded aligners and callers can produce order-dependent output unless explicitly constrained.

The core idea

A tag names an image. A digest is the image. Content-addressable identifiers — the sha256:… digest — guarantee that not a single byte of the environment, from the OS up through every library, has changed between runs.

Pinning, concretely

Reproducibility at tier two is mostly a matter of replacing every mutable reference with an immutable one. That means pinning the pipeline revision, the runtime, and the container by digest rather than tag:

# Pin the pipeline revision to a tagged release, not a branch
nextflow run nf-core/rnaseq -r 3.14.0 -profile singularity

# Pin the Nextflow runtime itself for a given run
NXF_VER=24.10.0 nextflow run ...

# Pin containers by digest in the config, not by tag
process {
  container = "quay.io/biocontainers/samtools@sha256:9f8401…"
}

The pattern generalizes. Anything referenced by a name that someone else can repoint — a tag, a branch, a channel — is a future discrepancy waiting to surface. Anything referenced by a hash is settled.

The part pinning can't fix

Even a perfectly pinned pipeline has two remaining exposures, and honesty about them is part of doing this well.

The first is data. A workflow pinned to the byte is still only as reproducible as the reference genome, annotation, and input files it consumes. Those live outside the container. Version-control what you can, and record checksums and accession versions for what you can't — a pinned pipeline against a silently updated reference is not a reproducible analysis.

The second is time. Registries change, images get repointed, external resources rot. A digest protects you against silent mutation, but only an archived copy protects you against disappearance. For work that must be reconstructable years later — which describes most clinical and regulatory contexts — the container images and references themselves belong in durable storage, not just referenced by hash from a registry you don't control.

In practice

Treat reproducibility as a property you configure, then verify — not one you inherit by choosing the right framework. The framework gives you the levers. Pulling them is the work.

The takeaway

Nextflow and nf-core give you every mechanism a reproducible workflow needs: version-tagged pipelines, container support, an execution cache, and portability across infrastructure. What they don't give you is a default posture of maximum rigor. The defaults favor convenience — floating tags, latest versions, whatever the reference happens to be.

Closing the gap is unglamorous and entirely mechanical: pin the pipeline revision, pin the runtime, pin containers by digest, version the reference data, archive what you can't re-fetch, and then confirm that two runs actually agree. Do that, and "the pipeline ran" and "the pipeline ran the same" finally become the same sentence.

Zetobit · Bioinformatics consulting · Lexington, KY zetobit.com