Batch Confounding in Public Data: Why the Correction You Apply Can Be Worse Than the Effect You Remove
Bioinformatics Insight Series
Batch Confounding in Public Data: Why the Correction You Apply Can Be Worse Than the Effect You Remove
In a designed experiment, batch is a nuisance you randomize against. In TCGA and GEO the design already happened — and batch is frequently entangled with the variable you care about. That is a state no correction method can undo, and some correction methods make it considerably worse.
Public genomic archives are the most productive infrastructure the field has built. TCGA alone put molecular data from more than 10,000 patients across 33 cancer types into open hands, and GEO holds tens of thousands of expression studies. A question that once needed a grant and three years now needs an afternoon.
What you cannot download is the experimental design. Someone else made those choices — which samples went on which plate, which center sequenced what, which platform was current that year — and those choices are now fixed. You inherit them along with the counts.
Most of the time that is fine. Occasionally the design has a property that makes a particular question unanswerable, and the analysis will not tell you when that happens. It will return results.
Two different problems that share a name
A batch effect is a systematic technical difference between groups of samples processed separately. Leek and colleagues describe them as occurring because measurements are affected by laboratory conditions, reagent lots and personnel differences — and note that this becomes a major problem specifically when batch effects are correlated with an outcome of interest.
That conditional clause separates two situations that get treated as one.
When batch is independent of the biology — every condition appears in every batch — the batch effect adds noise. It costs power, it can be modelled, and correction methods work as advertised. This is the case most people picture.
When batch is correlated with the biology — all the tumors ran in one set of plates and all the normals in another — the batch effect and the biological effect occupy the same statistical space. No procedure applied afterward can separate them, because the information required to do so was never collected. Correction here does not fix the problem; it decides how the shared variance gets allocated, invisibly.
Designed experiments avoid this by randomizing samples across batches. Public data reuse cannot, because the randomization decision is years in the past.
How entangled the archives actually are
This is not hypothetical, and the evidence is specific.
In TCGA RNA-seq, samples were collected from tissue source sites, allocated to 96-well sequencing plates, and processed at various times — and one methods paper notes that plates are completely confounded with times, making it difficult to distinguish plate effects from time effects. Some datasets, including uveal melanoma and kidney chromophobe, were generated using a single plate, which removes plate as a variable by removing the comparison entirely.
A systematic look at TCGA isomiR data found plate batch effects in every cancer entity examined, sequencing platform effects in every project that combined Illumina GA and HiSeq runs, and tumor purity acting as a batch effect in most projects. In several cohorts, principal components were significantly associated with tumor subtype — which is precisely the entanglement that makes a subtype comparison hazardous.
The germline side is starker. An analysis of TCGA exome data reported roughly 30% variability in the number of germline variants per sample across sequencing centers, with the effect extending to nucleotide composition and variant frequencies. The authors found batch-dependent differences in germline variant distributions across numerous genes, including prominent cancer predisposition genes — and for most known predisposition genes, a distinct batch-dependent difference. Separately, whole-genome amplification of DNA prior to sequencing produced an artificially increased burden of loss-of-function indel calls that confounded association analyses relating germline variants to tumor type despite stringent filtering.
Read that last one carefully. The technical variable was sample preparation method. The biological variable was tumor type. They were correlated, so the association test reported preparation chemistry as cancer biology.
In fairness, the same TCGA methods paper found that major known biological populations are well-distributed across plates in several datasets, showing the absence of large confounding effects there. Confounding is dataset-specific and question-specific. That is the point: it has to be checked, not assumed in either direction.
Figure 1. Whether a batch effect is a nuisance or a fatal confound is a property of the cross-tabulation, not of the data's quality or the method chosen. A balanced design permits correction; a fully confounded one does not, at any sample size. The distinguishing check costs one line and is skipped far more often than it is run — partly because nothing downstream fails when it should have been run.
Correction can make it worse, not just fail
The instinct on discovering a batch effect is to apply ComBat and proceed. In a confounded design that instinct is actively harmful, and the effect is quantified.
Nygaard, Rødland and Hovig showed that applying batch correction to studies with unbalanced designs may inadvertently exaggerate the differences observed. Their demonstration on a real dataset is the number worth remembering: one pipeline using the traditional ComBat approach yielded over 1,000 differentially expressed probesets, while an alternative approach using batch as a fixed effect and averaging technical replicates recovered 11.
Same data. Same biological question. Two orders of magnitude apart, decided by how batch was handled.
The mechanism is understood. A later analysis explains that removing batch effects with two-step methods such as ComBat introduces a correlation structure into the adjusted data, so if downstream modelling does not account for it, actual false-positive and false-discovery rates can be much higher than their nominal values. The correction is applied in step one; the test in step two proceeds as though the data were independent observations, which they no longer are.
The methylation literature reports the same pattern from simulation. Correcting randomly generated samples with ComBat produced alarming numbers of false-positive results in unbalanced as well as balanced sample distributions, with the number of false positives increasing exponentially as more factors were corrected. Larger sample sizes reduced but did not eliminate the effect. Those inputs contained no signal at all.
Nygaard's own diagnostic is elegantly simple and worth adopting: replace the real data with random numbers and rerun the pipeline. If the test statistic is still inflated, the inflation is coming from the design and the correction, not from biology.
Cross-study integration is the same problem, harder
Combining GEO studies raises a version of this that is often structurally unfixable.
When each study contributes samples of only one condition — three studies of cases, four of controls — study identity and disease status are the same variable wearing two labels. And in cross-platform meta-analysis the entanglement runs deeper still: one COPD meta-analysis noted that the study factor is directly related to the microarray platform type, then applied ComBat to correct for study-factor variation. Where platform, study, and condition all coincide, correcting for one corrects for all of them.
There is a real methodological alternative, and it is not batch correction. Meta-analysis proper — computing an effect size within each study and combining the effect sizes — never merges the samples, so it never asks a model to separate study from condition. It answers a slightly different question, with less power, and it is honest about the design it was given.
| Structure | Typical source | What is possible |
|---|---|---|
| Balanced | Conditions randomized across plates or runs | Model batch as a covariate; correction behaves as intended |
| Partly confounded | Uneven allocation; subtype enriched in certain plates | Separable in principle; reduced power; correction needs care |
| Fully confounded | One condition per batch, per center, or per study | Not separable at any sample size; report the limitation |
| Nested technical | Plate confounded with processing date; study with platform | Cannot attribute the effect; describe it as combined |
| Biological-technical | Tumor purity or WGA prep correlated with tumor type | Adjusting removes real biology along with the artifact |
What to do before the analysis
Cross-tabulate condition against every technical variable you have
Plate, sequencing center, platform, processing date, tissue source site, and for TCGA the barcode fields that encode several of these. An empty cell in that table is a finding — it means the comparison you want to make has no within-batch support. Do this before any normalization, because afterward the structure is harder to see.
Decide whether the question is answerable, then choose a method
The order matters. Method selection is a second-order concern once you know whether the design supports the inference. A fully confounded comparison does not become answerable by choosing a better correction tool — it becomes a comparison you report as unanswerable in this dataset, which is a legitimate scientific result.
Prefer modelling batch to removing it
Including batch as a covariate in the differential expression model keeps the uncertainty in the model, where the standard errors can account for it. Two-step removal followed by a naive test discards exactly that bookkeeping — which is the documented source of the exaggerated significance.
Run the random-data sanity check
Replace expression values with random numbers, keep the real design and the real pipeline, and see how many hits come back. It takes minutes and directly measures how much of your result the pipeline can manufacture without any biology present.
For multi-study work, meta-analyse rather than merge
Effect sizes computed within study and combined across studies keep the study boundary intact. Merging expression matrices and correcting for study is the option that quietly requires the model to distinguish study from condition — the thing the design may not allow.
Report the design you inherited
The cross-tabulation belongs in the supplement of any reuse paper. It is the single most informative object for a reader deciding how much to believe, and it is almost never included.
The short version
Batch effects are a nuisance when batch is independent of the biology and a wall when it isn't — and public archives were designed by someone else, so which situation you are in is a fact you discover rather than control. Correction methods do not distinguish the two cases. Applied to a confounded design they redistribute variance and can inflate significance, in one documented case from 11 differentially expressed probesets to over 1,000.
An inherited design, not a downloaded one
The framing that helps here is that a public dataset is not raw material. It is a completed experiment, with all the constraints a completed experiment carries, and reuse means adopting a design chosen for someone else's question.
Sometimes that design happens to support your question, and the reuse is close to free — which is the enormous, genuine value these archives provide. Sometimes it does not, and no amount of methodological sophistication converts an unanswerable comparison into an answerable one. The analysis will run either way. The distinguishing information sits in a two-by-two table that takes one line of code to produce.
The failure is rarely that people look at that table and misread it. It is that the pipeline never required them to make it.
Zetobit builds and validates analysis pipelines for genomics, including public-data reuse workflows with design auditing, confounding diagnostics, and batch handling matched to the structure actually present. If you are planning a TCGA or GEO reanalysis, or reconciling results that changed after batch correction, we're happy to talk.
References
- Leek JT, Scharpf RB, Bravo HC, et al. Tackling the widespread and critical impact of batch effects in high-throughput data. Nature Reviews Genetics 11(10):733–739 (2010). doi:10.1038/nrg2825.
- Nygaard V, Rødland EA, Hovig E. Methods that remove batch effects while retaining group differences may lead to exaggerated confidence in downstream analyses. Biostatistics 17(1):29–39 (2016). doi:10.1093/biostatistics/kxv027.
- Zhang Y, Parmigiani G, Johnson WE, et al. Overcoming the impacts of two-step batch effect correction on gene expression estimation and inference. Biostatistics 24(3):635–652 (2023). doi:10.1093/biostatistics/kxac034.
- Buhule OD, et al. Simulating ComBat: how batch correction can lead to the systematic introduction of false positive results in DNA methylation microarray studies. BMC Bioinformatics 21:271 (2020). doi:10.1186/s12859-020-03559-6.
- Sharma S, et al. Substantial batch effects in TCGA exome sequences undermine pan-cancer analysis of germline variants. BMC Cancer 19:774 (2019). doi:10.1186/s12885-019-5994-5.
- Buckley AR, Standish KA, Bhutani K, et al. Pan-cancer analysis reveals technical artifacts in TCGA germline variant calls. BMC Genomics 18:458 (2017). doi:10.1186/s12864-017-3770-y.
- Molania R, Foroutan M, Gagnon-Bartsch JA, et al. Removing unwanted variation from large-scale RNA sequencing data with PRPS. Nature Biotechnology 41:82–95 (2023). doi:10.1038/s41587-022-01440-w.
- Diener C, et al. On the impact of batch effect correction in TCGA isomiR expression data. NAR Cancer 3(1):zcab007 (2021). doi:10.1093/narcan/zcab007.