Cell-Type Annotation in Single-Cell RNA-Seq: Why a Confident Label Can Be Wrong, and Reference Mapping Can't Tell You It Is

Zetobit · Bioinformatics Insight Series

Cell-Type Annotation in Single-Cell RNA-Seq: Why a Confident Label Can Be Wrong, and Reference Mapping Can't Tell You It Is

Clustering gives you groups of cells; annotation gives them names. That naming step feels like the objective payoff of the whole experiment — but a label transferred from a reference is only as good as the reference, and a classifier will confidently call a cell something it has never actually seen.

By Kanna Nandakumar, PhD Zetobit LLC

Once single-cell data is clustered, each cluster has to be named — assigned a cell-type identity that every downstream conclusion then rests on. This annotation step is where the analysis becomes legible: "cluster 7" becomes "regulatory T cell," and the biology finally reads like biology. Three broad strategies dominate. Marker-based methods score each cluster against curated cell-type marker genes; correlation/reference methods match query cells to an annotated reference by expression similarity; and supervised label transfer trains a classifier on a labeled reference and applies it to new cells.¹ Tools like Azimuth, SingleR, and CellAssign have made this fast and largely automated.

The convenience hides a structural fragility. Every one of these approaches assigns a label by relating the query to prior knowledge — a marker list, a reference atlas, a training set — and inherits the limitations of that prior knowledge without flagging them. The label that comes out looks authoritative. Whether it is correct is a separate question the method often cannot answer, and sometimes actively obscures.

The forced-label problem

The sharpest failure is structural: a standard classifier assigns every query cell to one of the labels it was trained on. If a cell type in your sample is not represented in the reference — a novel state, a disease-specific population, a cell the atlas simply didn't include — the classifier does not say "unknown." It assigns the nearest known label, confidently.² The genuinely new biology is precisely what gets erased, relabeled as whatever it most resembles among things already catalogued.

This is why the ability to output "unassigned" or "unknown," and to perform explicit novelty detection, is a critical feature rather than a nicety. Benchmarks have highlighted the value of a rejection option — a classifier that declines to label low-confidence cells — and methods like scArches/treeArches exist specifically to flag query cells that don't match known reference states, catching real biology (new cell states, treatment effects) rather than forcing it into the nearest known label.³ Reference mapping without a rejection or novelty mechanism is a system that cannot, by construction, discover anything the reference didn't already contain.

A query population absent from the reference has nowhere valid to go. A hard classifier forces it to the nearest trained label; only a method with a rejection option or novelty detection can flag it as unknown and preserve it for investigation instead of erasing it.

The reference is not ground truth

Label transfer treats the reference as authoritative, but references carry their own errors — and those errors propagate directly into the query. Reference-based methods can show poor agreement with original annotations even when a reference is queried against itself, and differences in cell-type composition between query and reference can produce misaligned labels.⁴ A recent benchmark made the risk concrete: adding labels does not always help. Methods that lean heavily on reference labels excelled with perfect or clean annotations but degraded rapidly under structured errors — missing whole batches, or label swaps at lineage boundaries — sometimes falling below simpler unsupervised baselines.⁵ Whatever is mislabeled or missing in the atlas becomes mislabeled or missing in your results.

Platform and batch differences quietly corrupt mapping. When query and reference come from different scRNA-seq chemistries, protocols, or labs, the technical shift between them can dominate the biological signal the mapping relies on — degrading label accuracy even when the underlying cell types are identical.⁶ A label transferred across a platform boundary is doing double duty: telling you the cell type and silently absorbing a batch effect. The two are hard to separate after the fact.

Closely related types are where it breaks

Annotation is easy where biology is easy. Distinguishing broadly different populations — a B cell from a T cell — works well because their transcriptomes share few overlapping transcripts.⁷ The difficulty scales with similarity: closely related subtypes often sit within the same cluster and share most of their expression, so both clustering and label transfer struggle to separate them, and batch effects blur them together further.⁴ This matters because the interesting biology is usually in the fine distinctions — naive versus memory, one activation state versus another — exactly where annotation confidence is least warranted and error is most likely. A coarse label ("T cell") may be safe while the fine label the analysis actually needs ("exhausted CD8 T cell") is a guess dressed as a call.

Even LLM-based annotation inherits the same limits

Newer approaches use large language models to annotate from marker-gene lists, and they perform respectably — GPT-4-based annotation has shown full or partial agreement with manual labels in a majority of cell types across many tissues.⁸ But the same evaluation shows the pattern: agreement is high for well-characterized types with distinct marker sets and lower for cells lacking distinct signatures, and its knowledge is bounded by training data.⁸ Automation changes the source of the prior knowledge, not the fundamental dependence on it. A model — statistical or linguistic — still cannot reliably name a cell type it has no good prior for, and still tends to return an answer rather than an admission of uncertainty.

What a defensible annotation requires

Turning a cluster into a trustworthy cell-type label is a verification problem, not a lookup:

• Use a rejection option and novelty detection. Prefer methods that can output "unknown," threshold prediction confidence, and route low-confidence or out-of-distribution cells to review rather than forcing a label.³
• Two-step annotation. Follow automated primary labeling with expert manual review of marker expression per population; the combination is more accurate than either alone.⁷
• Check the reference's fit. Confirm the reference covers the tissue, states, and disease context of your sample, and evaluate cross-platform transfer explicitly rather than assuming it holds.⁶
• Match label granularity to evidence. Report coarse types confidently and fine subtypes only where markers genuinely support them; don't over-resolve a label the data can't sustain.⁴
• Treat surprising labels as hypotheses. A label that implies a novel or unexpected population is a prompt to validate — independent markers, orthogonal data, a held-out sample — not a conclusion.⁵

As with the other blind spots in this series, the honest analysis states what stands behind each name: the reference and its provenance, the annotation method and confidence thresholds, whether a rejection option was used, and whether close subtypes were manually verified. A cell-type label reported without that scaffolding looks like knowledge but may be the nearest guess the reference allowed.

The takeaway

Annotation is the step where single-cell data acquires meaning — and the step most easily mistaken for objective measurement. A transferred label is a claim about similarity to prior knowledge, and it inherits everything wrong with that prior: the reference's errors, its missing cell types, its platform, its inability to say "I don't know." The classifier's confidence is not evidence of correctness; it is often highest exactly where the biology is subtlest. Naming cells well means treating every label as a hypothesis about identity — checked against a reference that fits, a method that can abstain, and, where it matters, a human who knows the markers.

References

1. Automated methods for cell type annotation on scRNA-seq data (three strategies: marker databases, correlation-to-reference, supervised classification; rejection/"unassigned" as key feature). Comput Struct Biotechnol J. 2021. sciencedirect.com
2. Automated methods for cell type annotation — ability to classify cells as "unassigned"/"unknown" when identity not represented in the reference. PMC. 2021. PMC7873570
3. Benchmarking cell atlases for label transfer (rejection thresholds; scANVI low-confidence → "Unknown"; scArches/treeArches novelty detection to avoid forced assignment). 2026. evo-byte.com
4. Automated cell annotation using unique marker gene sets (poor agreement even querying reference against itself; composition mismatch; closely related subtypes within one cluster; batch effects). bioRxiv. 2024. biorxiv.org
5. A benchmark of semi-supervised scRNA-seq integration methods (adding labels doesn't always help; label-reliant methods degrade under structured errors/label swaps, falling below unsupervised baselines). PLOS Comput Biol. 2026. journals.plos.org
6. Automated cell annotation — platform/protocol differences between query and reference degrade mapping accuracy. bioRxiv. 2024. biorxiv.org
7. Current annotation strategies for T-cell phenotyping (label transfer depends on reference-annotation quality; two-step automated + expert manual review recommended; easy for B vs T, hard for subtypes). Front Immunol. 2023. PMC10768068
8. Hou W, Ji Z. Assessing GPT-4 for cell type annotation (full/partial match with manual labels in >75% of cell types in most studies; weaker where distinct marker sets are lacking; training-cutoff bound). Nat Methods. 2024. nature.com

Zetobit LLC · Bioinformatics Insight Series · Lexington, KY · zetobit.com