The Optimal Cutpoint Problem: Why a High-vs-Low Survival Curve Can Be Manufactured From Noise

Zetobit.

Bioinformatics Insight Series

The Optimal Cutpoint Problem: Why a High-vs-Low Survival Curve Can Be Manufactured From Noise

Splitting patients into high and low expression requires choosing a threshold. Searching every possible threshold for the one that yields the smallest p-value produces a false-positive rate around 40% rather than the nominal 5% — and the resulting Kaplan-Meier plot is indistinguishable from a real finding.

By Kanna Nandakumar, PhD · Zetobit

A two-curve Kaplan-Meier plot is one of the most legible figures in translational genomics. High expression above, low expression below, curves diverging over time, a log-rank p-value in the corner. It communicates a clinical claim in a single image, and it takes about ten seconds to read.

It also requires a decision that the data does not supply. Gene expression is continuous; the log-rank test compares groups. Somewhere between the expression matrix and the figure, someone chose a number and called everything above it "high."

How that number was chosen determines whether the figure means anything.

The minimum p-value method, and what it costs

The problematic approach is well defined and has been named in the statistical literature for three decades. Altman describes it precisely: each distinct observed value of the marker is taken in turn as a cutpoint, two groups are created, a log-rank test is performed for each grouping, and the cutpoint with the lowest p-value is called "optimal" — after which Kaplan-Meier curves are constructed with that cutpoint and the p-value reported.

He is equally precise about the damage, listing four consequences. Because of multiple testing the false-positive rate is around 40% rather than the nominal 5%. The p-value is far too small — p = 0.002 corresponds to a genuine p = 0.05. The value of the cutpoint has no clinical meaning. And the analysis gives no information about the shape of the relation between marker level and prognosis.

The first two of those deserve to be read together. A false-positive rate near 40% means that if you apply this procedure to genes with no relationship to survival, roughly two in five will nonetheless yield a "significant" split. And the correction factor is severe: a reported p = 0.002 from a cutpoint search is not strong evidence — it is the threshold at which you would begin to have ordinary, unremarkable evidence.

Altman and Royston's later verdict is unusually blunt for a statistics paper: use of the so-called "optimal" cutpoint runs a high risk of a spuriously significant result, the difference between groups will be overestimated perhaps considerably, the confidence interval will be too narrow, and this strategy should never be used.

A reported p = 0.002 from a cutpoint search is not strong evidence. It is roughly where ordinary evidence begins.

The cake-and-herpes demonstration

A 2025 paper in BMC Medical Research Methodology quantified how easily this manufactures findings, and its worked example is memorably absurd.

Using NHANES data on fasting plasma glucose and herpes simplex type 2 seropositivity in 1,484 adults, the authors report that a full logistic regression on the complete continuous data found no evidence of association (p = 0.545). Testing every possible cutoff and selecting the one that maximized group difference produced a threshold at roughly 4.3 mmol/L, at which herpes prevalence differed significantly between "low" and "high" glucose groups with an apparent hazard ratio of 2.04.

The groups were wildly unbalanced — 60 versus 1,424 — and the authors note that the sample sizes went unstated, preventing awkward questions about why the groups are so uneven. Their broader point is that a researcher could, on this basis, editorialize a causal story: cake causes herpes.

The simulation results generalize it. For a sample of just n = 100, a false positive rate of approximately 38% is achievable, rising above 66% if low-count scenarios are not excluded; at n = 1,000 and n = 10,000 the achievable rates exceed 66% and 83% respectively even with low-count exclusion. Larger cohorts make threshold-hunting more effective, not less — which is worth sitting with, given that public genomic cohorts keep getting bigger.

One detail from that example is directly transferable as a review technique: using any other clinically meaningful glucose threshold — 3.9, 5.6, or 6.9 mmol/L — destroyed the effect entirely, with p-values of approximately 0.51, 0.23 and 0.14. A finding that exists at exactly one threshold and nowhere near it is not a finding.

Figure 1. A cutpoint search runs a log-rank test at every candidate threshold and reports the best one. The published Kaplan-Meier plot shows a single split with a single p-value and contains no record of how many splits were examined to find it. Because the tests are performed on the same patients rather than fresh data, they are correlated, which is why the inflation is not a simple Bonferroni correction — and why dedicated adjustment methods exist.

The double-dipping structure

Readers of this series will recognize the shape from single-cell analysis: cluster the cells, then test for genes that differ between clusters, using the same data for both steps. The clusters were defined by expression differences, so testing those differences against the same data produces circular significance.

Cutpoint optimization is that failure in survival form. The threshold is chosen to maximize survival separation, and then the significance of that separation is tested on the same patients that determined the threshold. The p-value answers the question "how surprising is this split?" while ignoring that the split was selected precisely because it was the most surprising one available.

The difference from ordinary multiple testing matters technically. When cuts are made repeatedly on the same unchanged dataset, the tests are not independent — the 2025 authors note that this non-independence makes the false-positive calculation a distinctly non-trivial undertaking rather than the simple 1 − (1 − α)^m of independent tests. That is why the remedy is not a Bonferroni correction but the purpose-built adjustments developed for maximally selected statistics.

The median split is not a fix

The obvious escape is to use a fixed, data-independent threshold — usually the median. This removes the selection problem entirely and is genuinely better. It is not free.

Dichotomizing at the median has been described as equivalent to discarding roughly a third of the data from a statistical power perspective. A starker illustration from a clinical trial: sample size could be reduced from 800 to 88 — nearly an order of magnitude — by switching from a dichotomous to a continuous analysis of the same outcome.

Two further problems come with any dichotomization, optimal or not. It assumes a step function: that risk is constant below the threshold, constant above it, and jumps discontinuously at one value. Real biological relationships are generally smooth, and the minimum p-value literature notes explicitly that a stepwise functional relationship is biologically implausible. And it discards the shape of the relationship — a marker whose risk rises steeply at the top decile and one that rises linearly across the range can produce identical high/low curves.

So the honest ordering is: continuous modelling is best, a pre-specified fixed threshold is acceptable with a stated power cost, and a data-derived optimal cutpoint reported without adjustment is not defensible.

Table 1. Approaches to a continuous marker in survival analysis, and what each costs. Only the last is affected by the selection problem; the middle rows trade power for interpretability.

Approach	What it assumes	Cost
Cox model, continuous	Log-linear effect of the marker on hazard	No group plot; assumption should be checked
Splines / fractional polynomials	Smooth but flexible relationship	More parameters; harder to present clinically
Pre-specified fixed cutpoint	A step function at a threshold chosen in advance	Power loss; threshold must be justified externally
Median split	A step function at the cohort median	~⅓ of statistical information; cohort-dependent
Searched "optimal" cutpoint	A step function at the most favourable threshold	False positives ~40%; effect and CI both biased

Why genomics is especially exposed

Three features of expression-and-survival work amplify all of this.

The gene list is long. A cutpoint search is already a multiple-testing problem within one gene. Running it across a transcriptome multiplies an inflated per-gene error rate by tens of thousands of genes. The two layers compound, and only one of them is usually corrected.

The cutpoint doesn't transfer. Optimal cutpoints are dataset-dependent, so a threshold derived in one cohort is unlikely to be the right one in another. This is why the common workflow — derive an optimal cutpoint in TCGA, apply that exact numeric threshold to a GEO validation cohort — is weaker than it appears. Expression values are not on a shared absolute scale across platforms and normalizations, so the number may not even mean the same thing in the second cohort.

The barrier to trying is zero. A single R function will scan every cutpoint and return the best one. Nothing in the output signals how many tests were performed, and the resulting figure looks identical to one produced by a pre-specified threshold.

What to do instead

Model the marker continuously first

Fit a Cox model on the continuous value and report the hazard ratio per unit or per standard deviation. This is the primary analysis. If the continuous model shows nothing, a dichotomized version showing something is a warning sign, not a discovery — that is precisely the cake-and-herpes pattern.

If you must split, pre-specify the threshold

Median, tertile, or a clinically established cutoff decided before looking at outcomes. Say in the methods that it was pre-specified. A Kaplan-Meier plot with a pre-specified split is a legitimate visualization of a continuous result; the same plot with a searched split is a different object entirely.

If you search, adjust — and say that you searched

Methods exist for exactly this. Maximally selected rank statistics account for the selection, and p-value adjustment formulae are available for the minimum p-value approach. What is not acceptable is searching and reporting the raw p-value. The unadjusted figure is not conservative; it is wrong by roughly an order of magnitude.

Check the neighbourhood of the threshold

Recompute the p-value at nearby cutpoints. A real effect degrades gradually as the threshold moves; an artifact vanishes. This is a one-line robustness check, it needs no new data, and it is the single most useful thing a reviewer can ask for.

Report the group sizes

Searched thresholds tend to produce lopsided groups, because extreme splits maximize apparent separation. A 60-versus-1,424 split is visible immediately in a group-size table and invisible in a Kaplan-Meier plot.

Validate the threshold, not just the gene

If a cutpoint is going to be used clinically, it needs to be fixed in a training set and evaluated in a genuinely independent one — the sample-splitting logic used routinely in machine learning. A threshold and a p-value derived from the same patients have not been validated by anything.

The short version

Choosing the threshold that gives the smallest p-value is selecting on the outcome and then testing against it. The result is a false-positive rate near 40%, an overestimated effect size, and a confidence interval that is too narrow — packaged in a figure that looks exactly like a real result. Continuous modelling avoids the problem entirely; a pre-specified split is an acceptable compromise; a searched split reported without adjustment is not.

A figure that hides its own construction

Most of the failure modes in this series involve an inference that presents as an observation. This one is narrower and, in a way, more troubling: the analysis is not merely uncertain, it has been optimized to look convincing, and the optimization leaves no trace in the output.

A Kaplan-Meier plot from a pre-specified median split and one from an exhaustive threshold search are visually identical. Same axes, same curve shapes, same p-value in the corner. Nothing in the figure records how many splits were tried. The reader cannot distinguish a result from an artifact without asking a question the figure does not prompt.

The statisticians settled this in 1994. The tooling that makes it effortless arrived later, and the practice is now common enough that "high versus low expression" reads as a standard method rather than a decision requiring justification. It is a decision — and where it was made after looking at the survival data, the p-value beside the curves is not measuring what it appears to measure.

Zetobit builds and validates analysis pipelines for clinical and translational genomics, including survival workflows with continuous modelling, pre-specified thresholds, and selection-aware adjustment where cutpoints are genuinely needed. If you are developing a prognostic marker or reviewing survival analyses across cohorts, we're happy to talk.

References

1. Altman DG, Lausen B, Sauerbrei W, Schumacher M. Dangers of using "optimal" cutpoints in the evaluation of prognostic factors. Journal of the National Cancer Institute 86(11):829–835 (1994). doi:10.1093/jnci/86.11.829.
2. Altman DG. Suboptimal analysis using "optimal" cutpoints. British Journal of Cancer 78(4):556–557 (1998). doi:10.1038/bjc.1998.537.
3. Grimes DR, Heathers J. "Cake causes herpes?" — promiscuous dichotomisation induces false positives. BMC Medical Research Methodology 25:255 (2025). doi:10.1186/s12874-025-02712-0.
4. Altman DG, Royston P. The cost of dichotomising continuous variables. BMJ 332(7549):1080 (2006). doi:10.1136/bmj.332.7549.1080.
5. Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Statistics in Medicine 25(1):127–141 (2006). doi:10.1002/sim.2331.
6. MacCallum RC, Zhang S, Preacher KJ, Rucker DD. On the practice of dichotomization of quantitative variables. Psychological Methods 7(1):19–40 (2002). doi:10.1037/1082-989X.7.1.19.
7. Lausen B, Schumacher M. Maximally selected rank statistics. Biometrics 48(1):73–85 (1992). doi:10.2307/2532740.
8. Naggara O, Raymond J, Guilbert F, et al. Analysis by categorizing or dichotomizing continuous variables is inadvisable: an example from the natural history of unruptured aneurysms. American Journal of Neuroradiology 32(3):437–440 (2011). doi:10.3174/ajnr.A2425.