6. Case study: small cohorts
2026-10-02
1 Status note
This is a synthetic-data walkthrough demonstrating how the
events_per_parameter guardrail and fold-level reporting expose unstable
signatures. It is a vignette, not a study. The narrative follows the
three-tier structure used across the case studies: a naive analysis
first, then rnaSentry standing guard, then the counterfactual.
2 Setup
We build a 30-sample cohort with 23 events and a real (if weak) survival signal driven by five genes, then ask for a 5-gene signature. This is 4.6 events per parameter – below the guardrail of 5.
library(rnaSentry)
library(SummarizedExperiment)
set.seed(30)
n_sm <- 30
counts <- matrix(rpois(60 * n_sm, lambda = 500), nrow = 60, ncol = n_sm,
dimnames = list(paste0("gene", 1:60), paste0("S", 1:n_sm)))
sig_expr <- colMeans(counts[1:5, , drop = FALSE])
risk <- scale(sig_expr)[, 1] * 0.5
event_time <- rexp(n_sm, rate = 0.05 * exp(0.9 * risk))
censor_time <- rexp(n_sm, rate = 0.02)
cd <- S4Vectors::DataFrame(time = pmin(event_time, censor_time),
event = as.integer(event_time < censor_time),
row.names = colnames(counts))
se <- SummarizedExperiment(assays = list(counts = counts), colData = cd)
sig <- build_signature(se, "time", "event", top_n = 5,
repeats = 5, folds = 5, seed = 30)
#> Warning: Only 23 event(s) for a 5-gene signature (4.6 events per parameter),
#> below 'min_events_per_parameter' = 5. Cross-validated concordance and hazard
#> ratios are unstable at this event count; consider fewer genes or a larger
#> cohort.3 Tier 1: the naive analysis
A standard analysis reports the single number the summary returns and stops there:
sig$cv_summary
#> mean sd
#> 0.7430831 0.1411627The mean cross-validated concordance is 0.74 – which a careless reader would call “decent.” A naive manuscript would report “cross-validated C = 0.74” as evidence that the signature is prognostic, with no sense of how much that single mean is hiding.
4 Tier 2: rnaSentry standing guard
4.1 The guardrail fires
sig$flags
#> check severity
#> 1 events_per_parameter warning
#> detail
#> 1 Only 23 event(s) for a 5-gene signature (4.6 events per parameter), below 'min_events_per_parameter' = 5. Cross-validated concordance and hazard ratios are unstable at this event count; consider fewer genes or a larger cohort.
#> stage
#> 1 build_signatureThe events_per_parameter warning fires immediately – 23 events for 5 genes
is 4.6 per parameter, below the guardrail of 5 – and is recorded in the
stage’s flag ledger so it propagates into the rendered report.
4.2 The mean hides the spread
sig$cv_results
#> repeat_id fold c_index
#> 1 1 1 0.5000000
#> 2 2 1 0.6000000
#> 3 3 1 0.5789474
#> 4 4 1 0.7500000
#> 5 5 1 0.8750000
#> 6 1 2 0.7368421
#> 7 2 2 0.7647059
#> 8 3 2 0.8461538
#> 9 4 2 0.6923077
#> 10 5 2 0.6666667
#> 11 1 3 0.7142857
#> 12 2 3 0.6666667
#> 13 3 3 0.7692308
#> 14 4 3 0.9285714
#> 15 5 3 0.9166667
#> 16 1 4 0.7500000
#> 17 2 4 0.5000000
#> 18 3 4 0.8888889
#> 19 4 4 0.5000000
#> 20 5 4 0.8000000
#> 21 1 5 0.8750000
#> 22 2 5 0.8571429
#> 23 3 5 1.0000000
#> 24 4 5 0.8000000
#> 25 5 5 0.6000000The per-fold concordances range from 0.50 to 1.00 across repeats and folds. With only a handful of held-out samples per fold, single-fold C estimates are nearly coin flips, and the standard deviation (0.14) is enormous relative to the mean. A signature like this is unstable and should be treated as hypothesis-generating only.
set.seed(30)
cv <- sig$cv_results
stripchart(cv$c_index, method = "jitter", jitter = 0.12, pch = 19, cex = 1.1,
ylim = c(0, 1.05), ylab = "Cross-validated C-index (per fold)",
main = sprintf("Per-fold C-index: mean %.2f, sd %.2f",
mean(cv$c_index), stats::sd(cv$c_index)))
abline(v = mean(cv$c_index), lty = 2, col = "grey40")
Figure 1: Per-fold cross-validated concordance across repeats and folds (one point per fold)
The vertical dashed line marks the mean; the spread from 0.50 to 1.00 is the instability the mean hides.
5 Tier 3: the counterfactual
| Naive analysis (Tier 1) | rnaSentry (Tier 2) | |
|---|---|---|
| Reported statistic | mean CV C = 0.74, quoted alone | mean + fold-level distribution (range 0.50-1.00, sd 0.14) |
| Events per parameter | not computed | 4.6 < 5, events_per_parameter warning in the ledger |
| Scientific status | “a decent prognostic signature” | “hypothesis-generating only; unstable” |
What a guarded analysis can claim: the exact same cross-validated
concordance that a naive analysis would headline as 0.74 is shown by
build_signature() to be an average over near-coin-flip folds – and the
guardrail warns rather than silently proceeding. This is the documented
“small-cohort trap,” quantified in our simulation analysis
(validation/simulate_study.R): with fewer than ~5 events per signature
gene, neither the concordance nor the hazard ratios are trustworthy, and a
mean C-index hides the instability that the fold-level table exposes.
6 Reading
The takeaway is exactly the documented “small-cohort trap”: with fewer than
~5 events per signature gene, neither the concordance nor the hazard ratios
are trustworthy. This is why build_signature() reports the per-fold
distribution, not just the mean, and why the guardrail warns rather than
silently proceeding.
7 Session info
sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 24.04.5 LTS
#>
#> Matrix products: default
#> BLAS: /home/biocbuild/bbs-3.24-bioc/R/lib/libRblas.so
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0 LAPACK version 3.12.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_GB LC_COLLATE=C
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: America/New_York
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats4 stats graphics grDevices utils datasets methods
#> [8] base
#>
#> other attached packages:
#> [1] survival_3.8-12 SummarizedExperiment_1.43.0
#> [3] Biobase_2.73.2 GenomicRanges_1.65.4
#> [5] Seqinfo_1.3.2 IRanges_2.47.5
#> [7] S4Vectors_0.51.10 BiocGenerics_0.59.12
#> [9] generics_0.1.4 MatrixGenerics_1.25.0
#> [11] matrixStats_1.5.0 rnaSentry_0.99.19
#> [13] BiocStyle_2.41.0
#>
#> loaded via a namespace (and not attached):
#> [1] gtable_0.3.6 ggplot2_4.0.3 xfun_0.61
#> [4] bslib_0.12.0 httr2_1.3.0 lattice_0.23-1
#> [7] tzdb_0.5.0 vctrs_0.7.3 tools_4.6.1
#> [10] curl_8.0.0 tibble_3.3.1 pkgconfig_2.0.3
#> [13] R.oo_1.27.1 Matrix_1.7-6 data.table_1.18.6.1
#> [16] RColorBrewer_1.1-3 rentrez_1.2.4 S7_0.2.2
#> [19] lifecycle_1.0.5 farver_2.1.2 compiler_4.6.1
#> [22] tinytex_0.61 statmod_1.5.2 htmltools_0.5.9
#> [25] sass_0.4.10 yaml_2.3.12 pillar_1.11.1
#> [28] jquerylib_0.1.4 tidyr_1.3.2 R.utils_2.13.0
#> [31] DelayedArray_0.39.8 cachem_1.1.0 limma_3.99.0
#> [34] magick_2.9.1 abind_1.4-8 tidyselect_1.2.1
#> [37] digest_0.6.39 dplyr_1.2.1 purrr_1.2.2
#> [40] bookdown_0.48 labeling_0.4.3 splines_4.6.1
#> [43] fastmap_1.2.0 grid_4.6.1 GEOquery_2.81.28
#> [46] cli_3.6.6 SparseArray_1.13.4 magrittr_2.0.5
#> [49] S4Arrays_1.13.2 dichromat_2.0-1 XML_3.99-0.25
#> [52] readr_2.2.0 withr_3.0.3 scales_1.4.0
#> [55] rmarkdown_2.32 XVector_0.53.0 otel_0.2.0
#> [58] R.methodsS3_1.8.2 hms_1.1.4 evaluate_1.0.5
#> [61] knitr_1.52 rlang_1.3.0 Rcpp_1.1.2
#> [64] glue_1.8.1 BiocManager_1.30.27 xml2_1.6.0
#> [67] jsonlite_2.0.0 R6_2.6.1