EHR Risk Blog · Leakage & methods
Why many same-admission AI models look strong — until you check for label leakage
A 2025 JAMA Network Open study quantified a pervasive methods flaw: using discharge ICD codes as features for same-admission outcomes. Here is what the paper actually found, why it matters for EHR risk research, and how leakage-aware tooling helps labs and courses.
What the study measured (precision matters)
The headline number is often shortened to “40% of clinical AI models.” The paper is more specific: among 92 MIMIC-based studies that predicted same-admission outcomes, 37 (40.2%) included ICD diagnostic codes as input features—even though MIMIC documentation states ICD codes are derived after discharge and are not available for real-time prediction during the stay.
Source: Ramadan et al., JAMA Network Open (2025) · Full text · PMC.
Why this is “label leakage,” not clever feature engineering
ICD codes entered or finalized after the clinical event (or after discharge) can encode information about the outcome itself— palliative-care encounters, brain death, cardiac arrest coding, and related labels. Models trained on those codes can report very high discrimination (the paper’s ICD-based mortality models reached AUROCs around 0.97) while being unusable for prospective prediction, because those codes would not be available at the decision time.
That is the research crisis: notebooks and papers look excellent; honest index-time evaluation collapses.
What “fix it” means for methods work
- Define index time and horizon before feature engineering—what can the model see when the prediction is made?
- Ban post-index / post-discharge fields from feature matrices for same-admission targets (ICD finalized at discharge is a classic trap).
- Prefer temporal or patient-level splits over random row splits on longitudinal EHR.
- Run a leakage audit and report calibration (Brier / ECE), not only AUROC.
- Document limitations—synthetic demos ≠ clinical performance.
Related: Prevent data leakage guide · Temporal diagrams
How the EHR Risk Framework helps (research / teaching)
The open-source workbench is built around leakage-aware tasks (index + horizon), dataset health, train jobs, and an explicit leakage-audit path—so students and labs practice honest evaluation instead of chasing inflated AUROCs. It is not a clinical product and does not claim to “fix” published literature; it gives a shared loop: ingest → health → train → audit → calibrate → explain → serve (research API).
- Task YAML and Train UI force an explicit prediction setup
- Leakage audit job against trained artifacts
- Calibration metrics and optional isotonic calibration
- SHAP jobs for inspectable drivers (when generated)
Sources
- Ramadan B, et al. Diagnostic Codes in AI Prediction Models and Label Leakage of Same-Admission Clinical Outcomes. JAMA Netw Open. 2025. doi:10.1001/jamanetworkopen.2025.50454
Try the workbench
How it works A–Z Live demo Quickstart GitHub
Free demo server — it may be slow. Check it with a small amount of data. For larger workloads or freer experimentation, run locally or on your own server. Open live demo
Feedback: support@larucare.com · Cite: DOI & CITATION.cff