Blog · Clinical machine learning

How to prevent data leakage in clinical AI

Leakage is one of the fastest ways an EHR risk prediction study looks strong in a notebook and fails under honest temporal evaluation. This guide is for research and teaching—not bedside care.

What “leakage-safe AI” means for EHR data

In longitudinal electronic health records, features must respect an index time (when the prediction is made) and a horizon (what you are predicting afterward). Using labs, notes, or codes from after the index time inflates AUROC and destroys external validity.

Practical checklist

Define index and horizon in a task spec before feature engineering.
Use patient-level or temporal splits — never random row splits on longitudinal tables.
Run a leakage audit that checks post-index features and split integrity.
Report calibration (Brier, ECE), not only discrimination.
Document limitations — synthetic demos ≠ clinical performance.

How the EHR Risk Framework helps

The open-source EHR Risk Framework is a workbench for leakage-aware, calibrated, explainable EHR risk modeling: task YAML, Docker UI, leakage-audit jobs, isotonic calibration, and SHAP. Start with the quickstart or the risk model tutorial.

Next: Data guide · Temporal diagrams · Limits