EHR Risk Blog · Calibration
Why “accurate” risk models still mislead — the case for calibration
A model can rank patients well (AUROC) and still produce unusable probabilities. Calibration—matching predicted risk to observed frequency—is essential for research reporting and for any future decision-threshold thinking.
What calibration means
A probability is well calibrated when events happen about as often as the model claims (e.g., among patients with predicted risk ≈ 0.2, about 20% experience the outcome). Poor calibration yields overconfident or underconfident scores even when ranking looks strong.
Metrics the workbench emphasizes
- Brier score — mean squared error of probabilistic predictions (lower is better)
- Expected Calibration Error (ECE) — bin-wise gap between confidence and accuracy
- Reliability diagrams / calibration plots — visual check on hold-out
Literature across domains shows post-hoc methods such as isotonic regression and Platt scaling can improve probability quality for many classifiers. Exact numeric gains are dataset- and model-specific— always cite the paper and your own hold-out when you quote a percentage improvement.
How the EHR Risk Framework helps
- Reports Brier / ECE in evaluation outputs
- Optional isotonic calibration in Train (UI / CLI)
- Calibration figures in results packs when generated
Deep dive: Fine-tuning playbook · How it works A–Z
Sources
- Peer-reviewed calibration / forecasting literature (e.g., Cureus and related reviews)—cite the specific article when quoting numeric before/after tables.
Try the workbench
How it works A–Z Live demo Quickstart GitHub
Free demo server — it may be slow. Check it with a small amount of data. For larger workloads or freer experimentation, run locally or on your own server. Open live demo
Feedback: support@larucare.com · Cite: DOI & CITATION.cff