Fine-tuning playbook — EHR Risk Framework

1. What you can tune

Knob	Where	Effect
Dataset / mapping	Datasets page, map-import API	Better inputs → better ceiling
Task YAML	`tasks/*.yaml`, Train presets	Horizon, windows, index, label column
Windows	Config / Train / `windows_days`	Short vs long history signal (`w7d`…)
Split	Patient vs temporal	How hold-out generalizes
Model family	logreg, RF, XGBoost, LightGBM	Bias/variance trade-off
Calibrate	checkbox / `--calibrate`	Better probability quality (Brier/ECE)
Compare set	Config compare_models	Pick best of several families
Promote	train promote / compare promote_best	Active `model.pkl` for Predict

2. The iteration loop (diagram)

A. Define prediction taskHorizon H · index strategy · windows

B. Data health OK?Fix blockers / warnings on Datasets

C. Train baseline (logreg)Record AUROC · PR-AUC · Brier · ECE

D. Trust jobsLeakage audit + SHAP (+ fairness optional)

Metrics honest?No → revisit task / data / calibrate

Leakage clean?No → fix index / features / split

E. Compare familiesMulti-model · calibrate on/off · light HPO optional

F. Promote best → PredictActive model.pkl · document limitations

Iterate until metrics and audits look honest — then stop and write limits.

3. Workbench steps (recommended)

Datasets — select or upload CSV → Run health → Continue to train only when ready.
Config — set windows (e.g. 7, 30, 180), default model, compare models, optional calibrate.
Train — apply a task preset (diabetes / …) → Train once as baseline.
Train → Leakage audit — if temporal issues appear, fix index/horizon before chasing AUROC.
Train → Compare — multi-model ranking; promote best when satisfied.
Results / Analytics — metrics, importance, cohort filters (age/label/patient).
Results → Generate SHAP — explanations for demos and debugging.
Predict — schema form; keep “reuse as next input” while exploring what-if values.

Train page — presets, train, compare, leakage

4. CLI / Make recipes

# Baseline
make train

# Stronger evaluation habits
make train-patient          # patient-grouped split
make train-temporal         # time-based hold-out

# Trust
make leak-audit
make shap

# Explicit CLI with calibration + multi-window
PYTHONPATH=. python -m training.train \
  --format longitudinal \
  --data data/raw/paper_synthetic_cohort.csv \
  --model xgboost \
  --windows 7,30,180 \
  --split-by-patient \
  --calibrate

# Multi-model compare via API (stack running)
curl -sS -X POST http://127.0.0.1:8000/v1/jobs/compare \
  -H 'Content-Type: application/json' \
  -d '{
    "data_path":"data/demo/ehr_data.csv",
    "data_format":"longitudinal",
    "split_by_patient":true,
    "windows_days":[7,30,180],
    "promote_best":true
  }' | jq .

5. Knob reference (practical advice)

Windows

Too short → miss chronic signal. Too long → noise / leakage risk if index is wrong. Start with [7,30,180].

Patient vs temporal split

Patient split avoids same-person leakage across folds. Temporal split tests future performance — use when timestamps are reliable.

Model family

LogReg = interpretable baseline. Trees / XGBoost = more capacity; always pair with SHAP + calibration checks.

Calibrate

Turn on when Brier/ECE are poor even if AUROC looks fine. Isotonic wraps the fitted estimator.

6. Compare → promote (data flow)

POST /v1/jobs/compareModels from Config compare_models or request body

For each modelTrain hold-out → metrics (same split / task)

Rank by ROC-AUCResearch default ranking (report PR / Brier too)

Write comparisonreports/model_comparison.json

If promote_bestWinner → active model.pkl

Predict · schema · SHAPUse the promoted artifact · named runs under reports/runs/<id>/

Workbench: Train → Compare, or Results experiment browser → promote. CLI/API: promote_best: true.

7. Stop checklist (before you claim success)

Check	Pass criterion
Health	No blockers (or documented force with persona rules)
Leakage audit	No temporal / patient-disjoint red flags you ignore
Discrimination	AUROC / PR-AUC reported with split type
Calibration	Brier / ECE inspected; calibrate if needed
Explain	SHAP drivers make clinical-research sense
Limits	You can state what the model is not (see LIMITATIONS.md)

Diagrams of the wider system: Design & architecture diagrams.