2.7×
better calibrated than GPT-5 (ECE 0.031 vs. 0.084)
17%
lower Brier score than the clinical base rate
What we did
- Built ICU patient trajectories from MIMIC-III — vitals, lab panels, clinical notes, medications.
- Anchored forecasting questions at each point in the ICU stay (discharge, deterioration, escalation).
- Labeled each question using the actual downstream outcome recorded in the EHR.
- Trained a 32B model and benchmarked against GPT-5 and gpt-oss-120B.
Example datapoint
A sample training example — question, source, and outcome-derived label.
DATASET ICU trajectories
RL
Question
Given the first 24 hours of ICU admission (vitals, labs, initial note, ventilator settings at hour 18), will this patient die before discharge?
Question source
MIMIC-III admission note ICU hour 24
Septic shock on presentation; lactate 4.2, norepi 0.18 mcg/kg/min
Label
No.
Type
binary
Confidence
Label source
Discharge summary ICU day 11
Extubated day 6; discharged to ward in stable condition
Results
Benchmark comparisons against frontier models.
ICU Outcome Prediction: Brier and Calibration Error
Our 32B trained model matches GPT-5 on Brier (0.149 vs. 0.148) while achieving 2.7× better calibration (ECE 0.031 vs. 0.084) — and it beats the clinical base rate (0.149 vs. 0.180). gpt-oss-120B trails both on every metric. Initial results; more to come as we publish the case study.
Read more
Papers, models, datasets, notebooks, and write-ups for this case study.