What we did
- Collected timestamped administration statements, executive orders, and news coverage.
- Generated forecasting questions about policy actions, court rulings, and legislative outcomes.
- Labeled each with the actual subsequent action or decision.
- Fine-tuned a specialized model and evaluated on 682 held-out political questions vs. GPT-5.
Example datapoint
A sample training example — question, source, and outcome-derived label.
DATASET Policy actions
RL
Question
Will the Jan 27, 2025 executive order threatening 25% tariffs on Canadian imports face a successful federal court injunction or vacatur within 90 days of signing?
Question source
New York Times Jan 27, 2025
Trump Threatens Canada With 25% Tariffs Over Border Security
Label
Yes.
Type
binary
Confidence
Label source
Reuters Apr 22, 2025
Federal judge enjoins enforcement of 25% Canadian import tariffs pending merits briefing
Results
Benchmark comparisons against frontier models.
Calibration on 682 Political Test Questions
Trump-Forecaster achieves lower Expected Calibration Error than GPT-5 in both context-aware (ECE 0.079 vs. 0.091) and context-free (ECE 0.164 vs. 0.191) settings — 13–14% better calibration, with the largest gains when no additional context is available.
Read more
Papers, models, datasets, notebooks, and write-ups for this case study.
Blog post linkedin.com
We trained an LLM to predict what Trump will do
LinkedIn announcement with methodology and headline results
Notebook github.com
Trump forecasting fine-tune
Notebook demonstrating fine-tuning for political action prediction
Model huggingface.co
Trump-Forecaster
Fine-tuned model for political action prediction
Dataset huggingface.co
WWTD-2025
Political action prediction dataset