RGA Emerging-Risk Monitoring Practicum
Assessed whether Reinsurance Group of America (RGA) could rely on Metaculus crowd forecasts for forward-looking risk decisions — assembled a dataset of 11,000+ questions, evaluated forecasts across 6,184 resolved questions in Python, showed 30.5% lower forecast error than baseline, and presented an LLM-augmented workflow to senior stakeholders.
The problem
Step-change risks — GLP-1 drugs reshaping longevity, pandemic emergence, AI liability — move faster than historical actuarial data can track. RGA needed a statistically grounded answer to one question: can Metaculus, a crowd probabilistic-forecasting platform, be relied on as a forward-looking signal layer alongside the actuarial engine?
The approach
Assembled a dataset of 11,000+ Metaculus questions and evaluated forecasts across the 6,184 that had resolved — Brier scoring requires known outcomes. Computed Brier score (0.148), Brier Skill Score of 0.305 — a 30.5% reduction in Brier score versus baseline — and expected calibration error (0.070), then ran a Murphy decomposition to separate reliability from resolution. Built a domain-level intelligence map across six insurance-relevant risk domains, five structured forecast packages (GLP-1, H5N1, AI liability, life expectancy, climate & macro), and an LLM-augmented analyst workflow with explicit governance rules and a phased 90-day pilot roadmap.
The outcome
A 24-slide final presentation delivered to senior RGA stakeholders: a platform credibility assessment showing 30.5% lower forecast error than baseline across 6,184 resolved questions, with calibration and reliability analysis; a ranked domain intelligence map; five decision-ready forecast packages; an LLM-augmented analyst workflow; and a sequenced implementation roadmap with go/no-go gates.
Business value
Supports earlier identification of emerging risk signals before they appear in traditional actuarial data — enabling more proactive underwriting decisions in fast-moving risk categories.
Test it yourself
The model, not just the conclusion.
Score a forecast the way we scored the crowd.
A Brier score is the squared gap between the probability you gave and what happened (1 or 0). Zero is perfect; saying 50% on everything earns 0.25. Lower is better.
Your Brier score
0.090
Good — in the crowd’s league.
Brier score 0.090. Good — in the crowd’s league.
How it compares (lower is better)
- Metaculus crowd · 6,184 resolved questions0.148
- Baseline · historical base rates0.213
- Always answer 50%0.250
- Your single forecast0.090
The crowd’s 0.148 is 30.5% better than the baseline — real skill, measured only on questions that had already resolved.
Calibration check
You gave ten different events a 70% chance. How many actually happened?
Underconfident. Events you called 70% happened 100% of the time — being “right” every time means your 70% should have been higher. The crowd’s calibration error was 0.07: when it said 70%, outcomes landed close to 70%.
Written from this work
Signal, not oracle: what 6,184 resolved forecasts taught me about trusting the crowd
Our team tested whether a global reinsurer could use crowd forecasts to see emerging risks earlier. The platform beat its baseline by 30.5%. The more useful finding was where that number should — and shouldn’t — be allowed to go.
3 min readA 70% forecast should be wrong three times in ten
Calibration is the part of forecasting leaders most often misread. What the RGA practicum taught me about judging forecasters on the right thing.
2 min read