Signal, not oracle: what 6,184 resolved forecasts taught me about trusting the crowd

In spring 2026, through Olin’s Center for Experiential Learning, our six-person team took on a question from Reinsurance Group of America: could Metaculus, a public crowd-forecasting platform, serve as a forward-looking signal for emerging risks? My role on the team was strategy and forecasting.

The question matters because some of the risks a life reinsurer cares most about — GLP-1 drugs changing longevity, a new pandemic, AI-related liability — can move faster than the experience data actuaries rely on.

The forward-visibility gap

Actuarial science is excellent at pricing what has happened many times before. It is structurally slower with risks that are new. A therapy can begin changing mortality before long-run population data exists. A pathogen can become a portfolio stressor before there is any base rate for that specific scenario.

In that gap, risk often shows up first in public signals — research, regulation, expert judgment — rather than in claims. That is where an external forecasting signal could add value, if it is good enough to trust.

Only score what has resolved

To test that, we assembled a dataset of more than 11,000 Metaculus questions and evaluated the 6,184 binary questions that had already resolved. That restriction matters: you can only score a forecast against an outcome that is known.

The main measure was the Brier score — the squared gap between the probability given and what happened. Across the resolved set, the crowd scored 0.148, a 30.5% improvement over the baseline, with a calibration error of 0.070. Try the scoring yourself:

What 30.5% does — and doesn’t — mean

A 30.5% skill score means the crowd adds real information beyond historical frequencies. The calibration result means its probabilities are interpretable: when it says 70%, outcomes land close to 70%.

What it does not mean is that a crowd probability belongs in a pricing model. The platform cannot see a reinsurer’s book, underwriting rules, or exposures. It estimates how likely something is, not why it happens or how severe it would be. So our recommendation was deliberately bounded: use crowd forecasts as a signal, not an oracle — a structured trigger to investigate when the outside probability diverges from internal assumptions.

Where LLMs fit

Large language models were not the best forecasters in the research we reviewed. Their value is operational: gathering evidence, retrieving base rates, drafting scenarios, and red-teaming assumptions so an analyst can produce a decision-ready forecast packet faster.

The workflow we proposed kept humans at every decision point: the crowd signal flags a shift, an LLM drafts the evidence summary, an analyst reviews and decides whether to escalate, and anything that could touch pricing goes to actuarial review first. One rule was non-negotiable: no forecast enters pricing, reserving, or capital decisions without that review.

A bounded pilot, not a platform

We did not recommend enterprise integration. We recommended a 90-day pilot with a named owner, a focused set of tracked questions, and explicit go/no-go criteria — judged not only on forecast accuracy but on decision value: did the signal change what the risk team chose to investigate?

What I took from it

Accuracy is necessary, not sufficient. A well-scored signal without governance can create false confidence faster than no signal at all. Governance is part of the product.

Averages hide the useful part. Performance varied by risk domain, so the right playbook differs by domain rather than treating the platform as one number.

Calibration changes how leaders should read forecasts. A well-calibrated 70% should be wrong about three times in ten. Teams that punish every miss push forecasters toward vague language, and vague language can’t be scored.