Why backtesting is grading your own homework

Every major catastrophe model in the wildfire market reports impressive accuracy. Almost all of it is measured against data the model already saw. That's not a forecast. It's a memory test the model wrote for itself.

Here's how the typical cat-model validation works. You take decades of historical fire seasons, you fit the model to most of it, and you report how well it "predicts" the rest. The numbers look great — because the model was tuned, directly or indirectly, on the same statistical world it's now being graded against. When the structure of risk shifts — new fuel conditions, new wind regimes, new ignition patterns — the backtest gives you no warning at all.

A backtest tells you how well a model remembers the past. It tells you almost nothing about how it will handle the next fire that doesn't look like the last one.

Memorization wears the costume of prediction

The deeper problem is data leakage: subtle ways that information about the "test" period seeps into the model's design. Feature choices, hyperparameters, even which model architecture you pick are all decisions made by people who have already lived through the outcomes. You can't un-see 2017, 2018, or 2025. Every modeling choice is quietly informed by knowing how those seasons turned out.

This is why two models can both report 90%+ historical accuracy and still disagree wildly about next season. They've each memorized the past well. Neither has been forced to commit to the future in a way that can't be revised after the fact.

The only honest test is a sealed one

There's exactly one way to escape this trap: commit to your prediction before the outcome exists, in a form nobody can quietly edit later. That's what ARIS does. We locked our 2026 California forecast — parcel by parcel — and cryptographically timestamped it on a public record before fire season began.

• No hindsight. The forecast existed before the season it predicts.
• No cherry-picking. Every parcel score is fixed and verifiable, not selectively reported after the fact.
• No quiet revisions. The timestamp makes tampering evident to anyone who checks.

When the season ends, reality grades us. Not a backtest we designed. Not a validation set we chose. The actual fires, against a forecast that was already on the record. That is the difference between a model that survives diligence and one that just performs well in a slide deck.