How we prove the forecast
A blind test on months the model never saw, scored on your own data.
The holdout
We test on the months the model was not given.
- Recent months held back. Your most recent months are held back. The forecast is fitted without them, so the score shows how it would have done.
- Two comparisons. We score against a simple benchmark and against your team's forecasts for the same months. To pass, the error has to be lower than both.
- Months with no team forecast. Where your team made no forecast for a month, we mark it not comparable and leave it out.
Changes kept in layers
The model forecast, approved AI suggestions and planner changes are stored separately, so accuracy can be measured for each.
- Model. The system forecast, as fitted on your history.
- AI suggestions. Adjustments the AI market monitor proposed and a planner approved.
- Planner changes. Overrides and consensus edits, with the name of the person who made them.
What we report
The measures we report after the holdout test.
- Error and bias. Mean absolute error and root mean squared error on the held-back months, with bias shown on its own so over- and under-forecasting are visible.
- By level and horizon. Item, group and total, and how error changes with the forecast horizon.
- Forecast value added. Which changes made the forecast more accurate, and which did not.
Limits
A score on held-back months shows how the forecast would have done on your history. It does not promise the same result in the months ahead, and a short or patchy history limits what any method can show.