In short: Forecastability sets a ceiling that no model or vendor can pass, and it is measurable from your own history in about two days. Any accuracy figure quoted without a naive baseline is unfalsifiable, since 94 percent against a naive 92 percent means the model is contributing two points. Error rises sharply as the grain gets finer, so an item-week target can be arithmetically unreachable on data that supports a good monthly one. Produce four numbers per segment before signing anything: the naive baseline error, the achievable error, the ratio between them, and the grain at which that ratio stops beating one.
A planning director once told me their target for next year was 85% forecast accuracy at item level, weekly. When I asked where the number came from, the answer was that a vendor had quoted it and the board had liked the sound of it. Nobody had checked whether the data could support it.
It could not. Nothing could have. Roughly forty percent of that catalogue sold in fewer than half the weeks of the year, and for those items a weekly accuracy target is a statement about arithmetic rather than about effort. The team spent three quarters missing a goal that was unreachable on the day it was set, and the planners got blamed for it.
Forecastability is the ceiling your data imposes before any model touches it. You can measure it in a couple of days on history you already have, and doing so changes three decisions: what to target, what to automate, and what to stop trying to improve.
The floor tells you more than the target does
Start with the naive forecast. For a weekly series with annual seasonality, that means predicting each week with the value from the same week last year, or for a non-seasonal series, predicting next week with this week. It costs nothing, it requires no software, and it is the number every model in your stack has to beat before it deserves a budget line.
Now compute your candidate model's error and the naive model's error over the same periods, and take the ratio. This is the mean absolute scaled error, introduced by Hyndman and Koehler in 2006, and it does something percentage errors cannot. A value below one means the model beats the naive baseline. A value above one means you are paying for something that performs worse than a rule a clerk could apply from memory.
That single ratio reframes the conversation. A team reporting 72% accuracy sounds like it is struggling until you learn the naive baseline on that data lands at 61%, at which point the model is doing real work. Another team reporting 88% sounds excellent until the naive baseline turns out to be 89%, at which point the entire forecasting function is a net negative and nobody had noticed because the headline number looked healthy.
Run the scaled ratio by segment rather than in aggregate. The aggregate hides the two things you need to see, which are the segments where the model is adding a lot and the segments where it is adding nothing at all.
Error explodes as the grain gets finer
The second measurement is the one that saves the most money, and it is the one almost nobody runs before signing a contract.
Take the same data and evaluate the same model at four grains. Total by month. Total by week. Category by week. Item by week. Then plot the error at each. What you will see is closer to a cliff than a gentle slope, and where the cliff falls is a property of your business that no vendor can tell you in advance.
We ran this on a distribution dataset with a daily date column, so the grain could be cut properly rather than inferred. The monthly total was forecastable to a useful standard. Rolling a weekly model up to a monthly total came out materially better than the monthly model trained directly, which was a genuine surprise and worth its own investigation. At item-week, the weighted error came in near 99%, which is a polite way of saying the forecast carried no information at all. Same data, same feature set, same modelling effort. The difference was entirely the grain.
That result is common, and it says nothing about the quality of the modelling. It is aggregation working the way aggregation works. Independent noise cancels as you sum, so a total is always easier than its parts, and the question is only where on that curve your business sits.
The operational consequence is direct. If item-week carries no signal, then any decision that depends on an item-week forecast is being made on noise, and dressing the noise in a confidence interval does not fix it. Those decisions should either move to a coarser grain where the signal exists, or move to a policy that does not need a forecast at all, which for slow movers usually means a reorder point sized from the demand distribution rather than a prediction of next week.
Three properties that set the ceiling
When you know where the cliff is, it is worth understanding what put it there. Three things do most of the work.
How often the item sells. The average interval between non-zero demands is the first thing to measure, alongside the squared coefficient of variation of the demand sizes. Syntetos and Boylan proposed cutoffs around 1.32 for the interval and 0.49 for the variability, and those two numbers sort a catalogue into four regions with genuinely different achievable accuracy. Items in the lumpy quadrant, where demand is both infrequent and erratic when it arrives, are near the bottom of what forecasting can do. That is a well-established finding and not a local problem.
How much of the variation is driven by things you can observe. A product whose demand moves with price, promotion and season is far more forecastable than one whose demand moves because a single large customer changed their mind. The test is whether your driver columns explain variance out of sample. If adding every exogenous variable you have improves the holdout error by two percent, the demand is being driven by something you are not measuring, and no model change will fix that. More features from the same source will not help either. You need a different source.
How stable the relationship is over time. Run the model on the first half of your history and evaluate on the second, then reverse it. If the two results differ sharply, the process generating demand changed, and a model fitted on the full period is averaging two different worlds. Distribution drift is measurable with a population stability index or a two-sample test between the training and recent windows, and it is worth checking before you conclude a model is bad. Sometimes the model is fine and the business changed underneath it.
Say the number out loud before the project starts
The productive version of this exercise takes about two days and produces four numbers per segment: the naive baseline error, the achievable error from a reasonable model, the ratio between them, and the grain at which the ratio stops being better than one.
Publish those before anyone sets a target. The conversation that follows is uncomfortable and it is the most valuable conversation in the whole programme, because it is the moment the organisation stops treating forecast accuracy as a measure of how hard the planning team is trying.
It also changes what you automate. Touchless forecasting is a reasonable ambition for the segment where the model beats naive comfortably and the drivers are observable. It is a bad ambition for the lumpy quadrant, where the right answer is an inventory policy that tolerates being wrong rather than a better forecast. Gartner's own advice on where to start with automation points the same way, toward high-volume activities where the impact is measurable and the cost of an error is low, and forecastability analysis is how you find those segments rather than guessing at them.
What this does to a vendor evaluation
Any accuracy figure quoted to you without a baseline is unfalsifiable. Ninety-four percent means nothing on its own. Ninety-four percent against a naive baseline of ninety-two percent on stable, high-volume items means the model is contributing two points, and you should decide whether two points is worth the licence.
Ask for the comparison instead of the headline. What was the naive baseline on the same data. What grain was it measured at. What was the holdout window and how many folds. Was the incumbent method included as a competitor. Those four questions will separate the vendors who have measured their own performance from the ones who have not, and the sorting happens quickly.
I would also encourage you to be suspicious of your own results in one specific way. If a backtest looks unexpectedly good, check for label leakage before celebrating. We have caught this in our own work, where a feature built from a rolling statistic quietly included the period being predicted, and the resulting accuracy was excellent and completely fictional. The tell is a result that is much better than the naive baseline in a segment where the drivers are weak. That combination is usually a bug rather than a breakthrough.
Where this stops being useful
Forecastability analysis tells you the ceiling under the data you currently have. It does not tell you what the ceiling would be under data you could acquire, and that distinction matters commercially.
If your history is shipment data out of your own warehouse, a large share of the variance you cannot explain is your own ordering behaviour rather than consumer demand, and the fix is a different data source rather than a different model. The same applies when stockouts have censored the record, because the ceiling you measured includes the distortion. In both cases the honest statement is that this is the limit given what you are feeding the model, and there is a data question sitting underneath it that is worth answering separately.
There is also a sample-size caveat that gets skipped constantly. If your holdout is three months, the difference between two models is often inside the noise band, and declaring a winner from it is overconfidence with a spreadsheet attached. Use rolling origin evaluation with several folds, and treat a narrow win over a small number of periods as unproven rather than as a result.
None of this makes forecasting less valuable. It makes the value legible, which is the difference between a planning function that can defend its budget and one that spends every review apologising for a number it was never going to hit.