In short: A vendor accuracy figure was measured on someone else's data at someone else's grain, so the only comparable test is a backtest on your own history. Supply twenty-four months minimum at the grain you actually plan at, and evaluate with rolling origin folds rather than a single holdout, which Tashman set out in 2000. Include both a naive baseline and your incumbent method with its human overrides, because a result without them cannot be interpreted. Score weighted absolute percentage error as the headline with scaled error and bias alongside, and treat a large reported improvement as a fault in the evaluation.
Vendor accuracy claims are not comparable across vendors, because each one was measured on a different dataset at a different grain with a different baseline. A quoted number is an artefact of somebody else's data.
The alternative is a bake-off on your own history, and it is a couple of weeks of work that will tell you more than six months of demonstrations. Here is the protocol, including the parts that let a vendor make themselves look better than they are.
What to supply
Twenty-four months minimum, thirty-six preferred. Two years is the floor for seasonal patterns and it is thin if you have moving holidays or want to see year-on-year stability. Anything less and the evaluation cannot separate a good model from a lucky one.
The grain the decisions are made at. If you replenish at item-location weekly, supply item-location weekly. Evaluating at a coarser grain than you plan at will flatter every method, because aggregation cancels noise.
Your actual driver data, in the state it is actually in. Price, promotion calendar, calendar events, whatever you have. Do not clean it more than your production process cleans it. A model tuned on tidied data will disappoint later, and the gap between the pilot and the deployment is a well-known source of disappointment.
Stockout and availability flags if you have them. These matter for the reasons covered elsewhere in this series, and they also matter for the evaluation itself, since scoring a forecast against censored actuals penalises the model that got demand right.
How to split it
This is where most evaluations go wrong.
Use rolling origin evaluation rather than one holdout. Train on the first period, forecast the next block, then roll the origin forward and repeat. Tashman set this out properly in 2000 and it remains the standard. A single holdout gives one number and no way to tell whether it was luck. Six to twelve folds gives you a distribution, and the distribution is the result.
Never let the vendor choose the holdout. Specify the folds yourself and supply the data in a form where the evaluation periods are withheld. This removes the possibility of unconscious selection rather than implying bad faith.
Match the forecast horizon to your lead time. If you order twelve weeks ahead, the twelve-week-ahead accuracy is what matters. One-step-ahead accuracy looks much better and is irrelevant to the decision you actually make.
The fold count falls out of that arithmetic and it is worth doing before you agree a protocol. With thirty-six months of history, a twenty-four month minimum training window and monthly origins, you have twelve origins available, and a twelve-week horizon means the last three of them run out of actuals to score against. That leaves nine or ten usable folds, which is a workable number and is nowhere near a large sample. With twenty-four months of history the same arithmetic leaves you almost nothing, which is the real reason two years is described as a floor rather than a target.
The baselines that have to be included
Two competitors, and skipping either makes the result uninterpretable.
The naive baseline. Same period last year, or last period carried forward. Free, and the floor every method must clear. Without it, a percentage accuracy figure has no meaning.
Your incumbent method. Whatever you run today, including the human overrides, because that is the thing being replaced. Comparing a vendor model against a naive baseline while your current process is already well ahead of naive tells you nothing about whether to switch.
Reproducing the incumbent's overrides in a backtest is genuinely awkward, since the overrides were made with information the backtest does not have. The workable approach is to score the incumbent using the forecast versions actually published at the time, which requires having retained them. If you have not, that is worth fixing now.
When that step gets skipped, a specific and very common failure follows. The incumbent arm ends up being your raw statistical engine with none of the human work attached, and the vendor is then compared against a version of your process that has never actually run. Everyone involved believes the comparison is fair, because the incumbent's software is genuinely in the comparison.
The symptom is easy to check and almost nobody checks it. Compare the incumbent's measured error in the bake-off against the accuracy figures your own team publishes internally each month for the same periods and the same items. Those two numbers should be close. If the bake-off has your incumbent looking several points worse than your own reporting says it was, the incumbent arm is a straw man and every conclusion drawn from the exercise is inflated by the difference. Run that check before the results meeting rather than during it.
How to score it
Weighted absolute percentage error as the headline, because it handles zeros and weights by volume. Kolassa and Siemsen make the case for it as the default where volumes vary, which is every real catalogue.
Scaled error alongside it, so the comparison against the naive baseline is explicit and reported rather than inferred. Hyndman and Koehler proposed the scaled error in 2006 for exactly this situation, where series have different volumes and some contain zeros, and it has the property that a value below one means the method beat the naive benchmark on that series. Specify it by name in the protocol, because a metric with a defined benchmark built into it is much harder to present selectively.
Bias separately. A method can match on error and be systematically high or low, and directional bias is what fills or empties warehouses. Report it per segment.
By segment rather than only in aggregate. Fast movers, slow movers, promoted items, new items, intermittent items. The aggregate hides the pattern, and the pattern is where the decision lives. A method that wins overall by doing well on the easy segment and badly on the hard one is not the method you want.
Statistically rather than just numerically. Across your folds, count how often each method beat the other. A sign test on that count tells you whether the difference is real or noise. A method that wins by two percent in seven folds out of twelve has not proven anything, and treating that as a result is the most common overreach in these evaluations.
The numbers behind that are quick enough to carry in your head. With twelve folds and two methods of genuinely equal skill, each fold is a coin toss. Winning seven or more happens about thirty-nine percent of the time by chance. Eight or more, about nineteen percent. Nine or more, about seven percent. Ten or more, about two percent. So a vendor showing eight wins out of twelve is showing you something that occurs by luck roughly one time in five, and a vendor showing ten out of twelve has shown you something. Ask for the per-fold table rather than the average, and count the wins yourself. It takes a minute and it is the single most useful piece of arithmetic in the whole evaluation.
Where you want something stronger than a win count, Diebold and Mariano set out the standard test for comparing predictive accuracy in 1995, and it uses the size of the differences as well as their direction. It is more powerful than a sign test and it makes more assumptions about the error series, so it is worth having somebody who understands both apply it. The sign test is the one you can insist on in a protocol and check without help.
Four ways to be flattered
Things to check for, none of which necessarily indicate bad faith and all of which produce results that will not reproduce.
Label leakage. A feature that contains information from the period being predicted. Rolling averages computed over a window that includes the target, a promotional flag set retrospectively, a price field updated after the fact. The tell is accuracy that is much better than the naive baseline in a segment where the drivers are weak. We have caught this in our own work more than once, and it always looks like a breakthrough until you find it.
Hyperparameter tuning on the test set. If the model was tuned by looking at holdout performance, the holdout is no longer a holdout. Ask what was tuned and against what.
Item selection. An evaluation run on a curated subset of well-behaved items will beat one run on the whole catalogue. Specify the item set yourself and include the difficult segments.
Metric selection after the fact. If the reported metric changes between the proposal and the results, ask why. Agree the metrics in writing before the data is handed over.
What a good result looks like
Modest. If a vendor reports a twenty point accuracy improvement over your incumbent, something is wrong with the evaluation rather than right with the model.
Realistic improvements over a competent incumbent are usually a few points of weighted error, concentrated in specific segments, with some segments showing no improvement at all. That result is more credible than a uniform gain and it is more useful, because it tells you where the value is.
It is worth knowing the ceiling before you start. Measure the achievable accuracy on your own data first, so the bake-off result can be read against what was possible rather than against a target somebody set in a board meeting.
Beyond accuracy
Accuracy is the easiest thing to measure and it is not the whole decision. Three other things belong in the evaluation and they need to be arranged deliberately.
The inventory consequence. Simulate the buffers and orders each forecast would have produced, and compare the resulting service and stock. A forecast that is slightly less accurate and better calibrated in the tail can produce better inventory outcomes, and an accuracy-only comparison will pick the wrong one.
That simulation is less work than it sounds and it belongs in the same fortnight. Take each candidate's forecast, run it through your existing replenishment rules with your existing service targets, generate the order stream, and play it against the actual demand that occurred. The outputs are two numbers per method: the service level achieved and the average stock held. Those are the numbers the business actually cares about, they are on the same footing across vendors, and they settle arguments that a percentage error figure never will. Specify this in the protocol at the start, because retro-fitting it after the accuracy results are in turns it into a challenge to the winner rather than part of the test.
Explainability under challenge. Take three forecasts a planner would question and ask each system to explain them. The answer quality varies enormously and it determines whether the tool gets used.
Effort to reproduce. How much manual work went into the bake-off result. If a data scientist spent three weeks tuning per segment, that cost recurs and it belongs in the comparison.
The limits
A backtest measures how a method would have performed on the past. It cannot tell you how it will perform on a business that is changing, and it says nothing at all about implementation, adoption or support quality, which frequently matter more than the model.
A short evaluation window is also genuinely limited. Three months of holdout is a small sample and a narrow win over it is unproven, however carefully the arithmetic was done. Say so in the write-up rather than presenting a small margin as a decision.
None of that argues against running it. It argues for reading the result as one input with a stated confidence, which is a far better position than choosing between two vendor claims measured on data neither of you has seen.