In short: Two tests separate probabilistic demand forecasting from a point forecast wearing a wider confidence band. The first is whether lead time enters the model as a random variable or as a fixed number, which is where most systems fail. The second is whether the output carries its own economics, so the ordering decision falls out of the distribution rather than from a service level applied afterwards. Point forecast metrics cannot evaluate a distribution, so moving to probabilistic output means moving to proper scoring rules at the same time.
Probabilistic forecasting has become the phrase every planning vendor puts on the second slide, and the critique that follows it is fair enough that the category deserves to sit with it for a minute.
Lokad publishes a running assessment of who in this market is actually doing probabilistic forecasting and who is decorating a point forecast with a buffer. Their position, stated more bluntly than most vendors would like, is that if the end of your pipeline is a service level target or a safety stock percentile, you have a deterministic policy tuned by a variance estimate, and the probability language is marketing. They name vendors. Several of the biggest names in this market are on the list.
I think the critique is largely correct, and I would rather engage with it than pretend it was not made. There are two tests worth taking seriously, and a system either passes them or it does not.
Test one: is lead time a random variable or a number
This is the one that catches most systems.
The standard safety stock calculation takes an average lead time, an estimate of demand variability, a service level, and produces a buffer. Lead time enters as a scalar. If your supplier's lead time is sixty days on average with a spread from forty-five to ninety, the formula sees sixty and the spread disappears into whatever fudge factor the planner applies afterwards.
For an importer running ocean freight, that spread is the dominant risk. The demand forecast might be excellent and the plan still fails, because the container arrived three weeks late and no amount of demand accuracy addresses that.
Doing it properly means treating the quantity you care about as demand over lead time, which is a compound distribution: a random number of periods of demand, where the number of periods is itself random. Two ways to compute it, both standard.
If demand per period is described by a probability mass function, the demand over a fixed lead time of L periods is the L-fold convolution of that function with itself. Convolve it once for two periods, again for three, and so on. When lead time is also random, take the mixture over the lead time distribution, weighting each L-fold convolution by the probability of that lead time. This is exact and it runs fast enough on any realistic grid.
If you would rather stay closed-form, the compound normal result gives variance equal to L times the demand variance plus the squared mean demand times the lead time variance. That second term is the one the standard formula drops, and it is frequently the larger of the two. A supplier with a lead time standard deviation of two weeks against demand of a thousand units a week contributes far more to your uncertainty than the week-to-week demand noise does.
That single formula is a decent audit of any inventory system you are being sold. Ask whether lead time variance appears in it. If it does not, you have your answer about how probabilistic the system is.
Test two: does the output carry its own economics
The second test is about what comes out of the pipeline.
A system that produces a distribution and then asks the planner to pick a service level has handed the decision back. The service level is a dial, the dial is set by policy or by habit, and the probability distribution has been used to fill in a lookup table. The economics never entered.
The alternative is to push the distribution into a decision that prices its own outcomes. For a reorder quantity, that means computing, for each candidate quantity, the expected cost of holding what you did not need plus the expected cost of missing what you could not serve, and choosing the quantity that minimises the total. The newsvendor critical ratio is the simplest version of this and it has been around since Arrow, Harris and Marschak in 1951: order up to the quantile where the underage cost divided by the sum of underage and overage costs sits.
Note what that does to the service level. It stops being an input. It becomes an output, implied by the cost ratio, and if the resulting service level looks wrong to the business then the argument is about the cost of a stockout rather than about a target somebody set in 2019.
This is the more demanding of the two tests, because it requires the business to estimate a cost of shortage, and many will tell you that is impossible. It is usually uncomfortable rather than impossible. Lost margin on the missed unit is a floor. Whether to add a customer-relationship cost on top is a judgement call, and making the judgement explicit is better than burying it inside a service target nobody can trace.
Getting a distribution you can trust
Passing the two tests requires having a distribution worth using, and the usual method of producing one has a quiet flaw.
The standard approach fits a model, takes the residuals, assumes they are normal, and draws an interval at plus or minus some multiple of their standard deviation. Demand residuals are frequently skewed and rarely normal, so the interval is wrong in a direction that matters: it is too narrow in the tail you care about.
Three improvements, in increasing order of effort.
Use empirical residual quantiles instead of a normal assumption. Collect the residuals, take their quantiles directly, and widen them by the square root of the horizon to reflect error growth over the forecast period. This is cheap and it removes the worst of the distributional error. One caveat: quantiles estimated independently per horizon can cross, so that your ninetieth percentile ends up below your eightieth. Sort each row to repair it, which is the rearrangement fix from Chernozhukov and colleagues and is as simple as it sounds.
Model counts as counts. Intermittent demand is not approximately normal at any sample size, and a normal interval on an item that sells two units a week will happily predict negative demand. Fit a Poisson, or a negative binomial when the variance exceeds the mean, and use its actual mass function. Overdispersion is the common case in retail, so the negative binomial is usually the better default.
Calibrate the interval so that its coverage is verifiable. This is the part that turns a probability claim into a testable one. Conformalized quantile regression, from Romano, Patterson and Candès in 2019, takes a held-out calibration set, measures how far the true values fell outside the predicted interval, and widens the interval by the empirical quantile of that conformity score. The result carries a finite-sample coverage guarantee that does not depend on the residuals following any particular distribution.
The practical value of the third one is that it makes the claim falsifiable. If you say a band covers ninety percent of outcomes, you can check whether ninety percent of outcomes fell inside it. Run that check on a holdout and plot the result. A system whose ninety percent intervals cover sixty-eight percent of actuals is producing decoration rather than probabilities, and the check takes an hour.
Scoring a forecast that is a distribution
Point forecast metrics cannot evaluate a distribution. If you are moving to probabilistic output you need scoring that moves with you, and there are two standards worth knowing.
Pinball loss, also called quantile loss, from Koenker and Bassett in 1978, scores a single quantile by penalising misses asymmetrically according to which quantile it is. Predicting the ninetieth percentile and landing above it should cost less than landing below it, and pinball loss encodes exactly that asymmetry.
The continuous ranked probability score evaluates the whole distribution at once and reduces to absolute error when the distribution collapses to a point, which makes it directly comparable across probabilistic and point forecasts. In practice it is usually approximated by averaging pinball loss across a grid of quantiles, which is what the M5 uncertainty track did.
Alongside those, keep two diagnostics. Empirical coverage, which is the fraction of actuals falling inside each nominal interval, and a probability integral transform histogram, which should be flat if the distribution is well calibrated. A histogram with mass piled at the edges means your intervals are too narrow. A hump in the middle means they are too wide and you are holding stock you did not need.
What this actually costs
I want to be even-handed about the burden, because the case for probabilistic forecasting is often made without one.
It is more expensive computationally, though not dramatically so at the scales most planners operate at. Exact convolution on a sensible grid is fast, and empirical quantiles are nearly free once you already have the residuals.
It is harder to explain. A point forecast fits in a cell of a spreadsheet. A distribution does not, and the S&OP room will need a way to look at it that does not require anybody to interpret a density plot under time pressure. Usually that means reporting a small number of scenarios pulled from the distribution rather than the distribution itself.
And it exposes a data problem that point forecasting hides. To model lead time as a random variable you need lead time history, meaning actual receipt dates against promised dates, per supplier and per lane, going back far enough to have a distribution rather than an anecdote. A surprising number of businesses do not have that, because the receipt date got overwritten or the promised date was never stored. Discovering this is useful, and it is usually the first real piece of work in a probabilistic programme.
Where I would stop short
Probabilistic forecasting gives you an honest description of how unpredictable demand is rather than making it more predictable, which is a different thing and occasionally an unwelcome one. If item-week demand carries little signal, a well-calibrated interval on that item will be very wide, correctly, and the interval will not be actionable. The right response is to change the decision rather than to narrow the interval.
It also does not fix a censored history. If your sales record is truncated by stockouts, the distribution you fit is a distribution of what you managed to sell, and calibrating it carefully makes it a well-calibrated estimate of the wrong quantity. Recovering demand from censored sales comes first, and no amount of probabilistic machinery downstream compensates for skipping it.
The two tests remain the useful things to carry into a vendor conversation. Is lead time a random variable in your model, and does the output carry the cost of being wrong in each direction. Both are answerable in a few minutes by someone who knows the system, and neither is answerable at all by someone reading from a slide.