Six weeks to a forecast that outperforms yours See how
Back to planning notes

Service went up after go-live and so did four other things

Nov 11, 2026, 9 min read

In short

A before and after read across a go-live date compares two periods rather than comparing the change against what would have happened without it, and in a normal quarter four or five other things moved at the same time. Deleting 340 badly served items from a 4,000 item range lifts an aggregate line fill by a full point with nothing planned differently, and raising the safety stock target at go-live buys service with cash. The four designs that produce a defensible number are staggered rollout with matched arms, difference in differences with a pre-period parallel trends check, synthetic control where only one unit was treated, and interrupted time series with a control series. Where none of them is available, publish a decomposition with a named residual instead of an attribution. The one part that cannot be done retrospectively is writing the measurement down before go-live.

The quarter three review has the chart on it. Line fill was 94.1 percent in the six months before go-live and 97.3 percent in the six months after. That is 3.2 points, roughly a million and a half of recovered margin on volume that would otherwise have been missed, and the programme is declared a success on the second slide.

The finance director asks how much of it was the system. The room does not have an answer, because in the same two quarters the assortment lost 340 slow lines, a second source was qualified on the two components behind half of last year's shortages, the safety stock target went from 2.4 weeks of cover to 3.1 at go-live, and promotional intensity dropped when the category went through a price war. Everybody in the room believes the system helped. Nobody can say by how much, and the honest version of that sentence is the one that gets the renewal questioned a year later.

This is a measurement design problem with known answers. The answers cost something, and the cheapest one has to be paid for before go-live rather than after.

What a before and after actually compares

The quantity you want is the difference between what happened and what would have happened without the change. That second term never exists in the data, which is what makes this hard, and every design below is a different way of constructing a stand-in for it.

A before and after comparison uses the previous period as that stand-in. It is a reasonable choice when nothing else moved, and in a planning environment something else always moved. The demand environment is different, the assortment is different, the supplier base is different, the buffer settings are different, and the team spent six months with a project manager asking them daily how things were going.

Landsberger named the Hawthorne effect in 1958 after the Western Electric studies, and it is alive and well in planning pilots. A site that gets weekly attention from a central team improves on almost any measure you point at it, for the duration of the attention. So does a category that was chosen for the pilot because it was performing worst, since a low reading is partly a bad draw and the next reading will be closer to the average whatever you do. Galton described that in 1886.

Before designing anything, work out how much your measure moves on its own. Plot 36 months of the metric with control limits in the form Shewhart set out in 1931. If monthly line fill has a standard deviation of 1.1 points, a 3.2 point shift is real movement. If it has a standard deviation of 2.4 points, you are looking at a quarter that could have happened anyway.

Decompose before you attribute

Some of the gain can be assigned by arithmetic, with no experimental design at all, and doing that first shrinks the part that needs a harder method.

Start with the assortment. Before go-live the range was 4,000 items. The 340 that were deleted were about four percent of order lines and ran at roughly 71 percent line fill, while the surviving 3,660 ran at 95.1. The aggregate is 0.96 times 95.1 plus 0.04 times 71, which is 94.1, the number on the chart. Delete those items and change nothing else, and the aggregate becomes 95.1. A full point of the 3.2 came from removing the items you were bad at, and no planning decision was made differently.

Then the buffer. Cover on the A and B segments went from 2.4 weeks to 3.1, and inventory value rose about eight percent. Extra buffer buys service reliably, which is the entire point of buffer, and it buys it with cash rather than with method. Whatever service that 0.7 weeks bought belongs in a different line of the benefit case than the one labelled better planning, because it can be reversed by anyone with a parameter screen and it carries a holding cost.

Work down the list this way and the residual gets smaller and more interesting. Of 3.2 points, perhaps one is assortment, one is buffer, half is the second source that removed a recurring shortage, and what is left over is where the planning change might live. A residual of half a point on a measure with a monthly standard deviation of 1.1 is not an achievement anybody should defend in front of a finance director, and knowing that before the meeting is worth more than the slide was.

Four designs, in order of what they cost

Staggered rollout with matched arms. Split the estate into two comparable halves on pre-period levels and structure, go live on one half, and keep the other on the existing process for a defined window. Compare the arms on the same calendar weeks. This removes the demand environment, the seasonality, the price war and the weather in one move, because both arms lived through all of them. Everybody gets the system eventually, which answers the political objection that somebody is being held back. The cost is a delay of one quarter on half the estate, and the return is a number you can put in front of an auditor.

Difference in differences. Where a rollout has already happened, find a comparison group that did not receive the change: sites in another region, categories on a different planning process, a business unit that was descoped. Compare the change in the treated group against the change in the comparison group over the same period. The assumption it rests on is that the two groups would have moved in parallel without the change, and that assumption is checkable in the pre-period. Plot both series for the eighteen months before go-live. If they tracked each other, the design holds. Card and Krueger's 1994 minimum wage study is the standard published example of the method and of how the parallel trends check is argued.

Synthetic control. Sometimes there is one treated unit and no natural comparison, because the change went live across a whole region at once. Abadie, Diamond and Hainmueller set out the method in 2010: build a weighted combination of untreated units that reproduces the treated unit's pre-period path, then use that combination as the counterfactual afterwards. The weights are fitted on the pre-period only. It needs a decent pool of untreated units and a long pre-period, and where those exist it is the strongest thing available once the rollout has happened.

Interrupted time series with a control series. The weakest of the four and still much better than two averages. Model the pre-period with its trend and seasonality, project it forward, and compare. Adding any control series that shares the demand environment, even an imperfect one, catches the case where the whole market improved.

Underneath all four is the reason to bother. Kohavi, Tang and Xu, writing in 2020 from a large controlled experiment programme at Microsoft, report that only about a third of carefully designed changes produced the improvement their authors expected, with the remainder flat or negative. That is a rate measured in an environment with clean randomisation and enormous sample sizes. There is no reason to think planning changes are luckier, and the difference is that in planning almost nobody runs the control arm that would have found out.

Measure the constraint at the same time

An improvement measured on one axis is incomplete until the other axis is on the same page. Service rose 3.2 points, inventory value rose eight percent, and expedite freight moved as well. The net number is what the business got.

The pairing of an objective measure with a constraint measure is a scorecard design question with its own rules (BB12). For an attribution exercise the requirement is narrower: name the constraint measures before the change, measure them on the same arms and the same window, and report them beside the headline. A service gain funded by a buffer increase is a real gain that anyone could have bought without the project, and it belongs in the report labelled as such.

The other reason to declare them early is that a measure improving faster than its mechanism is the usual signature of a target being met in a way nobody intended. Forecast error flat, plan attainment flat, service up four points: the money went somewhere, and the ledger tells you where.

Write the measurement down before go-live

Everything above gets much cheaper if a single page exists before the change lands. It takes an hour to write and it cannot be reconstructed afterwards.

The page names the primary measure as a query against named tables, so that the same SQL runs before and after and nobody re-cuts the definition when the answer is disappointing. It names which sites or categories stay on the old process and until when. It names the window and the unit of analysis. It names two or three constraint measures. And it names the result that would count as a failure, in advance, in a sentence with a number in it.

That last item is the one people resist and the one that does the most work. A programme that has written down what disappointing looks like will recognise it. A programme that has not will find a cut of the data where the change worked, because with enough dimensions there is always one.

The offline version of this discipline is the bake-off run on frozen history before anything is bought (E4), and the two are complements. A model that beat the baseline on held-out history and then failed to move anything in production is a common and very informative outcome, usually about adoption rather than about the model.

Where this stops

Contamination is the first thing that breaks a matched arm design in practice. When the same planners work both arms, the habits, the exception rules and the new way of looking at a coverage screen cross over within weeks. That bias runs toward finding no difference, so a positive result from a contaminated design is still evidence, and a null result from one proves less than it appears to.

Sample size is the second. Six distribution centres is not a sample, whatever the arithmetic says. Moving the unit of analysis down to category-week or item-location-week gets thousands of observations, and those observations are correlated inside each site, so the naive standard error will be far too small and will manufacture significance out of nothing. Cluster the errors at the level at which the change was assigned, which is the site, and the honest power of the study reappears.

Comparability is the third. Matching sites on pre-period service does not match them on order profile, labour market, supplier base or the age of the racking. Difference in differences survives fixed differences between the groups. It does not survive a group that was already on a different trajectory, which is why the pre-period plot matters more than the model that follows it.

And there are changes where none of this is available. A single ERP cutover across an entire company, a regulatory change, a safety-driven policy that could not ethically be withheld from half the estate. In those cases the honest output is a decomposition with a named residual and a stated uncertainty, published as such. That is a weaker claim than an attribution, and a weaker claim that happens to be true will survive a finance review a year later.

Before your next go-live, write the one page: the metric as a runnable query, which part of the estate stays on the old process and for how long, the window, and the two constraint measures you will report beside the headline. It is an hour of work, and it is the only part of measuring the improvement that cannot be done afterwards.

More on this in evaluation on the DemandLabs platform.

See what DemandLabs could do for your plan

Request demo

More planning notes

View all

Nov 10, 2026

Half your benchmark gap is a definition and a product mix

The slide has two bars on it. Yours at 2.1 inventory turns, a peer group median at 3.8, and a gap converted into something like eleven million of released cash. By the time the deck reaches the operating review, the number has been written into next year's plan and given an owner.

Aug 23, 2026

Six questions that tell you whether an AI agent is an agent

Gartner coined the term in June 2025 and applied it to supply chain planning in May 2026. Agent washing, meaning conventional automation relabelled as agentic, and their warning was specific: most current offerings do not independently re-sequence objectives, negotiate trade-offs or adapt execution logic. Most of what is being sold as agentic improves the user experience through query interpretation, recommendations and conversational support, without changing decision quality or how decisions get made.

Aug 23, 2026

The 2027 deadline that turns a planning decision into an ERP project

Mainstream maintenance for SAP's Advanced Planning and Optimization suite ends on 31 December 2027. After that, extended support is available at a premium on the maintenance fee and comes with a commitment to move to S/4HANA.