A planning system upgrade moves numbers without any configuration change, through alternate optima, gap tolerances, changed defaults, floating point summation order and unspecified sort order in allocation. Comparing a release cell by cell produces a difference list nobody can work, so filter to the horizon where the plan commits money and to changes above the minimum action size. Build the regression baseline as frozen inputs, full outputs and a recorded version, and take releases singly rather than batching three, because the difference you have to explain is the sum of them.
The vendor ships release 24.3 over a weekend. The notes list performance improvements, a security fix, and two defect corrections in modules you do not use. Nothing under functional change.
Monday morning the supply plan runs. Of 11,000 planned item-locations, 1,900 differ from Friday's plan. Most differences are small. Four are large enough that a planner notices before lunch, and one of those is a purchase order that would have been released this week and now would not.
Nobody changed a parameter. Nobody edited a master data field. The inputs are the same inputs. And the first question in the room is whether the new version is broken, which is the wrong question and will take three days to answer badly.
Why an upgrade moves numbers with no configuration change
Five mechanisms account for most of it, and knowing which one you are looking at is the difference between a morning and a fortnight.
Alternate optima. Linear and mixed integer programmes routinely have many solutions with identical objective values. Sourcing the same 10,000 units from plant A or plant B at the same total cost is a tie, and the solver breaks ties by whatever its search happened to reach first. A new solver version explores in a different order and returns a different, equally optimal plan. The objective value matches to the penny and the plan looks different in a hundred places.
Gap tolerance and time limits. Most production supply solves stop at a stated optimality gap or a wall clock limit rather than at proven optimality. Where the run stops at a one percent gap, the returned solution is whatever incumbent the search held at that moment. Make the solver faster and it explores further in the same time, which changes the incumbent. The plan improves and it also changes, and the two are hard to separate without measuring the objective.
Changed defaults. A parameter that defaulted to zero now defaults to one. A rounding rule that truncated now rounds half up. A defect correction restores intended behaviour that you had unknowingly built a process around. This class is the most consequential and the least visible, because the release notes describe it from the vendor's point of view rather than yours.
Floating point summation order. Addition in floating point is not associative, so a sum computed across eight threads can differ from the same sum across four in the last bits. On its own this is irrelevant. Downstream of a threshold comparison or a rounding to case quantities, it flips individual decisions: a projected shortfall of 999.9999998 against a trigger of 1,000 goes one way before the upgrade and the other way after.
Unspecified sort order. Allocation, fair share and rationing routines process demand in an order. Where the order is defined by priority and date but ties are left to the query, an index change or a different execution plan reorders the ties, and order-dependent allocation produces different results for the same inputs. Of the five, this is the one most often reported as a defect when the system is behaving exactly as written.
Test the cheap hypothesis first. Rerun the previous version on today's inputs. If it reproduces Friday's plan, the difference is in the release. If it does not, the difference is in the data and the upgrade is a coincidence, which happens more often than anyone expects because upgrade weekends tend to sit next to period ends.
A regression baseline is frozen inputs and stored outputs
You cannot detect a regression without something to compare against, and production last week is unusable as a comparison because the inputs have moved.
The baseline is a set of scenarios, each consisting of a complete input snapshot at a fixed timestamp, the full output produced from it, the version of every component that produced it, and the random seed where anything stochastic is involved. Stored together, versioned, and restorable without a manual assembly step.
Coverage matters more than volume. A suite of six well-chosen scenarios beats forty copies of the same normal week. Include one period where a hard capacity constraint binds, one with a supply shortage forcing allocation, one containing a new product with no history, one at a period boundary where the calendar logic has to be right, and one that exercises whatever your business does that the vendor probably did not test, whether that is consignment stock, multi-level postponement or a currency the reference implementation ignores. AA8 covers how to construct the assertions that run against each of them.
Baselines decay. Every configuration change you accept in normal operation makes the stored outputs wrong in a way that has nothing to do with the vendor, so the baseline has to be regenerated deliberately whenever you accept such a change, with a note saying what moved and why. A baseline nobody has regenerated in a year will fail the next upgrade for reasons that predate it, and the team will spend two days rediscovering their own history.
Deciding which of the 1,900 differences matter
Cell by cell comparison generates a list nobody can work. The filter has three stages and it usually removes most of the list before anyone looks at a single line.
Start with the horizon. Differences beyond the point where the plan commits money are noise for this purpose. A planned receipt in month seven that moves by 40 units changes nothing that anyone will do, and it will be replanned eleven times before it becomes real. Restrict the comparison to the frozen and firm horizons, and to anything that generates an order, a transfer or a capacity booking inside the notice period.
Then apply the minimum action size. A change smaller than the lot size, the rounding value or the minimum order quantity cannot change an action, because the action is quantised. Comparing at finer resolution than the system can act on produces differences that are arithmetically real and operationally empty.
What survives is the set that crosses a decision boundary: an order that appears or disappears, a source that switches, a date that moves across a cutoff, a quantity that changes by more than one lot. On a plan of 11,000 item-locations with 1,900 raw differences, that filter commonly leaves a few dozen, and a few dozen is a list two people can work through in a day.
Look at the aggregate at the same time. Total planned receipts, total projected inventory value by month, total capacity consumed, and the objective value where the solve reports one. Individual differences that cancel in every aggregate point at alternate optima. Aggregates that shift in one direction point at a changed default or a changed rule, which is the case that needs a root cause before anyone signs off.
Where the plan contains a genuinely stochastic component, the comparison is a stability question rather than an equality question. Run the same scenario across several seeds on both versions and compare the distributions rather than two single runs, otherwise you will chase a difference that the old version also produces against itself.
The upgrade window collides with the planning calendar
The technical work is usually the smaller constraint. The binding one is that a monthly cycle leaves very little time when a planning system can safely change underneath it.
Take a common shape. Demand review in week one, supply review in week two, executive review in week three, publication and commitment at the start of week four. The system has to be stable through all of it, which leaves the back half of week four, and that is also when finance closes the period and wants nothing moving. The real window is often three or four days a month, and it is not the same days every month because of quarter ends.
Work backwards from that window. If the regression suite takes two days to run and review, it starts before the window opens, which means it runs against a release you have already accepted into a test environment. If a rollback takes six hours, it has to fit inside the window with the verification that follows it. Where those numbers do not fit, the choices are to shorten the suite, parallelise the runs, or accept the upgrade during a live cycle with a defined blast radius. SS2 covers the calendar design itself, and the point here is that the calendar sets the upgrade cadence more than the vendor does.
You do not get to decline, so decide how much to batch
Software as a service releases arrive on the vendor's schedule. The variable you control is whether you take them singly or let three accumulate and take them together.
Batching feels safer because there is one disruption instead of three. It is worse on the measure that matters, which is your ability to explain a difference. Take three releases at once and every difference has three candidate causes, the interactions between them are yours to untangle, and the rollback removes fixes you wanted along with the change you did not. Forsgren, Humble and Kim's 2018 analysis of delivery performance found smaller and more frequent changes associated with lower change failure rates and faster restoration, and the reasoning transfers directly: a smaller diff is a smaller investigation.
Two of their measures are worth keeping on your own upgrades, because they are your data rather than a vendor claim. Change failure rate, meaning the share of releases that required a rollback or an unplanned fix. Time to restore, meaning how long from noticing to being back on a known-good state. Both are countable from your own history after four or five releases, and both are more informative about your upgrade risk than any release note.
Where the architecture allows it, run the new version against production inputs in shadow before it becomes authoritative, publishing nothing, and diff the outputs on the schedule you already run. The site reliability practice described by Beyer and colleagues in 2016 calls the general form canarying, and it earns its cost on planning systems because the outputs are compared numerically rather than by watching for errors.
When the new answer is the right one
The uncomfortable case is a difference traced to a defect correction, where the old plan was wrong and nobody knew. Now you have a plan that changed, planners who noticed, and an explanation that begins by admitting the previous numbers were incorrect.
Handle that as communication work scheduled into the upgrade rather than as an incident discovered afterwards. Before the planners see the new plan, they should have a short written list of what moved, which mechanism moved it, and which items are affected by name or segment. A planner who reads that list first treats the change as maintenance. A planner who finds the change alone on a Monday treats the system as unreliable, and E5 covers what that does to adoption over the following months.
The same list is worth keeping. After a year it is the only record of why the system's behaviour differs from the specification signed at go-live.
Where this stops
Regression testing establishes that the new version does what the old version did on the cases you thought to include. It says nothing about the cases you did not, and the cases you did not include are correlated with the parts of your business that are unusual, which is where vendor defects concentrate.
The cost is real too. A full shadow environment at production data volume is the single largest line in this whole practice, and plenty of organisations genuinely cannot fund one. The honest fallback there is a reduced scenario suite on subset data, a rehearsed rollback with a measured duration, and a first-cycle plan that limits which categories execute on the new version. That is weaker and it is much better than taking the release blind.
Release notes will also never be complete, because completeness would require the vendor to know which behaviours you depend on. A tie-breaking rule nobody documented is still a behaviour you built a process around, and the only party who can discover that is you, by comparing outputs.
Before the next release lands, freeze one day of production inputs and the outputs they produced, store them with the version number attached, and put them somewhere a restore does not require the person who built them.
More on this in systems on the DemandLabs platform.
