Your planning support queue is three different jobs in one inbox

After go-live, a planning support team gets incidents, questions about correct answers, and configuration changes in one queue. How to separate them, keep the overnight chain running, and staff it.

Systems30 October 20269 min read
In short

Three kinds of work arrive on the same queue after go-live: incidents, requests to explain a number the system computed correctly, and configuration changes the business now needs. They want different people, different response times and different closure conditions, and the intake label is the only place you get to separate them. A chain of forty nightly jobs each running at 99.5 percent reliability fails on roughly one night in five, so the design that matters is restartability rather than prevention. Hypercare should exit on measured conditions, and the explanation queue is the one that shrinks when the product gets better rather than when the team gets faster.

At 05:40 the demand feed lands with zero rows, because an upstream export ran while the source table was locked. The chain treats an empty file as a valid file. Every downstream step succeeds, the run completes eleven minutes early, and at 07:00 the system publishes a supply plan built on demand history that stops a week ago.

At 08:15 a planner emails a screenshot asking why the replenishment proposal on a slow-moving line came out at 4,000 units when the item sells about forty a week.

Both land in the same inbox, with the same subject line format, and whoever opens them first decides which one gets looked at. The screenshot is the more interesting question and it will win. The empty feed is an outage that nobody has noticed yet, and it stays unnoticed until somebody downstream acts on the plan.

That morning is the argument for having a support model at all. What follows is the shape of one that holds up in the second year, when the implementation partner has gone and the queue is still arriving.

The three kinds of work that share one queue

ITIL 4, published by Axelos in 2019, separates incidents from service requests and both from changes, and the separation is useful here for a reason that has nothing to do with process certification. The three categories want different responders, different clocks and different definitions of done.

Incidents. The system did something it should not have done, or did not run at all. A job failed, a feed arrived corrupt, a screen times out, a published plan contains negative projected inventory. These are bounded, they have a technical owner, and they close when the behaviour is corrected.

Explanation requests. The system is right and somebody needs to know why. Why 4,000 units. Why the plan moved 300 units on an item nobody touched. Why this order shows as late when the supplier confirmed it. These have a planning owner rather than a technical one, and they close when a person understands the answer.

Configuration changes. The system is doing exactly what it was configured to do eighteen months ago, and the business has moved since. A new distribution centre, a changed lot size policy, a supplier whose lead time is now nine weeks rather than five. These close when a change is made, tested and recorded.

Planning has a wrinkle that generic service management does not anticipate. Explanation requests arrive labelled as incidents, because the planner reports them honestly as "the system is wrong". They only become explanation requests once somebody has looked. So the intake label has to be provisional, revisable once, and the revision has to be cheap enough that nobody resists it.

The mix decides your staffing. A queue of 120 tickets a month where 70 are explanations is a training and traceability problem being handled by engineers. The same queue where 70 are configuration changes is an unfunded roadmap being run through a support inbox. You cannot tell which one you have until you label ninety days of history, and almost nobody does that before hiring.

The overnight chain fails more often than anyone plans for

Take a nightly chain of forty steps: extractions, loads, master data refreshes, a forecast run, a netting pass, a supply solve, publication. Give every step 99.5 percent reliability, which is generous for anything reaching an external system. The chain completes on 0.995 to the fortieth power of nights, which is 0.818. Roughly one night in five, something in that chain does not do what it was supposed to.

That arithmetic settles a design argument. A support model built to prevent failures will be busy and will still be woken up. A support model built so that any step can be rerun in isolation, at any point, without corrupting the steps around it, turns most of those nights into a ten minute intervention.

The failures worth designing against hardest are the ones that report success. A feed that delivers a zero-row file. A currency table that arrives with last quarter's rates and a current timestamp. A partial load where the job counted its own rows rather than the source's. Every one of those produces a green chain and a wrong plan, and the wrong plan is more expensive than a red chain because it gets executed.

The protection is assertions on inputs, evaluated before the expensive steps run. Row count inside a band derived from the trailing eight weeks for that weekday. A control total on units or value that has to reconcile with the source within a stated tolerance. A freshness check on the maximum transaction date. A hard deadline by which the chain must have reached publication. Each of those is a few lines, and each of them converts a silent wrong answer into a loud stop.

Restartability decides what a failure costs

Two questions determine whether a 03:00 failure is a nuisance or a lost day. Can step 27 be rerun without rerunning steps 1 to 26. And does rerunning step 27 twice produce the same result as running it once.

Where the answer to both is yes, recovery is bounded by the runtime of one step. Where the answer to the second is no, because the step appends rather than replaces or writes a sequence number on each pass, recovery involves cleaning up before retrying, and that cleanup is where support teams do real damage under time pressure at four in the morning.

The second thing to settle in advance is the publication rule. Say the chain normally finishes at 06:00 and the plan is expected at 07:00. If it has not finished by 07:00, what appears on the planners' screens? The workable answer is the previous day's plan, clearly marked with its age, plus a notification to everyone who uses it. The answer that causes trouble is silence followed by a fresh plan appearing at 11:15, because by then half the planners have worked the old numbers and the other half are working the new ones, and the two groups will not discover the split until a review meeting.

Write that rule down, automate it, and name the person allowed to override it. Then exercise it deliberately once a quarter, because a fallback path that has never run is an assumption.

Explaining a correct answer is a design problem

The 4,000 unit question is answerable. Which forecast version fed the run, what the parameter set was at that moment, which constraint was binding, what changed since the previous run, and whether a manual override upstream is still in force. Every one of those facts existed at runtime.

The difference between a support model that scales and one that does not is where those facts live. If reconstructing them takes an analyst forty minutes across three databases, the explanation queue never falls, and it grows as adoption grows. If the planner can see the same chain in two minutes on the screen where the recommendation appeared, most of those tickets never get raised.

Seventy explanation tickets a month at forty minutes each is about 47 hours, which is more than a quarter of a full-time person spent reconstructing things the system already knew. That is the number to put in front of whoever owns the roadmap, because it converts a support cost into a product requirement. Storing the input versions, parameter values and binding constraints alongside each published plan is the mechanism, and T5 covers what a defensible decision record has to contain.

There is a second-order effect worth expecting. Explanation volume falls when planners learn the model, then rises again after any significant change, and a spike three weeks after a parameter revision is a signal that the change was not communicated rather than a signal that the system broke.

Configuration changes are the roadmap wearing a support badge

Small changes arrive at four to six a month in a mid-sized deployment, which is sixty a year. Individually each one is twenty minutes. Collectively they are the largest uncontrolled influence on what the system does, and after two years nobody can say why a given parameter has the value it has. P3 covers what that looks like when it lands on reorder points, and SS1 covers the same failure on forecast overrides.

Treat a configuration change with the discipline of a code change. Recorded with a reason and a requester. Attributed to a person. Reversible, with the prior value stored rather than described. Impact-run before it lands, meaning applied in a copy, rerun against a frozen input set, and reported as a count of what moved and by how much. And where the change is temporary, given an expiry date at the moment it is made, because a temporary change with no expiry becomes permanent by silence.

The impact run is the same machinery you need for vendor releases, which is the subject of AT2, and building it once for both is the reason to build it properly.

Staffing, and the ceiling on operational work

Google's site reliability practice, described by Beyer and colleagues in 2016, caps operational work at 50 percent of an SRE team's time, with the remainder protected for engineering that reduces future operational work. The specific number is arbitrary. The mechanism underneath it does the work, because naming a ceiling makes the reduction effort visible and gives someone grounds to defend it when the queue is loud.

A planning support team without that protection converges on a stable and unhappy state. The queue is answered, nothing that generates the queue is fixed, and next year's queue is the same size with more items in it.

Hypercare exit deserves the same treatment. It usually ends on a date written into a statement of work, which is a decision about a contract. The conditions worth using instead are measurable: ticket rate per planner per week has fallen across three consecutive cycles, no severity-one incident in the last two cycles, every severity-two has a written root cause rather than a workaround, and the publication fallback has been exercised at least once with the internal team running it.

On-call needs the same honesty. If the chain finishes at 06:00 and the plan is due at 07:00, there is one hour of slack. Nobody awake in that hour means recovery begins when the first person reads their email, and the deadline has already gone. Either fund the hour, or move the deadline and tell the business it moved.

Where this stops

A support model handles the queue it is given. It cannot repair a mismatch between the planning method and how the business actually decides. Where the proportion of tickets saying the recommendation is wrong stays flat cycle after cycle, the problem is upstream of support, in the model, the segmentation or the training, and E5 covers how that quietly becomes disuse.

Ticket metrics also reward the wrong behaviour if you only watch closure time. Time to close improves when a team stops looking for root causes and starts applying workarounds, and the workaround is faster every single time. Pair it with a recurrence measure: the share of this month's tickets whose root cause has been seen before. A rising recurrence rate with falling closure times is a team being managed into a corner.

And the model has to fit the installation. Three thousand item-locations, one planner and a weekly cycle cannot sustain a rota, and pretending otherwise produces a document nobody follows. The honest version there is a later publication deadline, a documented manual path, and an accepted risk that a Friday night failure is fixed on Monday.

The first quarter after go-live also tells you very little about steady state. Volume is high, the mix is heavily weighted to explanation, and both settle over two or three cycles. Size the permanent team on the third quarter rather than the first.

Pull the last ninety days of tickets and label each one incident, explanation or change. The split tells you which of the three jobs you have actually been staffing, and it takes an afternoon.

More on this in systems on the DemandLabs platform.

See DemandLabs on Your Own Data

A 45-minute walkthrough with a solution architect.