Lesson 4 of 6
Testing planning results: tolerances and parallel runs
10 min
Listen to this lesson
A goods movement has one correct result. A planning run does not. Two planning engines given the same inputs produce different forecasts and supply plans because they use different algorithms, buckets and rounding. Even the same engine can give a different result after a release upgrade. So planning tests compare outputs with tolerances, explain every difference outside them, and use parallel runs where old and new systems plan the same period from the same data.
Designing a parallel run
- Freeze the inputs. Take the history, stock, open orders and master data at one cut-off (Contoso: month-end, 18:00) and load the same snapshot into APO and IBP.
- Align settings that should be equal. Horizons, buckets, units, calendars and lanes; differences here are setup errors, not results.
- Run both cycles. APO DP and SNP as usual; IBP forecast, consensus and supply heuristic as designed.
- Compare at agreed levels. Network total, plant, product family, then top products; never cell by cell first.
- Classify every difference outside tolerance. Input difference, design change, engine difference or defect.
- Review with planners. Sunita and Jonas accept, reject or ask for analysis; decisions are logged.
Setting tolerances
Tolerances are business decisions, set with planners before the first run and written into the test plan. They tighten at higher levels, because errors cancel out in totals and because aggregate numbers drive S&OP decisions.
| Key figure and level | Tolerance (illustrative) | Rationale |
|---|---|---|
| Released demand, network total per month | ±1% | Drives S&OP and finance; must reconcile closely |
| Consensus demand, plant per month | ±3% | Capacity decisions are taken per plant |
| Statistical forecast, product family per month | ±5% | Model differences expected; families matter for S&OP |
| Statistical forecast, top 20 products per month | ±10% or explained | Single products vary most; explain every miss |
| Supply split Toledo, Detroit share | ±3 percentage points | Contract and capacity commitments |
| Capacity load ASSY-10, per month | ±2% | Overload decisions depend on it |
| Forecast accuracy (weighted MAPE), last 12 months | IBP no worse than APO by more than 1 point | The new model must not be worse |
Contoso's tolerance table for the planning parallel runs.
Worked example: the March parallel run
Released demand for CMC-ALT-100, network total, March: APO 9,840, IBP 9,910, a difference of 70 or 0.7%, inside ±1%. At plant level, Detroit: APO 6,880, IBP 7,150, +3.9%, outside ±3%. Drilling down, aftermarket demand explains +230 of the +270: APO's history included credit memos as negative quantities in two months, IBP's did not. Sunita classifies it as an input difference that favors IBP, because IBP uses the agreed, cleaner history. She accepts with reason code "history correction". The remaining +40 is within the product-level tolerance.
| Level | APO | IBP | Difference | Result |
|---|---|---|---|---|
| Network, March | 9,840 | 9,910 | +0.7% | Within ±1% |
| Detroit, March | 6,880 | 7,150 | +3.9% | Outside ±3%: classified |
| Detroit aftermarket | 1,770 | 2,000 | +13.0% | History correction, accepted |
| Chicago, March | 2,960 | 2,760 | −6.8% | Outside: under analysis |
Illustrative course data: March comparison and its classification.
| Difference class | Example at Contoso | Action |
|---|---|---|
| Input difference | Credit memos in APO history | Fix the load or accept the cleaner source |
| Design change | Toledo split now set by lane quota | Planner accepts with reason |
| Engine difference | IBP heuristic buckets monthly, APO weekly | Explain; tune if material |
| Defect | Chicago capacity loaded in hours, not pieces | Raise, fix, rerun |
Testing PP/DS by shadow scheduling
Detailed schedules cannot be compared with simple percentages. Contoso ran shadow scheduling for ASSY-10: for six weeks, the Detroit scheduler scheduled in APO as usual, while the same orders were scheduled in embedded PP/DS in the test system from the same snapshot. The comparison used measures that matter on the shop floor: number of late orders, total setup time, load per shift, and how many orders the scheduler had to move by hand. Tolerances were set per measure, for example no more late orders than APO and setup time within 5%.
| Measure (ASSY-10, week 3) | APO | Embedded PP/DS | Tolerance | Result |
|---|---|---|---|---|
| Late orders | 2 | 1 | No more than APO | Pass |
| Total setup time (hours) | 14.0 | 14.5 | ±5% | Pass (+3.6%) |
| Max load per shift | 96% | 101% | ≤100% | Fail: overload on Thursday late shift |
| Orders moved by hand | 6 | 9 | No more than APO | Fail: under analysis |
Illustrative course data: shadow scheduling results for one week.
The Thursday overload turned out to be a defect: the resource's shift calendar in S/4HANA lacked a planned maintenance break that APO had in its own resource. That is a typical finding, because embedded PP/DS reads master data from S/4HANA directly, so S/4HANA master data quality now decides scheduling quality.
What would you do?
Chicago is −6.8% in the March comparison, outside the ±3% tolerance. Jonas suspects IBP is right because Chicago lost a customer. What should Priya do?
- Planning outputs differ by design; compare with tolerances, not exact values.
- Freeze one snapshot for both systems and record its timestamp.
- Compare top down: network, plant, family, product.
- Classify every exception: input difference, design change, engine difference or defect.
Check your understanding · not graded
Lesson check
1. Why are tolerances tighter at the network level than at product level?
2. What is the first step of a parallel run?
3. Detroit is +3.9% because APO history included credit memos. How is this classified?
4. A difference is outside tolerance. What does that mean?
5. Chicago capacity was loaded in hours instead of pieces. Which class is it?