Lesson 5 of 5
KPIs driving testing
9 min
KPIs and testing usually live in different worlds: the business watches dashboards, the test team runs scripts. A process owner can connect them. KPI trends show where the process is fragile, and fragile areas deserve the most regression testing. In return, regression testing should protect the KPIs that matter.
From KPI regression to targeted tests
- KPI movesTouchless invoice rate drops from 66% to 52% after a feature pack
- LocateVariant or exception analysis: price blocks up for one supplier group
- HypothesisA change in pricing or tolerance behavior, or migrated data issue
- Root causeReproduce in QAS; compare configuration and notes in the release
- Fix and testCorrect, then add a regression scenario that would have caught it
The last step is the important one. Every KPI regression that a system change caused is a gap in the regression set. Add a scenario that reproduces it, with an expected result, so the next release cannot cause the same problem unnoticed.
Prioritizing regression by KPI risk
For Contoso, O2C billing and DSO are high impact; P2P invoice matching has been volatile since go-live. Both get the deepest regression for the next upgrade, including price variance and partial receipt scenarios for Lakeside Components.
KPIs as post-release checks
After every release, compare the key KPIs with the weeks before. This is a cheap, business-level safety net that catches what scripted tests miss. Agree thresholds in advance, for example "touchless invoice rate down more than 5 points for two consecutive weeks triggers investigation," so the response is automatic rather than debated.
| KPI | Post-release threshold (example) | First place to look |
|---|---|---|
| Touchless invoice rate | Down 5 points for 2 weeks | Price and quantity blocks by supplier; tolerance configuration |
| Perfect order rate | Down 3 points for 2 weeks | Credit blocks, incompletion, delivery dates |
| Days to close | Up 1 day in the first close after release | Close task durations; failed jobs; reconciliation differences |
| Manual journal entries | Up 20% in the first close after release | Automatic postings that failed and were replaced manually |
Example post-release KPI thresholds for Contoso.
KPIs also help decide where not to spend effort. If a process has been stable and low impact for several releases, a light smoke set and the KPI watch may give enough confidence, freeing testers to go deeper where the KPIs show fragility. That is a defensible, evidence-based way to size regression, much better than retesting everything every time or testing whatever the last incident touched.
Share the KPI watch with the test lead. When Priya can see that perfect order rate dipped for plant 1720 after the last release, she can check whether her regression set covers Chicago deliveries properly, and the next release benefits.
Closing the loop with governance
- Before releaseSelect regression by KPI riskTest lead and process owner agree depth per process
- ReleaseExecute and sign offRegression evidence reviewed; known risks accepted
- Weeks 1–2KPI watchThresholds checked weekly by the process owner
- Month 1Release reviewKPI movements explained; new scenarios added for any escaped defect
What would you do?
After an upgrade, Contoso's regression tests all passed, but two weeks later manual journal entries are up 35%. What does this most likely mean and what should Priya do?
- A KPI regression after a change is a gap in the regression set; add a scenario that would catch it.
- Set regression depth by KPI impact and volatility.
- Use KPIs as a post-release safety net with agreed thresholds and owners.
- Sign up for the KPI watch when you sign off a release.
Check your understanding · not graded
Lesson check
1. What should follow every KPI regression caused by a system change?
2. Which processes deserve the deepest regression?
3. Why agree post-release KPI thresholds in advance?
4. Manual journal entries rose 35% after an upgrade. What is a likely cause?