Lesson 6 of 7
Metric pitfalls
8 min
Listen to this lesson
Once a number is used to judge people or to release money, people start managing the number. The economist Charles Goodhart's observation is usually paraphrased as: when a measure becomes a target, it ceases to be a good measure. TCoE metrics are especially exposed, because the people who produce the data (testers, developers, partners) are often the people the data judges. This lesson lists the ways SAP testing metrics get gamed or misread, and the controls that keep them honest.
How SAP test metrics get gamed
| Metric | How it gets gamed or misread | Control |
|---|---|---|
| Escaped defects | Incidents logged without a change reference; severity 2 incidents downgraded to 3; issues reclassified as "user error" | Mandatory release link in the incident record; severity reviewed by the process owner; monthly sample of closed incidents |
| Defect detection percentage | Many trivial defects logged before release to inflate the numerator | Weight by severity or report DDP for severity 1 and 2 separately |
| Pass rate | Flaky automated tests excluded from the denominator; hard tests deferred to "next cycle" | Report excluded tests and deferrals explicitly; pair with coverage of critical processes |
| Regression cycle time | Scope quietly reduced so the cycle finishes faster | Pair with critical process coverage; record every scope reduction with the release manager |
| Service level attainment | Clock stopped by asking requesters to resubmit; "on hold" status overused | Clock rules on the KPI card; audit a sample of requests each quarter |
| Defects found | Testers split one defect into several; defects closed as "works as designed" to clear the backlog | Never use as a performance target; review rejection reasons in triage |
Common ways SAP test metrics are gamed, with the control that exposes each one.
Notice that most of the gaming in the table is not dishonest in intent. A developer who downgrades an incident from severity 2 to 3 may genuinely believe the workaround is fine. A test lead who defers a hard test may believe it is low risk. The problem is that once the metric matters, these judgment calls drift in one direction. Controls exist to make the drift visible, not to accuse anyone.
Misreadings that need no gaming
- Averages hide sites. A combined escape rate can look healthy while plant 1720 Chicago, with its different warehouse setup, has most of the escapes. Break KPIs down by value stream and organizational unit before concluding.
- Small numbers swing. With two release events a year, one bad release doubles the index. Show a rolling four-quarter view alongside the latest figure.
- Survivorship. Automation stability measured only on scripts still in the suite ignores scripts that were retired because they kept failing. Count retirements.
- Correlation is not cause. Escapes may fall because fewer changes were made. Normalize per 100 changes (lesson 1) and check volume first.
- Watermelon reporting. Green outside, red inside: every KPI on target while the process owners report pain. Ask process owners for one sentence each in the pack.
Controls that keep metrics honest
What would you do?
In the week before a release, Priya notices that three severity 2 defects in billing were downgraded to severity 3, which would let the release meet its exit criteria. What should she do?
- When a measure becomes a target, judgment calls drift toward the target; design controls to make drift visible.
- Pair every KPI with a balancing measure, and report reclassifications, deferrals and exclusions explicitly.
- Break down averages by value stream and site, use rolling views for small numbers, and normalize for volume.
- Extract data from systems of record, audit samples quarterly, and never rank individuals with TCoE KPIs.
Check your understanding · not graded
Lesson check
1. Which control best exposes a regression cycle time that improved because scope was cut?
2. Why report the share of defects reclassified as change requests or rejected?
3. Contoso's combined escape rate is on target. What should Priya check before reporting success?
4. With two release events a year, one bad release doubles the escape index. What reporting practice helps?