Lesson 6 of 7
Measuring and improving
9 min
Listen to this lesson
A framework that is never measured slowly turns into ritual. The final principle, that the framework improves, needs data: how many problems the sign-off process missed, how long it takes, how often evidence is sent back, and how risk acceptances behave after go-live. This lesson defines a small set of metrics and the loop that uses them.
Core metrics
| Metric | Definition | What it tells you |
|---|---|---|
| Escaped defects | Defects found in production that the agreed test scope should have caught, per process | Where sign-off missed real problems |
| Defect leakage rate | Escaped defects divided by all defects found in testing and production for the same scope | How much of the problem testing finds before go-live |
| Sign-off cycle time | Time from end of test execution to signed acceptance, per process | Whether the decision process is a bottleneck |
| Evidence rework rate | Share of submissions returned for missing or weak evidence | Whether the evidence standard is understood and practical |
| Gate first-time pass rate | Share of gates passed without conditions on the first review | Whether criteria are realistic and teams prepare |
| Risk acceptance health | Open acceptances by age and severity; acceptances past expiry; issues that occurred | Whether accepted risk is being worked off or piling up |
A small set of metrics for a sign-off framework.
Escaped defects are the most important and the most argued about. Define them carefully: a production defect counts as escaped only if the scenario was in scope and the test should reasonably have caught it. A defect caused by a business process nobody asked to test is a scope gap, which is also useful to know but is a different finding.
- Was the scenario in the agreed test scope?No → scope gap: review how scope was derived · Yes → next question
- Was there a test case that covered it?No → coverage gap: improve test design · Yes → next question
- Did the test pass with evidence?Yes → evidence or data gap: the test did not prove what it claimed · No → next question
- Was the failure risk-accepted?Yes → accepted risk occurred: review the acceptance · No → gate gap: why did it pass?
Metrics for continuous release
After go-live, releases become routine, and delivery metrics from software engineering apply. The DORA research program defines software delivery metrics including change lead time, deployment frequency, failed deployment recovery time and change fail rate, the share of deployments that need immediate intervention such as a rollback or hotfix. For an SAP landscape, read "deployment" as a transport import or release into CMP. A rising change fail rate is a direct signal that release gates are too light.
- MeasureCollect metrics per gate and per release from SAP Cloud ALM and the incident system
- ReviewMonthly in hypercare, then quarterly: Grace, Priya, Dan and process owners
- Find causesClassify escaped defects; read returned evidence; examine slow sign-offs
- Change the frameworkUpdate templates, criteria, decision rights or test scope
- CommunicatePublish what changed and why; train affected roles
What would you do?
Six months after go-live, the change fail rate for transports into CMP has doubled while sign-off cycle time has halved. What is the most likely reading?
| Metric | Definition at Contoso | Quarter 1 after go-live (illustrative) |
|---|---|---|
| Escaped defects | Production defects in tested scope, classified by root cause | 11 (6 in output) |
| Sign-off cycle time | Days from test completion to signed acceptance | 4 days median |
| Expired risk acceptances | Acceptances past expiry without decision | 1, escalated |
| Change fail rate | Share of releases needing immediate intervention | 1 of 9 releases |
A small, paired set of metrics reviewed every quarter.
- Track escaped defects, leakage, sign-off cycle time, evidence rework, gate first-time pass rate and risk acceptance health.
- Classify escaped defects as scope, coverage, evidence or gate gaps to know what to fix.
- After go-live, add release metrics such as change fail rate.
- Pair metrics with counterweights and use them to improve the framework, not to rate people.
Check your understanding · not graded
Lesson check
1. Which metric best shows that sign-off missed real problems?
2. A production defect is in a process nobody included in test scope. How should it be classified?
3. Why pair sign-off cycle time with escaped defects?
4. Six escaped defects were in output because tests only checked that output was sent. What should change?