Lesson 5 of 6
Coverage-against-usage reporting
10 min
Listen to this lesson
Coverage reports in SAP programs usually count test cases: planned, executed, passed. Coverage against usage asks a different question: what share of the business's real activity is represented by tests that passed? It is the number Dan asked for in lesson 1, and it belongs in every go/no-go pack next to the pass rate.
Four measures
| Measure | Definition | Contoso O2C (illustrative) |
|---|---|---|
| Case coverage | Share of cases whose variant is represented by a passed scenario | 89% |
| Value coverage | Share of net value in variants represented by a passed scenario | 91% |
| Top-variant coverage | How many of the top N variants by volume have a passed scenario | 6 of 6 |
| Uncovered high-volume variants | Variants above an agreed threshold (for example 1% of cases) with no scenario | 1 (manual price override, 1.4%) |
| Dead tests | Test cases mapped to variants with no cases in the period | 6 retired (consignment) |
Measures that turn a test count into a statement about real usage.
Building the report
- Refresh the mining data for the agreed period, with the same case definition and filters as before.
- Export the variant list with case counts and value per variant.
- Export test results from SAP Cloud ALM with each test case's variant IDs (lesson 3).
- Join them: a variant is covered when at least one mapped scenario passed in the current cycle.
- Calculate case coverage, value coverage and the list of uncovered variants above the threshold.
- List dead tests: scenarios whose variants had no cases.
- Review the result with the process owner before it goes into the pack.
Coverage counts only passed scenarios. A scenario that ran and failed, or was blocked, does not cover its variants. Show both views if useful: "planned coverage" (variants with a scenario) and "achieved coverage" (variants with a passed scenario). The gap between them is the open risk that the decision group is being asked to accept.
For the conversion itself, the usage evidence comes from ECC (EP1), and the coverage report answers a sharper question: did the converted system pass tests that represent how Contoso actually ran in ECC? That is a useful statement for Grace's go/no-go 2 on the cutover Sunday, where Dan's reconciliations and the smoke tests are signed. The smoke tests themselves can be chosen to hit the top variants first, so the few hours of validation cover the most business.
Coverage for automation and over time
The same join answers a question Ravi cares about: what share of real cases is covered by automated scenarios? It is a better automation KPI than "number of scripts", because it rewards automating the paths the business uses most. Track it per release alongside case and value coverage, so the trend shows whether the regression set is keeping up with the business.
| Release (illustrative) | Pass rate | Case coverage | Value coverage | Automated case coverage |
|---|---|---|---|---|
| Conversion regression (CMQ) | 97% | 71% | 74% | 38% |
| First quarterly release (CMR) | 98% | 89% | 91% | 55% |
| Second quarterly release (CMR) | 99% | 90% | 92% | 68% |
Illustrative course data: Contoso's order-to-cash trend. The pass rate barely moves; the coverage figures show the real improvement.
Be careful what you reward. If coverage becomes a target, teams may map tests to variants loosely to raise the number. Keep the mapping reviewed by the key user, and sample a few mappings each release: does test case O2C-03-A really exercise the credit release that variant V2 contains?
In the go/no-go pack
Contoso's readiness pack for each release has one slide per end-to-end process: pass rate, case coverage, value coverage, uncovered variants above 1% with owner decisions, and policy-floor paths with status. For order-to-cash in the example: pass rate 98%, case coverage 89%, value coverage 91%, one uncovered variant (manual price override, 1.4% of cases) accepted by Maria with a documented workaround and a test added for the next release. That one line tells the decision group more than the pass rate alone.
- Pass rateExecuted and passed tests, as before
- Case coverageShare of real cases represented by passed scenarios
- Value coverageShare of order value represented
- GapsUncovered variants above threshold, each with an owner decision
- Policy floorStatus of the rare but critical paths
What would you do?
The order-to-cash pass rate is 99%, but value coverage is only 72% because the credit-block scenario was blocked by a defect for the whole cycle. The decision group meets tomorrow. What should Priya report?
- Report case coverage, value coverage, top-variant coverage, uncovered variants and dead tests.
- Only passed scenarios count as coverage; show planned versus achieved.
- Join mining data and test results on variant IDs kept on test cases.
- Pair the pass rate with coverage on the main go/no-go slide.
Check your understanding · not graded
Lesson check
1. When does a variant count as covered?
2. What does value coverage measure?
3. What causes coverage to jump for reasons unrelated to testing?
4. Who reviews the coverage report before the decision group?