Lesson 1 of 5
Metrics that inform decisions
11 min
Listen to this lesson
Test metrics exist for one reason: to help someone make a decision. Should we start the next cycle? Where should we put the extra testers? Are we ready to go live? A number that does not change any decision is decoration. This lesson sorts SAP test metrics into four families and shows what each can and cannot tell you.
Four families of metrics
| Family | Useful SAP metrics | What it cannot tell you |
|---|---|---|
| Progress | Test cases planned, executed, passed, failed, blocked per cycle and value stream; daily burn-down | Whether the right things were tested or whether they work well |
| Coverage | Share of in-scope process steps, requirements and high risks with at least one executed test; end-to-end scenarios completed | Whether the tests were deep enough |
| Quality | Open defects by severity, priority and process; new vs closed per day; reopen rate; average age of open high-severity defects | Whether undetected defects remain in untested areas |
| Risk | High and medium risks by residual status; accepted risks with owners | Anything, if the risk register is not maintained |
What each family measures, and its blind spot.
Vanity metrics
A vanity metric looks impressive and supports no decision. SAP programs produce plenty of them:
- Total number of test cases. 4,000 test cases sounds thorough. If 3,000 are trivial display checks and the payment run has two, it is the opposite.
- Overall pass rate without coverage. 98% passed means little if only 40% of planned critical-process tests have run.
- Defects found as a success measure. It rewards logging many minor defects and says nothing about the system's readiness.
- Hours spent testing. Effort is an input, not a result.
| Vanity metric | Why it misleads | Report this instead |
|---|---|---|
| Total test cases | Volume says nothing about what was covered | Critical-process end-to-end scenarios executed and passed |
| Overall pass rate | High rate can hide untested critical processes | Pass rate paired with execution rate for critical processes and high risks |
| Defects found | Rewards logging many minor defects | Open critical and high defects by process and age |
| Hours spent testing | Effort is an input, not a result | Burn-down of remaining planned tests against days left |
Defining metrics precisely
Write a one-line definition for every metric in your test strategy. "Pass rate" could mean passed divided by planned, passed divided by executed, or passed at first attempt. At Contoso Motor Co., Priya defines pass rate as passed tests divided by executed tests in the current cycle, and always reports it next to execution rate, executed divided by planned. She defines a critical defect using the severity scale agreed with Grace and the process owners: a defect that stops a critical process or produces wrong financial results with no workaround.
Where the data comes from
Metrics are only as good as the records behind them. Test management in SAP Cloud ALM covers test planning, execution with history, defects, and analytics with traceability back to requirements and processes; SAP Solution Manager offered similar test suite capabilities. Whatever tool your project uses, the rule is the same: testers record results and defects in the tool as they go, not in spreadsheets afterward, so reports can be produced from the same data at any time.
What would you do?
Two weeks before Contoso's go/no-go, Grace receives a slide: "3,912 test cases executed, 97% pass rate." She asks Priya whether that means the program is ready. What is the best answer?
Count what matters in SAP
SAP test libraries mix very different units: a single-transaction functional test, a forty-step end-to-end scenario from sales order to cash receipt, an interface test, a role test. Counting them all as "one test case" distorts every percentage. Report end-to-end scenarios separately from component tests, and weight coverage by process rather than by test count. At Contoso, "order to cash: 18 of 20 end-to-end scenarios passed, both open ones in fleet pricing" is far more informative than "212 of 240 test cases executed", even though both are true.
Also report by organizational scope when it matters. If plant 1720 Chicago went live with a different warehouse setup than plant 1710 Detroit, a combined number can hide a plant that has barely been tested.
- Use four families: progress, coverage, quality and risk. Each has a blind spot.
- Avoid vanity metrics such as raw test counts or pass rate without coverage.
- Define each metric precisely and pair progress numbers with coverage or quality numbers.
- Record results and defects in the test management tool as you go so reports come from one source.
Check your understanding · not graded
Lesson check
1. Which metric family answers "how much do we know?"
2. Why is a 98% pass rate misleading on its own?
3. Which is a vanity metric?
4. Why define "pass rate" in writing?