Lesson 2 of 7
The KPI set
10 min
Listen to this lesson
A TCoE needs few KPIs, precisely defined. Ten is plenty; more than that and nobody remembers them, and each one gets less attention. Contoso's quality council agreed nine, in four families: quality (are we protected?), speed (are we fast?), cost (are we efficient?) and service (do the people we serve get what they need?). They extend the scorecard in sap-tst-401 and use the baseline from the TCoE business case in sap-tcoe-401.
| Family | KPI | Definition | Baseline | Target (24 months) |
|---|---|---|---|---|
| Quality | Severity-weighted escapes per quarter | Production incidents linked to a change, weighted 10 / 3 / 1 for severity 1 / 2 / 3 | About 18 a quarter (baseline year: 1 / 9 / 36 escapes, index 73) | Under 8 |
| Quality | Defect detection percentage (DDP) | Defects found before release ÷ (found before + escaped within 30 days of the import) | 93% (640 ÷ 686) | Above 95% |
| Quality | Critical process coverage | Critical end-to-end processes with maintained regression tests ÷ critical processes | 55% | 100% |
| Speed | Regression cycle time | Working days from the build in CMR to signed regression results | 15 days | 5 days |
| Speed | Change lead time | Median days from transport release in CMD to import into CMP (lesson 3) | 34 days | 20 days |
| Cost | Cost per regression cycle | All hours at $95 plus partner fees and tool run cost, per full regression | 1,400 h, about $133,000 | 700 h |
| Cost | Cost of quality mix | Prevention, appraisal, internal and external failure as shares of the total | Prevention 3%, external failure 26% | Prevention 8%, external failure under 15% |
| Service | Service level attainment | Requests delivered within their published service level ÷ all requests | Not measured | 90% |
| Service | Key-user hours in close weeks | Business tester hours booked to testing in month-end close weeks | 380 hours a year | Zero |
Contoso's nine TCoE KPIs. Regression cycle time has a year-1 service level of 10 working days (sap-tcoe-402); 5 is the target, reached as automation matures. Baselines come from the business case in sap-tcoe-401; the coverage, lead time and key-user baselines are illustrative course data.
Why weight escapes by severity
Counting escaped defects treats a typo on a form like a blocked billing run. A severity-weighted index fixes that. With weights of 10, 3 and 1, Contoso's baseline year (one severity 1, nine severity 2 and 36 severity 3 escapes) scores 10 + 27 + 36 = 73, about 18 a quarter. Report it per quarter and per 100 changes, and write the attribution rules into the definition: an incident counts as an escape when it is linked to the change that caused it, and DDP counts only escapes found within 30 days of that change's import, so every period is judged the same way. Pair the index with DDP, which shows the share of defects caught before production regardless of severity.
A KPI card for every measure
Each KPI gets a one-page card, approved by the quality council and changed only through it. The card is what makes a number trustworthy: when someone asks "how is this calculated?" the answer exists, and it was the same last quarter.
- Name and question: what decision the KPI informs.
- Definition and formula: exact numerator, denominator, window and exclusions.
- Data source: the system of record and the report or extract used.
- Owner: who produces it and who acts on it.
- Frequency: per release, monthly or quarterly.
- Baseline and target: with dates, from the business case where possible.
- Balancing KPI: the measure that would expose gaming of this one (lesson 6).
| Card field | Severity-weighted escapes per quarter (Contoso) |
|---|---|
| Question | Are fewer serious defects reaching production after each release? |
| Formula | 10 × severity 1 + 3 × severity 2 + 1 × severity 3 incidents linked to a change imported into CMP in the quarter; also shown per 100 changes |
| Window and exclusions | Incidents raised within 30 days of the import; excludes incidents closed as user error after process owner review |
| Source | IT service management incidents with the change reference; change records for the change count |
| Owner | Produced by Priya's TCoE; acted on by the quality council |
| Frequency | Quarterly, with a rolling four-quarter view |
| Baseline and target | 73 in the baseline year, about 18 a quarter; under 8 a quarter within 24 months |
| Balancing KPI | Share of incidents without a change reference, and severity changes reviewed by the process owner |
A worked KPI card. The balancing line exposes the two easiest ways to make the number fall without improving quality.
Where the data comes from
Most inputs already exist in SAP Cloud ALM, which Contoso's charter names as the system of record for tests and defects. Its analytics apps include Test Execution Analysis (execution history by status, executed versus remaining manual test actions, distribution by scope or test plan), Defects Reporting, Requirement Traceability and Solution Process Traceability, and Quality Gate Reporting. Escapes come from the IT service management tool, provided every production incident is linked to the release or change that caused it. Effort comes from timesheets and partner invoices, and cost from finance at the agreed blended rate.
- SAP Cloud ALMTest runs, defects, traceability, quality gates per release
- Service management toolIncidents linked to the release or change that caused them
- Timesheets and invoicesInternal and partner hours per release event and service
- Change records and transportsApproval, release in CMD, import into CMP
- KPI workbookFixed definitions, one extract per period, reviewed by Priya
- Quality council packNine KPIs with trend, target and commentary
What would you do?
The council proposes adding "number of automated test cases" as a tenth KPI, because the automation program is the biggest investment. What should Priya advise?
- Keep the KPI set small: nine KPIs in quality, speed, cost and service families.
- Weight escaped defects by severity, fix the attribution window and pair the index with DDP.
- Give every KPI a card with definition, source, owner, frequency, baseline, target and balancing KPI.
- Most data is already in SAP Cloud ALM, the service management tool, timesheets and change records; the incident-to-release link is essential.
Check your understanding · not graded
Lesson check
1. Why does Contoso weight escaped defects 10 / 3 / 1 by severity?
2. What is the purpose of the balancing KPI on a KPI card?
3. Which SAP Cloud ALM app shows executed versus remaining manual test actions over time?
4. Which data link must exist before escaped defects can be measured?