Lesson 3 of 6
SLAs and KPIs in contracts
17 min
Listen to this lesson
What a contract measures is what a partner optimizes. Pay per test case executed and you get more test cases executed. Reward defects found and you get more defects, mostly trivial. Measure hours and you get hours. The skill in writing testing contracts is choosing measures that reward what the TCoE actually wants: protection from escapes, speed, stable automation and trustworthy evidence.
Measures that drive the wrong behavior
| Measure | What it rewards | Better alternative |
|---|---|---|
| Price per test case executed | Splitting tests, running low-value tests, ignoring risk | Price per regression cycle for an agreed, risk-ranked scope |
| Defects found | Trivial and duplicated defects | Escapes attributable to regression scope, by severity |
| Automated scripts created | Many brittle scripts | Share of critical processes automated and stable |
| Hours or FTEs delivered | Effort, whatever the result | Service levels on cycle time and repair turnaround |
| Partner-reported pass rate | Optimistic status | Pass rate from SAP Cloud ALM, with sampled evidence |
Contoso's service levels
| Service level | Definition | Target | Consequence |
|---|---|---|---|
| Regression cycle time | Working days from entry criteria met in CMR to signed results in SAP Cloud ALM | 10 working days in year 1; 5 as the target once automation matures | Service credit |
| Automation repair turnaround | Working days to repair automated tests broken by an SAP or Contoso change | 3 working days for critical processes | Service credit |
| Automation stability | Failures caused by the script, not the system, as a share of all failures in a cycle | Under 5% | Service credit |
| Evidence quality | Sampled passed results with complete evidence, per cycle | 95% or better | Service credit; sample size doubles next cycle |
| Knowledge currency | Runbooks and test documentation updated after a change | Within 10 working days | Reported; repeated misses escalate |
| Key personnel continuity | Named key roles replaced only with notice and overlap | 4 weeks' overlap | Reported; breach escalates |
Contoso's service levels for the managed regression and automation service. The at-risk outcome component is added in year 2.
Two contract details matter as much as the targets. First, the clock starts when Contoso meets its own entry criteria: the build in CMR, the scope list from Luis, data and environments ready. The partner cannot be held to the service level when the system arrived late. Second, measures are taken from Contoso's systems: SAP Cloud ALM for execution and defects, the service management tool for incidents, never from the partner's own spreadsheets. The partner can and should check the numbers; it should not produce them.
Service credits, earn-back and the outcome component
Service credits reduce the monthly fee when a service level is missed. They are not meant to recover Contoso's losses, only to make misses visible and costly enough to fix. Contoso caps them at 10% of the monthly fee and allows earn-back: if the missed service level is met for the next three months, the credit is returned. Earn-back rewards recovery, which is what Contoso actually wants. From year 2, the outcome component puts another 10% at risk against two measures from sap-tcoe-406: severity 1 and 2 escapes attributable to regression scope (a scenario that should have been in the pack, or a test that ran and missed it), and automation stability over the half-year.
Pricing the service unit
A managed service needs a unit of price, and the unit shapes behavior as much as the service levels do. Contoso prices its managed regression service per regression cycle for an agreed, risk-ranked scope, with a fixed monthly base for automation maintenance and first-level triage. Scope changes go through a simple change note: if Luis adds a value stream to a cycle, the partner quotes the extra effort from an agreed rate card before the cycle starts. Two pricing traps are worth naming. Pricing per script maintained rewards keeping dead scripts alive; Contoso retires scripts through the TCoE's quarterly pack review instead. Pricing per defect triaged rewards splitting failures; triage is part of the base fee.
| Element | Contoso choice | Why |
|---|---|---|
| Regression execution | Price per cycle for the agreed scope | Rewards finishing the agreed scope well, not running more tests |
| Automation maintenance and triage | Fixed monthly base | Removes any reason to inflate scripts or failures |
| Scope changes | Change note priced from a rate card before the cycle | Keeps the cycle fee honest when scope grows |
| Peaks such as stack updates | Capacity call-off at agreed day rates | Flexible capacity without renegotiating the service |
How Contoso prices the managed service. Each unit was chosen for the behavior it encourages.
KPIs that are not in the contract
Not every useful measure belongs in the contract. Contoso tracks others with the partner, such as defect rejection rate, data recipe failures and turnover in the offshore team, but uses them for improvement, not payment. Putting too many measures in the contract makes the partner manage the contract instead of the service, and every new measure becomes a negotiation. Keep contractual service levels to five or six, and review the list each year.
What would you do?
The partner proposes a service level of "98% of planned test cases executed per cycle." What should Priya say?
- Volume measures (test cases, defects found, scripts, hours) reward activity; replace them with outcome and service measures.
- Start service level clocks when the customer meets its entry criteria, and measure from the customer's systems.
- Cap service credits and allow earn-back; add an outcome component only with a written attribution rule.
- Keep five or six contractual service levels; track other KPIs for improvement, not payment.
Check your understanding · not graded
Lesson check
1. Why is price per test case executed a poor basis for a testing contract?
2. When should the clock for the regression cycle time service level start?
3. What is the purpose of earn-back in Contoso's contract?
4. Why are the service level measures taken from Contoso's systems?