Lesson 2 of 6
Baselining the cost of quality
17 min
Listen to this lesson
A baseline is a measured picture of what quality costs today. It is the number every benefit is measured against, so it must be built from Contoso's own data and survive a skeptical finance review. ISTQB's Advanced Test Management syllabus (CTAL-TM v3.0) frames the cost of quality in four categories, following Feigenbaum's classic prevention, appraisal and failure model. Prevention plus appraisal is often called the cost of good quality, and the two failure categories together the cost of poor quality. The syllabus also notes why the model persuades: appraisal and internal failure usually cost far less than external failure. Use those categories, because finance teams recognize them.
The four categories, translated to SAP
| Category | Meaning | SAP examples to count |
|---|---|---|
| Prevention | Spending that stops defects being made | Design reviews of functional specs, ATC and code reviews, tester and key-user training, release assessment |
| Appraisal | Spending to find defects | Regression, enhancement and UAT test execution, test preparation, test data creation, environment checks |
| Internal failure | Fixing what testing finds | Developer fix time, retests, repeated transports, delayed imports |
| External failure | What escaped defects cost after release | Incident handling, credit memos and re-billing, expedited freight, manual workarounds, audit findings |
The cost of quality categories from CTAL-TM v3.0, mapped to SAP activities.
Where the data lives
Most of the data already exists; it just sits in different systems. Collect it for a fixed period, such as the six months after hypercare, and annualize it. Use one blended hourly rate agreed with finance for internal and partner time so that nobody argues about rate cards later. Contoso used $95 per hour.
- SAP Cloud ALM: test plans, test case counts, execution results and defects per release, with dates. Effort is not recorded there, so pair it with timesheets.
- Timesheets and partner invoices: hours booked to testing per change, including partner call-offs and premium short-notice rates.
- The IT service management tool: production incidents, with the change or release that caused them. If incidents are not linked to releases, fix the tagging first, or escaped defects will look artificially low.
- Finance: credit memos, re-billing and expedited freight caused by system errors. Ask Dan, the finance process owner, to confirm the amounts.
- Line managers: key-user hours spent testing. A short survey per release is acceptable if you say so.
Contoso's baseline
Priya measured the six months after hypercare and annualized them. Effort is converted at the agreed $95 per hour. External failure was estimated with Dan, incident by incident for severity 1 and with average values for lower severities.
| Line | Basis (annualized) | Hours | Cost |
|---|---|---|---|
| Prevention | Some spec reviews and tester training | 380 | $36,100 |
| Appraisal: release regression | 2 events × 1,400 h | 2,800 | $266,000 |
| Appraisal: enhancements | 120 changes × 30 h | 3,600 | $342,000 |
| Appraisal: notes and patches | 12 checks × 60 h | 720 | $68,400 |
| Appraisal: mobilization and rework of stale assets | 2 events × 300 h | 600 | $57,000 |
| Internal failure | 640 defects found in test × 3 h fix and retest | 1,920 | $182,400 |
| External failure | 1 severity 1 ($180,000), 9 severity 2 ($12,000 each), 36 severity 3 ($1,500 each) | n/a | $342,000 |
| Total cost of quality | $1,293,900 |
Contoso's annualized cost of quality baseline. Figures are illustrative course data.
Two ratios from the baseline matter as much as the total. Prevention is under 3% of the cost of quality, and external failure is over a quarter of it. Defect leakage is also visible: 640 defects found before release and 46 escaped gives a defect detection percentage of 640 ÷ 686, about 93%. Those three numbers tell the sponsor where the money is: too little is spent preventing defects, and too much is spent after they escape.
Making the baseline survive scrutiny
- State the period, the rate and every estimate, and mark which lines are measured and which are estimated.
- Do not count the same hour twice: a retest is internal failure, not appraisal.
- Leave out what you cannot defend. A smaller, solid baseline beats a large one finance picks apart.
- Keep the workbook. The same definitions will be used to report value every quarter, so a change in the number means a change in reality.
What would you do?
Priya's baseline shows only 12 production incidents linked to releases in six months. The service desk lead mentions that most incidents are logged without a change reference. What should Priya do?
- Use the four cost of quality categories (prevention, appraisal, internal failure, external failure); finance recognizes them.
- Collect one period from Cloud ALM, timesheets, incidents, finance and line managers, at one agreed blended rate.
- Link incidents to releases, or escaped defects and external failure will be understated.
- Report the ratios as well as the total: Contoso spent 3% on prevention and 26% on external failure.
- Mark measured versus estimated lines and leave out what you cannot defend.
Check your understanding · not graded
Lesson check
1. A developer fixes a defect found in SIT, then a tester retests it. Which category is this?
2. Why should the baseline use one blended hourly rate agreed with finance?
3. Contoso found 640 defects before release and 46 escaped. What is the defect detection percentage?
4. Which baseline line is most important to mark as an estimate rather than a measurement?