Lesson 5 of 5
Classifying failures
9 min
Listen to this lesson
A failed test case is a question: is the system wrong, or is something around it wrong? Getting the answer quickly decides who works on it, whether testing can continue, and whether the defect counts against the release. This lesson gives a practical classification: environment, data, configuration, code, plus a fifth that testers must be honest about: the test itself.
| Class | Typical signs | Usual owner | Example at Contoso |
|---|---|---|---|
| Environment | Fails for many, unrelated scenarios; system log errors; RFC or job problems; timing-dependent | Basis (Sam) | Interface job not scheduled after the CMQ refresh |
| Data | Fails only for certain master data or documents; works with other data | Data owner or test data team | Customer in company code 1910 missing a payment term after the Wave 2 mock load |
| Configuration | Fails consistently for a process variant; recent customizing transport; message points to settings | Functional consultant | Output determination missing for a new billing type |
| Code | Dump in Y/Z code; logic error reproducible with good data and settings | Development (Kenji) | Custom pricing routine divides by zero for zero-quantity items |
| Test | Expected result wrong, outdated script, wrong user or data in the test | Test team (Priya) | Script still expects the ECC transaction screen sequence |
A decision path
- Do unrelated tests fail at the same time, or does SM21 show system errors?Yes → environment: route to Basis and pause dependent tests · No → next question
- Does the same step work with different master data or documents?Yes → data: fix or replace the test data · No → next question
- Is there a dump in custom code or a logic error with correct data and settings?Yes → code: route to development with the dump and data · No → next question
- Does the result contradict the configured settings or a recent customizing change?Yes → configuration: route to the functional consultant · No → review the test case: expected result, user, data and steps
Evidence for each class
Classification is a hypothesis, so attach the evidence that supports it: the dump for code defects, the SM21 extract for environment issues, the master data comparison for data issues, the customizing setting and transport for configuration. That evidence lets the receiving team start working instead of re-investigating, and it gives the test lead a defensible record if the classification is later disputed. A defect that bounces between teams three times usually lacked evidence on day one.
Mixed causes
Real failures do not always fit one box. A custom program may dump only because a mock load produced an unexpected blank field: that is a data issue that reveals a code weakness, and both deserve a ticket. Classify by what must change for the test to pass now (fix the data), and raise a second defect for the robustness gap (the program should handle the blank). Keep the link between the two so the release decision sees the whole picture.
- Minute 0Test failsAutomated run or tester reports the failure with step, data and message
- Minutes 0–15First lookTester or automation engineer checks ST22, SM21 and whether other tests failed
- Minutes 15–60ClassifyInitial class assigned with evidence; defect created and routed
- Same dayConfirmOwning team confirms or reclassifies with its own evidence
- After fixRetestOriginal scenario re-run; class and root cause recorded for metrics
What would you do?
Overnight, 40 of 300 automated tests failed in CMQ across finance, sales and purchasing. All 40 failures happened between 01:00 and 01:20, and SM21 shows database connection errors in that window. How should Priya classify them?
- Classify failures as environment, data, configuration, code or test.
- Unrelated failures together, plus system log errors, point to environment.
- Attach evidence to every classification so the owner can act immediately.
- Track initial and final class; use the mix at sign-off to judge result reliability.
Check your understanding · not graded
Lesson check
1. Many unrelated tests failed in the same 20-minute window. What class is most likely?
2. A step fails for one customer but works for others. What class comes first to mind?
3. What should accompany a defect classified as code?
4. Why record both the initial and the final defect class?