Lesson 3 of 5
Failure analysis
18 min
Listen to this lesson
After a release, a suite of 250 tests can produce 40 failures overnight. Analyzing each from scratch takes days; the release window does not have days. Structured failure analysis turns a pile of red into a short list of causes, each with an owner, within the first morning. The ISTQB syllabus describes the core of it: inspect test logs, performance data, setup and teardown; run a few isolated tests; and remember the cause can be in the test, the system under test, the framework, the hardware or the network.
Capture the right evidence every run
Analysis is only as good as the evidence the run left behind. Build evidence capture into the framework so nobody has to reproduce a failure just to see what happened.
On the SAP side, the usual places to look are well known to any Basis or ABAP team: short dumps in ST22, the system log in SM21, application logs in SLG1, background jobs in SM37, update errors in SM13, and for OData services the gateway error log /IWFND/ERROR_LOG. A framework that records the system, client, user and exact timestamps of each failed step makes these searches quick.
Group by signature first
Before analyzing individual tests, group failures by signature: the failing step, the error message and the object involved. Forty failures often collapse into five or six signatures. One signature with twenty tests is almost never twenty defects; it is usually a shared helper, a changed navigation path, a missing role or an environment problem.
- CollectExport all failures with step, message, object, timestamp and run id.
- GroupCluster by signature; sort clusters by size.
- Check run-wide signalsEnvironment checks, pool levels, refresh or upgrade events in the window.
- Analyze one per clusterPick the clearest example; reproduce in isolation if needed.
- Classify and assignData, environment, test or defect; assign an owner and a due time.
- VerifyRerun the affected cluster after the fix; close or reopen.
| Signature at Contoso after the feature pack stack | Tests | Classification | Owner |
|---|---|---|---|
| Launchpad tile "Create Sales Orders" not found | 18 | Test: tile title changed in the new catalog | Automation engineer |
| Authorization error on new app in test user role | 9 | Environment: role not updated in CMQ | Security team |
| Field "Incoterms Location 1" mandatory | 6 | Defect: custom validation clashes with new standard field logic | Kenji's team |
| Stock shortage for CMC-RTR-200 | 4 | Data: replenishment job failed during upgrade | Automation engineer |
| Timeout at goods issue post | 3 | Unclear: reproduce in isolation | Automation engineer with Basis |
Example triage board: 40 failures, five signatures.
Read failures and passes
The ISTQB syllabus also asks you to check unexpected passes, and to verify that the assertions are in place, because missing assertions give inconclusive results. After an upgrade, a test that suddenly passes faster than ever may have skipped a step because a popup no longer appears; check that it still asserts the business outcome. When a release changes behavior on purpose, the right fix is to update the expected result, with the process owner's agreement, and record why.
- Did the system behave as the new release intends?Yes → update the test and its expected result, with process owner agreement · No → continue
- Does the failure reproduce manually with valid data?Yes → log a defect with the evidence package · No → continue
- Is the environment or data at fault?Yes → route to Basis, security or the data owner · No → fix the test itself
What would you do?
After the upgrade of CMQ, six order-to-cash tests fail because a new mandatory field appears on the sales order screen. SAP's release information describes the field as a deliberate change. Maria confirms the business will fill it. What is the right action?
- Capture test, screen, service, data and system evidence on every run.
- Group failures by signature before analyzing individual tests.
- Check run-wide signals such as environment health, pool levels and upgrade events first.
- Time-box analysis per cluster and escalate with an evidence package.
- Intended changes mean updating the test and expected result with process owner agreement.
Check your understanding · not graded
Lesson check
1. Twenty tests fail with the same message after an upgrade. What is the most likely conclusion?
2. Where would an analyst look for ABAP short dumps that occurred during a failed run?
3. A test now passes in half its usual time after an upgrade. What should the team check?
4. A release changes a behavior on purpose and Maria agrees the business will adapt. What happens to the failing tests?