Lesson 5 of 6
Analyzing results
16 min
Listen to this lesson
A red result is the start of the work, not the end. The value of automated regression is not the run itself but how quickly the team turns failures into decisions: a defect for SAP, a fix to configuration, an update to a custom automate, or a data top-up. The Test Automation Tool gives you an app for that analysis, and a disciplined triage routine makes the difference between a three-week upgrade window that works and one that drowns in noise.
The Analyze Automated Test Results app
SAP Learning describes the Analyze Automated Test Results app as showing results from both post-upgrade tests and customer-specific tests, with graphical dashboards for monitoring. To investigate a failed test plan, you drill down into the specific steps and view screenshots of the error messages. Dashboards can be shared by email link or saved as tiles on the launchpad, which is useful for a test lead who wants the upgrade status on her home page.
- DashboardSpot failed plans and processes in Analyze Automated Test Results.
- Drill downOpen the failed process and the first failed step.
- Read evidenceStep details and the screenshot of the error message.
- ClassifyUpgrade change, configuration, automate, data or environment.
- ActDefect, configuration fix, automate update, data top-up or rerun.
- RecordOutcome and evidence in Cloud ALM for the upgrade sign-off.
Classify every failure
The first failed step tells you where; the screenshot and message tell you why. Classify each failure into one of five buckets before anyone starts fixing. Classification is the key skill, because each bucket has a different owner and a different urgency in the upgrade window.
| Class | Typical evidence | Owner | Action |
|---|---|---|---|
| Upgrade change in SAP standard | Standard process now behaves differently; message new since the upgrade | Test lead with SAP | Check release information; raise an incident if it is a defect |
| Customer configuration | Message caused by your settings, such as a missing determination | Configuration expert | Fix configuration in the test system and rerun |
| Automate out of date | Step cannot find a field or button; screen changed; custom step with ? | Automate owner | Update the custom automate and rerun |
| Test data | No stock, blocked customer, closed period | Automate owner | Top up or correct data; rerun |
| Test user or environment | Logon fails, pre-check fails, integration partner unavailable | Admin or integration owner | Fix user, password, roles or connection |
From result to evidence
Results in the embedded app help the team work; sign-off needs evidence in the project's test management. SAP describes test plans as created and executed in the Public Edition with execution results visible in SAP Cloud ALM alongside manual test plans. Use that: report automated and manual regression together, link defects to the failed runs, and keep the final green rerun as the evidence of record. For each failure, the closing note should say which class it was and what fixed it.
Test data quality
Many failures are data failures in disguise. SAP offers a Test Data Refresh service, described in the same lesson, which copies recent master and transactional data from production to the test system and can depersonalize sensitive data; SAP notes it requires an additional fee. Whether or not you use it, keep a data top-up routine and data that represents your real business, because results are only as meaningful as the data they ran on.
What would you do?
After the upgrade of the test system, eleven of forty processes fail in the what-if. Nine fail at the first step with a logon error; two fail on a changed field in a custom automate. What should Priya do first?
- Analyze Automated Test Results shows post-upgrade and customer test results; drill into failed steps and screenshots.
- Analyze the first failed step; later failures are often consequences.
- Classify each failure: upgrade change, configuration, automate, data, or user and environment.
- Report automated and manual results together in Cloud ALM and keep the green rerun as evidence.
- Data quality drives result quality; keep data replenishable and representative.
Check your understanding · not graded
Lesson check
1. Which app is used to drill into failed automated test steps and view screenshots?
2. Why analyze the first failed step of a process before the others?
3. A custom automate cannot find a button that moved in the new release. How is this classified?
4. Where should automated and manual regression results come together for sign-off?