Now accepting guest editorial from SAP, Oracle, Workday and ServiceNow practitioners.Pitch a guide
The Sign-Off Table

Lesson 5 of 6

Reporting and integration

9 min

Listen to this lesson

0:00 / 5:55

An automated run that nobody can find, read or trust has no value at sign-off. The gTAA names an interface to test management for exactly this reason: mapping test case definitions to automated tests. Reporting is part of the architecture, not something added at the end, and for an SAP program the target is clear: results flow into the program's test management (at Contoso, SAP Cloud ALM), failures become defects there, and evidence supports the go/no-go decision.

Map tests to test cases

Every automated test needs a stable identifier that maps to a test case in test management, and through it to the process, requirement or user story it proves. SAP describes Cloud ALM test management as building test cases from solution process activities, supporting manual and automated execution, and offering test automation APIs for integrated providers, with reporting and triggering of automated test cases in Cloud ALM. Design the mapping first: one automated test per Cloud ALM test case, or explicit many-to-one rules, recorded in the test definition, not in someone's spreadsheet.

Results land in SAP Cloud ALM against the test case they prove, next to the manual results.
The Sign-Off Table · thesignofftable.com/academy/sap-tau-301
  1. Test definitionCarries the test case ID, process reference and variant.
  2. ExecutionRun in a named environment; steps logged with timestamps and business outputs.
  3. Result publishingStatus, duration and evidence links sent to test management through its API.
  4. Failure handlingClassified; a defect created or updated with evidence, linked to the failed run.
  5. AnalyticsCoverage and progress across manual and automated tests feed readiness reporting.
From automated run to test management and defects.
The Sign-Off Table · thesignofftable.com/academy/sap-tau-301

Evidence that convinces

  • Business outputs: document numbers (sales order, delivery, billing document, journal entry) and the values asserted, so anyone can open the documents in CMQ.
  • Step log: each keyword or component call with inputs, outputs and duration.
  • Screenshots at key UI steps and on failure; API request and response for service steps, with secrets masked.
  • Context: environment, client, users and roles, data set version, framework version, and for Fiori the app and UI5 versions.
  • Failure detail: the failed step, expected versus observed state, and the system message.

Defects without noise

Automation can flood a defect tracker. Design the failure path deliberately. First, classify: product defect, test defect (script or component), data, or environment. Only product defects go to the program's defect process; the others go to the automation backlog or the environment owner. Second, deduplicate: a failure signature (component, step, message) that matches an open defect updates that defect with the new occurrence instead of creating another. Third, link: every defect references the failed run and its evidence, and every retest run references the defect.

Forty failures with one cause become one defect: a known signature updates the open defect instead of creating another.
The Sign-Off Table · thesignofftable.com/academy/sap-tau-301
  1. Did the system do something wrong for valid data and a correct test?Yes → product defect in test management, linked to the run · No → next
  2. Is the script, component or adapter wrong?Yes → automation backlog item; fix and rerun · No → next
  3. Is the data missing or consumed?Yes → data fix or factory change; rerun · No → next
  4. Is the environment unavailable or misconfigured?Yes → environment owner; rerun when fixed · No → investigate manually
  5. Does an open defect match the failure signature?Yes → add occurrence to it · No → create a new one
Routing automated failures without flooding the defect process.
The Sign-Off Table · thesignofftable.com/academy/sap-tau-301

Metrics that matter

The ISTQB syllabus devotes a chapter to collecting, analyzing and reporting test automation data for stakeholders. For SAP programs, a few measures carry most of the signal: pass rate by process with trends; flaky rate (tests whose result changes without a change in system or test); time from failure to classification; repair effort per release; and coverage of critical processes by automated tests. Report them next to manual testing, not in a separate automation dashboard nobody reads.

What would you do?

Contoso's nightly run fails 40 tests because the same pricing condition record is missing in CMQ. The framework creates 40 defects in Cloud ALM. Dan complains about the noise. What should change?

Key points
  • Reporting is part of the architecture: map each automated test to a test case in test management.
  • Evidence needs business document numbers, step logs, screenshots or request/response, and full context.
  • Classify failures and deduplicate by signature before anything reaches the defect process.
  • Link defects to failed runs and retests to defects.
  • Track pass rate, flaky rate, time to classify, repair effort and coverage, next to manual testing.

Check your understanding · not graded

Lesson check

  1. 1. Which gTAA interface describes mapping test case definitions to automated tests?

  2. 2. Which evidence most helps a process owner verify a passed automated order-to-cash test?

  3. 3. Forty tests fail with the same signature from one missing data record. What should the framework do?

  4. 4. What is the flaky rate?