Now accepting guest editorial from SAP, Oracle, Workday and ServiceNow practitioners.Pitch a guide
The Sign-Off Table

Lesson 5 of 6

Observability

10 min

Listen to this lesson

0:00 / 6:33

A failed end-to-end test tells you that something went wrong. Observability tells you what happened inside the systems while the test ran: which interface call failed, which job did not run, which IDoc stopped in status 51, how long the database took. In a landscape of S/4HANA, BTP and partners, a test failure without system signals can take a day to diagnose. With correlated signals it can take minutes, and you can tell an environment issue from a defect with evidence rather than opinion.

Signals worth correlating

SignalSAP sourceWhat it tells you
Interface and message errorsSAP Cloud ALM Integration and Exception Monitoring; IDoc status monitor; gateway error logWhich message failed, where, and with which error
System healthSAP Cloud ALM health monitoring; system logs and short dumpsWhether the system was degraded during the run
Jobs and automationSAP Cloud ALM Job and Automation Monitoring; job overviewWhether background steps the process needs ran on time
User experienceSAP Cloud ALM Real User and Synthetic User MonitoringResponse times of real and scripted user journeys
Application tracesTelemetry from BTP applications, for example in OpenTelemetry formatThe path of one request across services

Monitoring signals that help explain automated test failures.

SAP positions SAP Cloud ALM for Operations as the central place for these signals: its learning material describes real and synthetic user monitoring, integration and exception monitoring, health, job and automation monitoring, business process monitoring and configuration and security analysis, and notes that it can ingest OpenTelemetry data from SAP and non-SAP applications. The same tools that watch production can watch your quality system during test runs, if they are connected to it.

Correlate by time and by identity

  1. Tag the runEvery test run gets an id; every created document and message carries it in a reference field where possible.
  2. Record timestampsEach step logs system, client, user and start and end time.
  3. Collect signalsMonitoring data for the same systems and time window is queried or exported.
  4. JoinFailures are matched to errors, alerts and slow responses by time window and by business identifiers.
  5. ClassifyEnvironment, data, test or defect, now with system evidence attached.
Correlating automated test results with monitoring data.
The Sign-Off Table · thesignofftable.com/academy/sap-tau-401

Correlation works on two keys. Time is the coarse key: what was happening in CMQ between 02:14 and 02:16. Identity is the precise key: the sales order number, the delivery, the IDoc number, the message id or the run reference you wrote into the document. Design tests to capture identities at every step, and the join becomes exact.

The run reference written into each document turns a vague test failure into an exact match with the system signal.
The Sign-Off Table · thesignofftable.com/academy/sap-tau-401
  1. Did health monitoring show the system degraded in the window?Yes → environment, with the alert as evidence · No → continue
  2. Did an interface or message error occur for this run's identifiers?Yes → inspect the error: partner, mapping or SAP processing · No → continue
  3. Did a required job fail or not run?Yes → environment or job configuration · No → continue
  4. Was the process result wrong with all signals green?Yes → likely defect, with signals attached · No → check the test itself
Using monitoring signals to separate environment issues from defects.
The Sign-Off Table · thesignofftable.com/academy/sap-tau-401

From monitoring back to testing

Observability also runs the other way. Production monitoring shows which processes and interfaces fail most often, which are slowest and which are used most; that is direct input to test prioritization. Synthetic user monitoring, which runs scripted user journeys at intervals, is a cousin of automated testing: some teams reuse a small set of read-only journeys as production smoke checks after each deployment. Keep such checks strictly read-only or use dedicated test objects agreed with the business, because anything that posts in production is a real transaction.

Monitoring and testing feed each other: production signals set test priorities, and read-only journeys guard production after each deployment.
The Sign-Off Table · thesignofftable.com/academy/sap-tau-401
Logs after the factSystem logs and dumps read manually the next day.
Traced but slowDetailed traces collected but analyzed by hand.
Alert dashboardsHealth and integration alerts visible in real time, not tied to runs.
Correlated runsEach failed step linked automatically to the alerts and errors for its identifiers.
Maturity of observability in continuous testing.
The Sign-Off Table · thesignofftable.com/academy/sap-tau-401

What would you do?

Nightly end-to-end tests for order to cash fail at the billing step about twice a week. The test logs show only "billing document not found". What would most speed up diagnosis?

Key points
  • Observability explains what happened inside systems during a test, beyond pass or fail.
  • SAP Cloud ALM for Operations covers integration, health, job, user and process monitoring and can ingest OpenTelemetry data.
  • Correlate by time window and, precisely, by business identifiers captured at each step.
  • Use monitoring signals to classify failures as environment, data, test or defect with evidence.
  • Feed production monitoring back into test priorities, and test that monitoring alerts work.

Check your understanding · not graded

Lesson check

  1. 1. Which signal helps distinguish environment issues from defects?

  2. 2. Which key gives the most precise correlation between a test step and system signals?

  3. 3. What data format can SAP Cloud ALM ingest from SAP and non-SAP applications, per SAP learning material?

  4. 4. How can negative tests with virtual services improve operations readiness?