Lesson 6 of 6
Monitoring as test evidence
9 min
Listen to this lesson
Run-time monitoring and testing are usually separate worlds. They should not be. Monitoring data tells the test lead which interfaces and jobs fail in real life, which is input for regression scope. And after every release, monitoring is the fastest way to verify that production behaves as before: if error rates, job runtimes and process KPIs stay in their normal range, the release did not break the business. This lesson connects the two, building on sap-bas-401.
Monitoring feeds regression scope
| Monitoring signal | What it tells the test lead | Effect on scope |
|---|---|---|
| Interfaces with recurring errors | Fragile mappings or data quality problems | Add negative and boundary tests for those interfaces |
| Jobs whose runtime grows month on month | Volume growth or inefficient code | Add a performance check to the regression |
| KPIs that spike at month end | Process stress points | Schedule period-end regression around them |
| Exceptions from custom code (ABAP runtime errors) | Defects that escaped testing | Add the failing path to the regression set |
Run-time signals that should change what is tested next release.
Priya reviews a monitoring summary with Sam before each quarterly regression window in CMR: the top ten failing interfaces, the jobs with the largest runtime growth, the KPIs that left their range, and the custom-code exceptions by object. Each item either maps to an existing test (which then gets priority) or becomes a new test case. It is the same principle as usage-based scoping in sap-tst-311, with run-time failures as the evidence.
Post-release verification
- Before the release: record the baseline for each critical business service: interface error rate, job runtimes, KPI ranges for the same weekday.
- During the import: Luis puts the affected business services into a maintenance window, so expected disruption does not alert.
- After the import: end the maintenance window, run the smoke tests, and let synthetic checks run if they are configured.
- Compare: the first interface batches, the first nightly jobs and the first KPI measurements against the baseline.
- Decide: back to normal, or an incident with an owner. Record the comparison as evidence of the release.
| Signal | Baseline (illustrative) | After release | Verdict |
|---|---|---|---|
| Great Lakes ORDERS IDocs in error per day | 0 to 2 | 1 | Normal |
| MRP Live plant 1710 runtime | 80 to 95 minutes | 140 minutes | Out of range: incident to Kenji's team |
| Deliveries without goods issue after 24 h | 5 to 12 | 9 | Normal |
| Billing job application status | Green | Green | Normal |
Illustrative course data: Contoso's post-release comparison for one quarterly release.
- Fri 18:00Window opensLuis sets the order-to-cash and procure-to-pay services to maintenance
- Fri 18:30ImportRelease transports imported into CMP
- Fri 20:00Window closesMaintenance ended; smoke tests and synthetic checks run
- Sat 06:00First nightNightly jobs and first interface batches compared with the baseline
- Mon 10:00First business dayKPIs compared; release verified or incidents assigned
The same signals define the end of hypercare. Contoso's exit criteria for hypercare after go-live include monitoring: two consecutive weeks with interface error rates, job runtimes and KPIs inside agreed ranges, no open severity 1 or 2 alerts, and every recurring alert either fixed or owned. Hypercare that ends by calendar date instead of by evidence tends to end with the run team inheriting the problems.
Synthetic User Monitoring adds a check that does not wait for real traffic: scripted user journeys, such as opening Manage Sales Orders and displaying sales order 1000457, run on a schedule and measure availability and response time. Run them right after the maintenance window ends, and the run team knows within minutes whether a key app works for users, before the first fleet order of the morning arrives.
What would you do?
After a quarterly release, all smoke tests passed. Two days later the MRP Live run for plant 1710 takes 140 minutes instead of its normal 80 to 95, and the run delays the morning output jobs. Nobody noticed at release time. What should change in Contoso's release process?
- Recurring interface errors, growing job runtimes, KPI spikes and code exceptions should change regression scope.
- Use maintenance windows during imports and end them before verification.
- Verify releases by comparing interfaces, jobs and KPIs with their baselines, not by smoke tests alone.
- Store the comparison with the release record as evidence.
Check your understanding · not graded
Lesson check
1. Why put a business service in maintenance during imports?
2. What is the risk of leaving a maintenance window open?
3. Which problem do smoke tests usually miss?
4. How should recurring interface errors affect regression?