Lesson 6 of 6
Testing and monitoring asynchronous flows
10 min
Listen to this lesson
The business events course gave consumer teams their test catalog: duplicates, out-of-order events, outages, poison messages. This lesson adds the tests the integration layer owes them: that the broker routes exactly what it should, that dead message queues catch what they should, that replay and ordering features behave as designed, and that monitoring raises the alarm before a backlog becomes a business problem. Priya's rule: the integration layer is proven before consumer teams test against it.
The integration-layer test catalog
| Test | How | Expected result |
|---|---|---|
| Positive routing | Publish one event per subscribed topic in the test stage | Each arrives in every queue that subscribes, once |
| Negative routing | Publish 1910 events and unrelated topics | Nothing arrives in queues that must not receive them |
| Wrong stage | Create an order in CMQ (and in CMR after each copy) | No event in development or production brokers |
| DMQ routing | Consumer rejects a crafted event until max redelivery | Event lands in that queue's DMQ; alert raised |
| Backlog | Stop the consumer; publish 500 events; restart | Queue holds all 500 within quota; drains; none lost |
| Ordering (partitioned) | Publish Created then Changed for the same key under load | Same consumer instance gets both, in order |
| Replay | Replay from a time after a simulated bug fix | Expected events redelivered; consumer handles duplicates |
| TTL | Leave an event beyond its time to live | Event leaves the queue as configured (DMQ or discard) |
Tests the integration team runs before consumer teams start. Replay and partitioned queues apply where advanced event mesh is used.
Replay: powerful and dangerous
In advanced event mesh, replay redelivers persistent messages from a replay log to a queue or topic endpoint: all messages, messages after a given message ID, or messages from a start time. SAP's documentation describes replay for non-partitioned queues and topic endpoints, and practitioners note that initiating replay on a queue removes messages already waiting in it to preserve order. That makes replay a planned operation, not a button to press in an incident: the consumer must be idempotent, the replay window must be agreed, and the queue's current contents must be understood. Contoso rehearses every replay in the test stage first.
Duplicates and ordering at the integration layer
Duplicates are not only a consumer problem. Replay produces them on purpose, a publisher retry after a timeout can produce them, and a consumer that crashes before acknowledging gets the same event again. The integration-layer test for duplicates is therefore simple: replay or republish a known event and confirm that the consumer's business effect happens once, checked in the target system, not in the consumer's log. Ordering is similar: where a partitioned queue promises order per key, Priya publishes Created and Changed for the same agreement under load with several consumer instances running, and checks that one instance processed both, in order.
Monitoring that catches problems early
| Signal | Why it matters | Contoso threshold (illustrative) |
|---|---|---|
| Queue depth | Consumer slower than producers, or stopped | Alert at 1,000 messages or 50% of quota |
| Age of oldest message | Backlog is growing even if depth looks small | Alert at 15 minutes for the fleet notifier |
| DMQ depth | Poison messages need a person | Alert on any message |
| Publish failures in S/4HANA | Channel or credentials broken | Alert on any failure in the event monitor |
| Silence | No events where events are expected | Alert if no SalesOrder events in 2 business hours |
Monitoring signals for Contoso's event flows. Advanced event mesh offers Insights dashboards; the Event Mesh capability has its own queue views; SAP Cloud ALM can collect alerts.
The last signal is the one teams forget. A broken outbound binding produces no errors and no backlog, only silence. Contoso's monitoring knows that order-to-cash publishes SalesOrder events every few minutes on a business day and alerts when it does not.
- Day 1WiringChannel and binding check; wrong-stage test
- Day 1RoutingPositive and negative routing for every queue
- Day 2Failure pathsDMQ routing, TTL, backlog with consumer stopped
- Day 3Advanced featuresOrdering per key and replay rehearsal, where used
- Day 3MonitoringTest alert for each signal reaches its team
Evidence for sign-off
- For each test in the catalog: date, stage, event ids, timestamps and result.
- For replay: the agreed window, the reference count and the consumer's recomputed totals.
- For monitoring: a triggered test alert for each signal, received by the right team.
- For the landscape: the stage pairing record and the latest wrong-stage test.
What goes wrong in testing
- Happy path only. Every event arrives in the demo; DMQs are never exercised until production.
- Untested alerts. Alerts are configured but go to an old distribution list.
- Improvised replay. Replay in production during an incident deletes queued messages nobody expected to lose.
- Silence ignored. No events means "all quiet" to everyone, until month-end.
What would you do?
On a Tuesday afternoon the fleet portal stops showing order changes. No alerts have fired: queue depth is zero and the DMQ is empty. What should Theo check first?
- Test routing both ways: what must arrive and what must not.
- Exercise every DMQ, and test alerts end to end before go-live.
- Replay is a planned, rehearsed operation that needs idempotent consumers and an agreed window.
- Monitor depth, oldest-message age, DMQ depth, publish failures and silence.
Check your understanding · not graded
Lesson check
1. Which test proves that a queue does not receive events meant for 1910?
2. According to SAP's documentation, which endpoints support message replay?
3. Why must replay be rehearsed before it is used in production?
4. Queue depth is zero and the DMQ is empty, but no updates arrive. Which signal catches this?
5. What proves an alert works?