Lesson 4 of 5
Ordering, retries and dead messages
10 min
Listen to this lesson
Duplicates are one asynchronous surprise; order is the other. Maria creates a sales order and changes it two seconds later. The fleet order notifier may well receive Changed before Created: the events can take different paths, a first delivery can fail and be retried after the second succeeds, or two consumer instances can pick them up in parallel. Unless a broker and design explicitly guarantee order for a key, assume events can arrive out of order, and design so that the final state is still right.
- 09:00:00Created publishedSalesOrder Created for order 1000473
- 09:00:02Changed publishedDelivery date moved; SalesOrder Changed for 1000473
- 09:00:03Changed processedConsumer instance B processes Changed first
- 09:00:05Created processedInstance A, delayed by a retry, processes Created last
- ResultDepends on designIf the consumer writes event data blindly, the old Created data wins; if it re-reads the order, the result is current
Designing for out-of-order events
- Re-read current state. For notification events, read the order from S/4HANA when processing. Whatever the arrival order, the last processing writes the current truth.
- Compare versions or times. If you must use data from the event, store the event
time(or a business version or change timestamp) per object and ignore events older than what you already applied. - Treat "not found yet" as transient. A Changed event for an order the consumer has never seen is not an error; process it from current state or retry it later.
- Serialize per key when you must. Where a broker offers ordering for a partition or key, use the business key; accept that this limits parallelism for that key.
Retries that help, not hurt
Retries recover from transient failures, and they can also turn a small outage into a big one. Good retry policies share a few traits: a small number of attempts, exponential backoff so each wait is longer than the last, jitter so many consumers do not retry in lockstep, and retries only for errors that can succeed later. SAP's own Cloud SDK guidance makes the same point: use retries with caution, because they often mask a problem that should be fixed, and pressing the same button repeatedly does not help with consistent failures. Its default retry middleware for JavaScript makes three retries with exponentially growing waits.
- Is the failure transient (timeout, 503, connection refused)?Yes → retry with backoff and jitter · No → continue
- Is it a client or data error (400, 404 for a bad key, invalid payload)?Yes → do not retry; handle or dead-letter · No → continue
- Have the retries been exhausted?Yes → move to the dead message queue and alert · No → retry again later
Dead message queues
Some events will never succeed: a payload that breaks the schema, an order type the consumer does not support, a bug. Left in the queue, such a poison message is redelivered again and again and can block or slow everything behind it. Brokers let you route messages that exceed a redelivery limit or cannot be delivered to a dead message queue (often called a dead letter queue). Check how your broker configures it: the attribute names and limits differ between SAP's broker offerings. The dead message queue is not a bin; it is a work queue with an owner, an alert and a procedure to fix and replay.
| Setting | Contoso choice for the fleet notifier queue | Why |
|---|---|---|
| Consumer retries | 3 attempts, exponential backoff with jitter | Covers short S/4HANA or portal hiccups |
| Redelivery from broker | Unacknowledged events redelivered | Covers consumer crashes |
| Dead message queue | Enabled, with an alert to the integration team | Poison messages leave the main queue |
| Replay procedure | Fix, then republish from the dead message queue | Recovery is planned, not improvised |
| Monitoring | Queue depth and age of oldest message | A growing queue is the earliest sign of trouble |
Contoso's retry and dead-message settings (illustrative).
What would you do?
S/4HANA test system CMQ is down for two hours of maintenance. The fleet notifier keeps receiving events for CMQ orders, and every API call fails. What behavior do you want?
- Assume events can arrive out of order; design so the final state is right anyway.
- Re-read current state, or compare event time or version, before applying data.
- Retry only transient failures, a few times, with exponential backoff and jitter.
- Poison messages go to a dead message queue that has an owner, alert and replay procedure.
- Monitor queue depth and message age.
Check your understanding · not graded
Lesson check
1. A Changed event arrives for an order the consumer has never seen. How should it be treated?
2. Why add jitter to retry backoff?
3. Which failure should NOT be retried?
4. What is a dead message queue for?