Lesson 3 of 5
Designing idempotent, resilient consumers
9 min
Listen to this lesson
The hardest truth of event-driven systems is that "exactly once" delivery is not something you can count on. Brokers that guarantee delivery do it by keeping a message until the consumer acknowledges it. If the consumer processes the event and crashes before the acknowledgement arrives, the broker delivers it again. Producers may also resend: the CloudEvents specification says a duplicate event re-sent after a network error may carry the same id, and consumers may assume events with identical source and id are duplicates. So design for at-least-once delivery: every consumer must be idempotent, meaning processing the same event twice has the same effect as processing it once.
- ReceiveTake the event from the queue; do not acknowledge yet
- Check identityHas source + id already been processed? Yes → acknowledge and stop
- Read current stateCall the released S/4HANA API for the order named in the event
- Apply the effectWrite the result as a set-to-state update, inside one transaction with the record of the processed event
- AcknowledgeOnly now confirm the event to the broker
Three ways to be idempotent
Deduplicate by event identity. Keep a table of processed source + id pairs, written in the same transaction as the business effect. A repeat is recognized and skipped. Keep the entries at least as long as the broker could redeliver; Contoso keeps them for seven days.
Make the effect naturally idempotent. "Set the delivery date of order 1000473 to 12 May" can run twice harmlessly; "add one day to the delivery date" cannot. Prefer upserts keyed by business keys and absolute values over increments, counters and appends.
Fetch current state. Because most S/4HANA events are notifications, the consumer reads the current order through the API rather than trusting old data in the event. Processing an old duplicate then produces the current, correct result. This also softens ordering problems, as lesson 4 explains.
| Effect | Idempotent? | Safer design |
|---|---|---|
| Send an email to the fleet manager | No: two events, two emails | Record notification sent per order and status; send only on a real change |
| Set order status in the portal to Delivered | Yes | Keep as is |
| Increase open order count by one | No | Recount open orders from the source, or store per-order state |
| Create a warranty case for a return agreement | No, unless keyed | Use the return agreement number as the case key and upsert |
Acknowledge after, not before
The order of the last two steps decides what failure looks like. Acknowledge before processing and a crash loses the event. Acknowledge after processing and a crash causes a redelivery, which the idempotency check absorbs. For business events, losing data is worse than handling a duplicate, so acknowledge after the effect is safely stored.
Resilience in the consumer
A consumer depends on more than the broker. The fleet order notifier also calls S/4HANA and the fleet portal. Each call needs a timeout, and failures need a decision: a temporary failure (S/4HANA busy, portal timeout) should leave the event unacknowledged or schedule a retry; a permanent failure (order deleted, payload invalid) should not be retried forever. Lesson 4 covers retries and dead message queues; here the design rule is to classify every failure as transient or permanent in code, not by accident.
| Failure | Type | Consumer behavior |
|---|---|---|
| S/4HANA API times out | Transient | Do not acknowledge; let the broker redeliver, or retry with backoff |
| Order not found (deleted since the event) | Permanent but expected | Log, acknowledge, no further action |
| Payload does not match the schema | Permanent | Move to the dead message queue and alert |
| Duplicate source + id | Expected | Acknowledge and skip |
Classifying failures in the fleet order notifier.
What would you do?
In testing, Priya sees two identical emails to the fleet manager for one delivery date change. The logs show the same event id was processed twice after a consumer restart. What is the right fix?
- Design for at-least-once delivery: duplicates are normal.
- Idempotent consumers: deduplicate by source + id, prefer set-to-state effects, read current state.
- Store the dedup record and the business effect in one transaction; acknowledge after commit.
- Classify failures as transient or permanent in code.
- Protect deduplication in the database when the consumer scales out.
Check your understanding · not graded
Lesson check
1. What does idempotent mean for an event consumer?
2. Why acknowledge an event only after the effect is stored?
3. Which effect is naturally idempotent?
4. How long should processed event ids be kept?