Skip to content

Idempotent Consumers

Why Do Idempotent Consumers Exist?

Every other chapter in this section has been building toward this exact point. At-least-once delivery (Messaging Fundamentals) means the broker will redeliver a message unless it receives explicit confirmation of success. Retry strategies (the previous chapter) mean a consumer or producer will keep trying an operation until it works. Both of these guarantees share the same underlying assumption: that repeating the operation is safe. An idempotent consumer is what makes that assumption actually true, instead of just hoped-for.

Consumer receives "charge_customer" message for $500

↓

Charges the card successfully

↓  CRASHES before committing the offset / acknowledging the message

Broker never received confirmation → REDELIVERS the same message

↓

A non-idempotent consumer processes it again → charges the card a SECOND time

This isn't a rare edge case or a sign of a buggy consumer — it's the direct, expected, designed-in behavior of at-least-once delivery working exactly as intended. The redelivery is correct. What's missing is a consumer that can safely receive the same message twice and produce the same real-world outcome either way.

This chapter is the message-queue-specific application of the general Idempotency concept covered in depth in the API Design section — the same core idea (repeating an operation shouldn't repeat its effect), applied specifically to the mechanics of a message consumer rather than an HTTP endpoint.

What Problem It Actually Solves

  • Prevents duplicate side effects from redelivery — a charge, an email, an inventory decrement that would otherwise happen twice from one logical event.
  • Makes retries and redelivery genuinely safe, rather than merely "usually fine in practice" — the entire retry and at-least-once delivery model depends on this being true, not optional.
  • Decouples correctness from message-count — a correct system produces the same end state whether a message was delivered once or five times, which is the only sane way to reason about a distributed system where "exactly once" delivery is, as covered in Messaging Fundamentals, genuinely hard to guarantee natively.

Two Ways to Achieve Idempotency

Natural Idempotency

Some operations are idempotent simply because of how they're expressed — no extra tracking required at all.

# NOT naturally idempotent — running this twice adds 500 twice
db.execute("UPDATE accounts SET balance = balance + %s WHERE id = %s", (500, account_id))

# NATURALLY idempotent — running this twice still leaves the balance at exactly 1500
db.execute("UPDATE accounts SET balance = %s WHERE id = %s", (1500, account_id))

# NOT naturally idempotent — a second call creates a duplicate row
db.execute("INSERT INTO orders (id, status) VALUES (%s, %s)", (order_id, "placed"))

# NATURALLY idempotent — an upsert converges to the same end state regardless
db.execute(
    """INSERT INTO orders (id, status) VALUES (%s, %s)
       ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status""",
    (order_id, "placed"),
)

Whenever an operation can be expressed as "set the final state directly" rather than "adjust the current state by some delta," it tends to become naturally idempotent — applying the same absolute value twice converges to the same result, while applying the same relative adjustment twice compounds it.

Enforced Idempotency (Deduplication)

Not every operation can be naturally expressed this way — "send a confirmation email" or "charge a card" has no equivalent "set the final state" formulation; sending the same email twice is still sending it twice. For these, you need to explicitly track which messages have already been processed and skip duplicates.

def process_message(message):
    message_id = message.id

    with db.transaction():
        # atomically claim this message ID — fails if already claimed
        claimed = db.execute(
            "INSERT INTO processed_messages (message_id) VALUES (%s) ON CONFLICT DO NOTHING",
            (message_id,),
        )

        if claimed.rowcount == 0:
            return  # already processed — safe no-op, do NOT repeat the side effect

        send_confirmation_email(message.data)  # the actual side effect

    acknowledge(message)  # only ack after the transaction commits successfully

The Detail That Matters Most: Atomicity Between the Side Effect and the "Processed" Marker

This is the single most important mechanical point in this entire chapter, and the one most likely to separate a correct implementation from a subtly broken one.

WRONG — two separate, non-atomic steps:

1. Check "have I processed message_id X?" → no
2. Send the email
3. CRASH right here, before step 4
4. Record "message_id X has been processed"

↓

Message gets redelivered (crash meant no ack/offset commit happened)

↓

Step 1 again: "have I processed message_id X?" → still says NO,
              because step 4 never ran

↓

Email gets sent AGAIN — the dedup tracking didn't actually prevent the duplicate

RIGHT — the side effect and the "processed" marker committed as ONE atomic unit:

BEGIN TRANSACTION
    INSERT INTO processed_messages (message_id) ...  (the dedup marker)
    [ side effect happens here, in the SAME transaction, e.g. a DB write ]
COMMIT

↓

Either BOTH the marker and the side effect are recorded, or NEITHER is —
there is no possible crash point that leaves them out of sync with each other

This is exactly the Transactions chapter's core guarantee, applied here — wrapping the dedup marker and the actual side effect in one atomic transaction is what closes the gap the "wrong" version above leaves open. Whenever the side effect itself is a database write, this is straightforward: both go in the same BEGIN/COMMIT. Whenever the side effect is something that can't participate in a database transaction — sending an email, calling a third-party API — this atomicity is fundamentally harder to achieve, and is exactly the kind of problem the transactional outbox pattern (recording the intent to send the email as a database row, inside the same transaction as the dedup marker, and having a separate process actually send it afterward, reading from that outbox) is built to solve.

What to Actually Deduplicate On

Broker-assigned ID:   Kafka's (partition, offset), SQS's MessageId

Business/domain ID:   an order_id, a payment intent ID, an idempotency
                       key set explicitly by the PRODUCER

Deduplicating on a business-level identifier set by the producer is generally the safer, more portable choice — a broker-assigned ID's exact redelivery semantics can vary between systems and configurations (a Kafka offset reset, a message republished with a new broker-assigned ID after some internal reprocessing) in ways that can silently break dedup logic keyed purely on broker internals. A business ID (the same order can never have two different order IDs) stays a stable, meaningful anchor for deduplication regardless of what the underlying broker does internally.

Real-World Examples

Stripe Webhook Consumers

The exact scenario from the Webhooks chapter, revisited here from the consumer's side — Stripe explicitly documents that webhook events may be delivered more than once, and requires consumers to deduplicate by the event's id field (a business-level identifier), for precisely the reasons covered in this chapter.

Kafka Consumers Writing to a Database

A very common real pattern: the consumer's database write (the actual side effect) and a record of the consumed offset are committed together in a single database transaction — rather than committing the Kafka offset separately from the database write, which reopens exactly the non-atomic gap described above between "did the work" and "recorded that the work was done."

Payment Processing Consumers

Perhaps the highest-stakes real-world case for this entire chapter — a payment event redelivered and processed twice without idempotency protection is a direct, customer-visible financial error, which is exactly why payment-adjacent message consumers are usually held to the strictest standard of idempotency guarantees in a real engineering organization.

Common Interview Questions

Why must a consumer under at-least-once delivery be idempotent?

Because at-least-once delivery's entire mechanism is built on redelivering a message whenever the broker can't confirm successful processing — including cases where processing actually succeeded but the acknowledgment itself was lost. The broker cannot distinguish these two situations, so redelivery of an already-processed message is routine, expected behavior, not a rare failure — idempotency is what makes that safe.

What's the difference between natural idempotency and enforced idempotency?

Natural idempotency comes from how an operation is expressed — "set the value to X" converges to the same result no matter how many times it's applied, with no extra tracking needed. Enforced idempotency is needed when an operation has no such natural formulation (sending an email, charging a card) — it requires explicitly tracking which messages have already been processed and skipping duplicates.

Why does the "processed" marker need to be committed atomically with the side effect itself?

Because if they're two separate, non-atomic steps, a crash between them leaves the system in exactly the vulnerable state idempotency was supposed to prevent — the side effect already happened, but the marker recording that fact never got written, so a redelivered message looks unprocessed and triggers the side effect again. Wrapping both in one transaction (the same guarantee from the Transactions chapter) closes this gap.

Should you deduplicate on a broker-assigned message ID or a business-level ID?

Generally a business-level ID (an order ID, an explicit idempotency key set by the producer) is safer — broker-assigned IDs' exact redelivery semantics can vary across systems and configurations in ways that can silently break dedup logic tied to them, while a stable business identifier remains a meaningful, consistent anchor regardless of the underlying broker's internal behavior.

How do you achieve idempotency for a side effect that can't participate in a database transaction, like sending an email?

The transactional outbox pattern — record the intent to perform the side effect (e.g. "send this email") as a row in the same database transaction as the dedup marker and the rest of the business logic, then have a separate process read from that outbox and actually perform the side effect afterward. This achieves atomicity between "did we decide to do this" and "did we mark it as handled," even though the side effect itself happens outside the transaction.

Common Mistakes

❌ Checking for a duplicate and marking a message as processed as two separate, non-atomic operations.

A crash between the two leaves the exact gap idempotency was meant to close — the side effect already ran, but nothing recorded that fact, so a redelivered message looks unprocessed.

❌ Deduplicating solely on a broker-assigned message ID without considering business-level identity.

Broker redelivery and ID-assignment semantics can vary in ways that silently undermine dedup logic tied purely to broker internals — a stable business ID is generally the safer anchor.

❌ Assuming an operation is naturally idempotent when it's actually expressed as a relative adjustment.

balance += amount compounds on every redelivery; only an absolute balance = new_value (or an explicit dedup check) is actually safe to repeat.

❌ Treating idempotent consumers as optional for "low-stakes" side effects.

Even something as seemingly minor as sending a duplicate notification email erodes user trust and generates support complaints — idempotency isn't only for financial operations, just most urgently necessary there.

Key Takeaways

  • Idempotent consumers exist because at-least-once delivery's redelivery mechanism is a designed, expected behavior, not a rare failure — a consumer that can't safely process the same message twice will eventually double-process a real one.
  • Natural idempotency (expressing an operation as an absolute end state rather than a relative adjustment) requires no extra tracking; enforced idempotency (a dedup marker) is needed for side effects with no such natural formulation.
  • The dedup marker and the actual side effect must be committed as one atomic unit — treating them as two separate steps leaves exactly the gap idempotency is meant to close, reopened by a crash at the wrong moment.
  • Deduplicating on a stable, business-level identifier is generally safer than relying solely on a broker-assigned message ID, whose redelivery semantics can vary across systems.
  • For side effects that can't participate in a database transaction (sending an email, calling an external API), the transactional outbox pattern achieves the same atomicity guarantee indirectly, by recording intent transactionally and executing the side effect separately afterward.