The outbox pattern: surviving the dual write

Here's a bug I've watched bite more than one team. An order is placed: you write the row to your database, then publish an OrderPlaced event so the rest of the system — email, inventory, analytics — can react. Two writes, two systems. Most of the time it's fine. Then one day the process dies in the gap between them, and now the order exists but the event never fired. Or you publish first, the commit rolls back, and you've announced an order that doesn't exist.

This is the dual-write problem, and it has no clean solution as stated, because a database transaction and a message broker are two separate systems with no shared commit. You cannot make "insert the row" and "publish the event" atomic across them. So you stop trying.

Write the event to the same database

The outbox pattern turns two writes into one. Instead of publishing to the broker inside your request, you insert the event into an outbox table in the same transaction as the business change:

BEGIN;
  INSERT INTO orders (id, ...) VALUES (...);
  INSERT INTO outbox (id, topic, payload, created_at)
    VALUES (..., 'OrderPlaced', '{...}', now());
COMMIT;

Now the state change and the intent to publish commit together or not at all — one transaction, one system, actually atomic. A separate process reads unpublished rows from the outbox, sends them to the broker, and marks them done. If it crashes mid-flight, the rows are still sitting there on restart.

The details that make or break it

• The relay is at-least-once, so consumers must be idempotent. A publish can succeed and the "mark as sent" can fail, so the same event goes out twice. This is exactly where idempotency keys on the consumer earn their keep.
• Preserve order where it matters. If OrderPlaced must precede OrderShipped, the relay reads in insertion order and doesn't skip ahead past a stuck row.
• Don't let the outbox grow forever. Delete or archive dispatched rows on a schedule; a table that only grows becomes its own incident.

For reading the outbox you have two options. Polling is boring and it works — SELECT ... WHERE published_at IS NULL ORDER BY created_at on a short interval, which is what I reach for first. The lower-latency version tails the database's change log directly (change data capture), so events flow the moment the transaction commits with no polling loop at all.

What I like about the pattern is that it doesn't paper over the failure — it moves the atomicity to the one place you can actually get it, the database transaction you were already committing. The broker becomes a downstream detail instead of a second source of truth. "The event definitely fired if the order was placed" stops being a hope and becomes a property of the schema.