Stripe webhooks from our Paris region: one retry storm, one fix

Our billing service runs in eu-west-3 (Paris) on a single 4 vCPU instance, Postgres 15 alongside it, roughly 40k webhook events a day from a payment provider. Nothing exotic. In March we saw a pattern: every morning around 08:40 CET, Stripe webhook deliveries would spike to 400+ pending, the worker queue would back up for 9–14 minutes, and a handful of subscription updates would land twice because our idempotency key was derived from the event payload hash, which changed on retry when the provider re-serialized the body.

The symptom was never a 500. It was latency: p99 webhook handling went from 180ms to over 6s, and the retries were the provider’s, not ours. What made it worse is that our endpoint returned 200 before the DB write committed, so the provider considered it delivered and moved on, while our queue kept a half-applied update.

Three things fixed it, in this order.

First, we made the idempotency key the provider’s event id, stored before any mutation, in the same transaction:

CREATE TABLE webhook_events (
  event_id text PRIMARY KEY,
  received_at timestamptz NOT NULL DEFAULT now(),
  processed_at timestamptz
);

INSERT INTO webhook_events (event_id)
VALUES ($1)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;
-- zero rows means duplicate, return 200 immediately

Second, we stopped doing the provider-facing 200 until the queue row was committed. The endpoint now writes the event row, enqueues a job id, commits, then returns 200. If the commit fails, we return 503 and let the provider retry — that is what retries are for.

Third, the worker no longer touches the payment API on the hot path. It writes to a local outbox table, and a separate poller drains it with a 250ms jitter and a per-customer serialization key. That cut the morning pile-up because two workers stopped racing on the same subscription.

def handle(event):
    with db.transaction():
        row = db.execute(INSERT_EVENT, [event["id"]])
        if not row:
            return 200
        db.execute(ENQUEUE, [event["id"], event["data"]["object"]["customer"]])
    return 200

The cost is real: we now hold a transaction open across two writes, and a slow DB means slower webhook responses and more provider retries. We accept that because a retry is cheaper than a double charge. We also added a 90-second reaper for rows with processed_at IS NULL, otherwise a crashed worker leaves an event stuck forever.

Where this breaks: if the provider changes the event id format across API versions, old dedupe rows become useless — we saw this once in staging and had to key on (event_id, api_version). Also, per-customer serialization only works if the customer id is present in the event; some events (disputes, for us) are account-level and fall back to a global lock, which is exactly the bottleneck we started with. And none of this protects against out-of-order delivery: a subscription.updated can arrive before subscription.created, so the worker still has to check state, not just apply deltas.

What I have not solved: the reaper and the outbox poller both run in the same process, so a deploy at 08:40 CET still costs us a few minutes of lag. I have been meaning to move the reaper to a separate timer, but I keep putting it off. If you run webhooks on a single box in France, how do you handle deploy-time queue drain without doubling the infrastructure?