Why Stripe Webhooks Fail Silently (and How to Catch It Before Revenue Does)

This guide covers the mechanics Stripe documents, the failure modes engineers report in production, and the monitoring setup that turns silent failures into visible ones.

The delivery mechanics you're building against

Per Stripe's documentation and widely reported production behavior:

• Stripe expects a fast 2xx response; slow handlers are recorded as failed deliveries and retried.
• Failed deliveries in live mode are retried with exponential backoff for up to 3 days (test mode retries far less).
• After roughly 3 days of continuous failures, Stripe disables the endpoint and notifies you by email. After that, new events are not delivered at all until re-enabled.
• Missed events are recoverable (via the Dashboard for a window of days, longer via the API/CLI) but only if you know they're missing.
• Signature verification requires the raw request body; frameworks that parse and re-serialize JSON break it.
• Events can arrive out of order and more than once; handlers must be idempotent.

None of this is hidden. But each mechanic fails quietly, and they compound.

The five ways events get lost in production

1. The deploy window. Server restarts during deployment create a short window of 5xx responses. Stripe retries, so a single miss self-heals. Unless deploys are frequent enough that some events exhaust retries across repeated windows.

2. The slow handler. Handlers doing synchronous heavy work (emails, third-party calls, big writes) blow past the response window. Stripe marks the delivery failed and retries an event you may have already half-processed. Duplicates on one side, timeouts on the other.

3. Signature drift. Rotated signing secrets updated in one place but not the other, or a middleware change that alters the raw body. Every delivery starts returning 400. Three days later the endpoint is disabled.

4. The disabled endpoint. The compounding failure: whatever caused persistent failures, after ~3 days the endpoint goes dark entirely. Teams report the notification email landing in unmonitored inboxes. Engineers describe week-long holes in payment_failed events discovered only during churn audits.

5. The permanent-error loop. A handler that treats a permanently broken event (bad data, deleted record) as retryable returns 5xx forever, burning the retry window and (again) marching the endpoint toward disablement.

Why invoice.payment_failed is the event that hurts most

Most webhook events, missed, cause stale data. Missed invoice.payment_failed events cause silent revenue loss: this event is how your dunning, notification and recovery logic learns a renewal failed. Miss it and no recovery email sends, no retry schedule starts, and the subscription slides toward cancellation with the customer unaware. The portion you never even attempted to recover because the event vanished is pure leakage. It's why this single event type deserves its own monitoring, and why we use it as the canonical example of webhook drift across PayRes.

The reliable-handler baseline

The pattern that survives production, condensed:

1. Verify against the raw body: configure your framework to hand the handler unparsed bytes.
2. ACK fast, process async: persist the event to a queue/table, return 200, do the work in a worker. Persist and respond atomically: if the queue write fails, return 5xx so Stripe redelivers.
3. Be idempotent on event.id . Record processed IDs in the same transaction as the business change, so a crash between the two can't double-fulfill.
4. Classify errors: transient errors return 5xx (let Stripe retry); permanent ones get ACKed and parked in a dead-letter queue for review. Never let a permanently broken event burn the retry window.
5. Tolerate disorder: treat events as notifications and re-fetch the object's current state from the API rather than trusting event payload order.

Monitoring: the part everyone skips

Handler quality determines how often you lose events. Monitoring determines whether you know. Three layers, cheapest first:

Look at what Stripe already shows. The Dashboard's event-delivery view lists failed deliveries per endpoint. Practitioners' blunt advice: check it on a schedule, because failure notification emails routinely go to inboxes nobody reads.

Alert on your own signal. Emit a metric per event received/processed/failed; alert on failure spikes, and on silence (no events for N hours on an endpoint that should be busy is itself an alarm).

Reconcile independently. The safety net that catches everything else: a scheduled job pulls recent events from Stripe's Events API and diffs them against your local event log; anything present in Stripe but absent locally gets flagged and re-processed. This is the only layer that catches events that never arrived at all.

That last layer generalizes into a principle: webhook-driven state needs an independent source of truth to reconcile against. One endpoint and one provider, you can build this in a day. Multiple providers, multiple endpoints and multiple billing systems, and the reconciliation surface itself becomes architecture, which is the point where webhook reliability stops being a code problem and becomes a payment resilience problem. Continuously checking retry configuration, DLQ coverage and event-flow drift across a whole payment stack is what the PayRes control library does with a read-only connection.