Webhook Retry Observability: Metrics to Track Before Failures Go Silent

A webhook retry is easy to underestimate. It looks like the provider is helping you recover from a temporary failure.

Sometimes that is true. A retry may rescue an event that failed during a short outage, a deployment window, or a database hiccup.

But retries can also hide a deeper problem. If the same endpoint keeps returning 500, timing out, or rejecting requests with 401 or 422, the retry system only delays the moment when someone has to investigate.

That is why webhook retry observability matters. You need to know not only that retries exist, but whether they are helping, piling up, or making a broken workflow harder to understand.

What webhook retry observability means

Webhook retry observability means tracking enough signals to answer practical production questions.

• Is the endpoint reachable?
• Is it returning the expected status code?
• Are failures isolated or repeating?
• Are providers retrying old events?
• Are duplicate deliveries being handled safely?
• Are failed events stuck in a queue or dead-letter system?
• How long did it take to recover?

Without these answers, webhook failures become guesswork. You may know something broke, but not when it started, how long it lasted, which endpoint failed, or whether the retry backlog has cleared.

1. Endpoint response status

The first metric is the simplest: what HTTP status code does your webhook endpoint return?

Most webhook providers treat a 2xx response as accepted delivery. Non-2xx responses usually mean the delivery failed or was not accepted.

Track status codes by endpoint, not only across the whole application. A normal website route returning 200 does not prove your billing webhook route is healthy.

This metric helps you catch the most obvious failure mode: the webhook URL exists, but it is not responding in a way the provider accepts.

2. Timeout rate

A timeout is not the same as a clean error response. It is often worse because the sender does not know whether your application received the request, processed part of it, or never reached the handler.

Timeout rate tells you how often the endpoint fails to respond within the allowed window.

Common causes include:

• doing too much work inside the webhook request
• waiting on slow third-party APIs
• database locks or slow queries
• cold queues, overloaded workers, or synchronous processing
• network problems between the provider and your server

A good webhook endpoint should usually verify, store, queue, and respond quickly. Heavy work belongs outside the request path.

If timeouts are a recurring problem, read webhook endpoint timeouts .

3. Consecutive failure count

One failed check may be noise. Ten failed checks in a row usually means the endpoint is broken.

Consecutive failure count is useful because it shows whether a problem is continuing. It also helps avoid overreacting to one short network glitch while still catching real outages quickly.

For webhook endpoints, consecutive failures are often more useful than total failures for the day. A route that failed five times in five minutes needs faster attention than a route that failed five isolated times across a month.

4. First failure time

When an incident starts, the first question is usually simple: when did this begin?

First failure time gives you an anchor. You can compare it with deployments, configuration changes, database migrations, DNS updates, firewall rules, SSL renewals, or provider-side incidents.

Without a clear first failure time, investigation becomes slower. Developers end up searching logs around a wide time range instead of starting with the exact moment the endpoint became unhealthy.

5. Last successful response

Last success is just as useful as first failure.

If a webhook endpoint failed at 14:10 but last succeeded at 14:08, the incident window is small. If it has not succeeded since yesterday, you may have missed a long silent failure.

This is especially important for endpoints that do not receive real traffic all day. A low-volume webhook can be broken for hours before a real provider event exposes the problem.

6. Retry count

Retry count shows how much recovery pressure exists.

A small number of retries may be normal during temporary failures. A rising retry count means events are not being accepted cleanly, or processing is failing after delivery.

Watch retry count by provider and event type when possible. A billing provider retrying invoice events is more urgent than a low-priority notification provider retrying a harmless status update.

Retry count is also an early warning for duplicate-event risk. Every retry is another chance for your handler to repeat work unless idempotency is enforced.

7. Duplicate event rate

Duplicate webhook events are not always a bug. Many providers use at-least-once delivery, which means the same event can be delivered more than once.

The real question is whether your system handles duplicates safely.

Track how often duplicate provider event IDs appear, and whether they are ignored, skipped, or accidentally processed again.

Duplicates become risky when handlers perform actions such as:

• creating invoices
• granting access
• sending emails
• adding credits
• updating subscription states
• triggering internal jobs

For the safer design pattern, read webhook duplicate events and idempotent webhooks in Laravel .

8. Dead-letter queue size

If your application queues webhook processing jobs, your dead-letter queue is one of the clearest signs that automatic recovery is no longer enough.

A dead-lettered webhook event usually means the system tried to process the event and still failed after the allowed retry attempts.

Track:

• how many webhook jobs are dead-lettered
• which event types are affected
• which error messages appear repeatedly
• how old the oldest dead-lettered event is
• whether dead-lettered events were replayed later

A dead-letter queue is not a trash folder. It is a review queue for work your system could not finish safely.

For more detail, see webhook dead-letter queues .

9. Recovery time

Recovery time measures how long the endpoint stayed unhealthy before it returned to the expected behavior.

This matters because a five-minute webhook incident and a six-hour webhook incident are not the same operational problem.

Recovery time helps you understand:

• how quickly your team noticed the issue
• how long real provider deliveries may have failed
• whether retries had enough time to recover automatically
• whether customer data may need manual correction

Over time, this metric is useful for improving alert rules, incident response, deployment checks, and webhook endpoint design.

10. Event age during replay

Retried or replayed events are not always fresh. Some may arrive minutes or hours after the original business action happened.

Event age tells you how old an event is when your system finally processes it.

This matters for workflows where order is important. For example, a customer subscription may be created, updated, paused, resumed, and cancelled across several events. If an old event is replayed after a newer event, careless processing can move the local record backward.

Track event timestamps and compare them with the current state before applying important changes.

If event order has caused problems before, read webhook event ordering problems .

A practical webhook observability table

You do not need hundreds of metrics to start. Begin with the signals that explain delivery health, retry pressure, and recovery.

Separate delivery monitoring from processing monitoring

A clean webhook setup usually has two layers of observability.

The first layer is delivery monitoring. It asks whether the endpoint can be reached and whether it responds correctly.

The second layer is processing monitoring. It asks whether the stored event was handled correctly after the endpoint accepted it.

Mixing these two layers creates confusion. A webhook endpoint can return 200 while the queued job fails later. It can also return 500 after doing some side effects. You need visibility into both parts.

Where WebhookWatch helps

WebhookWatch focuses on the delivery monitoring layer.

It checks your webhook endpoint from the outside, records whether the response matches your expected HTTP status range, and helps you see when the endpoint starts failing or recovers.

That is useful for webhook routes that do not get normal browser traffic and may stay broken after a deployment, config change, routing mistake, or timeout issue.

Your application should still keep internal logs for real provider events. WebhookWatch gives you the outside-in signal: is the webhook endpoint behaving correctly right now?

For a broader production process, read the webhook incident playbook .

For the difference between ordinary uptime checks and webhook-specific monitoring, see webhook monitoring vs uptime monitoring .