Skip to content

Webhook Health Alert Emails

Sparrow already tracks each webhook’s health (unknown / healthy / degraded / unhealthy) and retries failed deliveries until they succeed or exhaust their attempts. This feature lets you get emailed the moment either of those things happens, without polling the API or wiring up your own monitor.

Sparrow emits three events into a Sparrow-owned internal consumer, _sparrow, using its own event/subscription/delivery pipeline — the same one your events flow through:

Event Fires when
sparrow.webhook.health_changed A webhook’s health transitions (e.g. healthy → degraded). The very first unknown → healthy transition is skipped — every webhook starts unknown and this would otherwise fire on every registration.
sparrow.webhook.delivery_failed A delivery exhausts all of its retries and is marked permanently failed.
sparrow.webhook.disabled Sparrow paused a webhook automatically because its receiver kept failing (see Automatic disabling). The payload carries the reason, consecutive_failures and failing_since.

Two pieces work together:

  • Who gets emailed: an alert config per email address, scoped to a consumer or one webhook.
  • How the email is sent: one webhook under _sparrow, subscribed to the sparrow.webhook.* events, usually the SendGrid recipe (see below). Without it, nothing is sent.

You register an alert config naming an email address; Sparrow embeds every matching recipient into the event’s payload (payload.alert_recipients) at the moment it’s pushed, and a mail-sending recipe (see below) fans that out to a real email via SendGrid.

  1. Consumer-wide — every webhook under a consumer:

    Terminal window
    curl -X POST http://localhost:8080/v1/consumers/acme/alert-configs \
    -H 'Content-Type: application/json' \
    -d '{
    "email": "ops@acme.example.com",
    "event_types": ["sparrow.webhook.health_changed", "sparrow.webhook.delivery_failed", "sparrow.webhook.disabled"]
    }'
  2. One specific webhook — add webhook_id:

    Terminal window
    curl -X POST http://localhost:8080/v1/consumers/acme/alert-configs \
    -H 'Content-Type: application/json' \
    -d '{
    "webhook_id": "wh_123",
    "email": "oncall@acme.example.com",
    "event_types": ["sparrow.webhook.delivery_failed"]
    }'
  3. List or remove configs the same way you would any other resource:

    Terminal window
    curl http://localhost:8080/v1/consumers/acme/alert-configs
    curl -X DELETE http://localhost:8080/v1/consumers/acme/alert-configs/{alert_config_id}

A webhook whose own consumer is _sparrow never generates these events — that would create a feedback loop.

The system events above only become emails once you wire a mail-sending webhook under the _sparrow consumer. Sparrow ships a bundled SendGrid recipe for exactly this — apply it with the CLI:

Terminal window
sparrow --consumer _sparrow use sendgrid \
--param api_key=$SENDGRID_API_KEY \
--param from_email=alerts@yourdomain.com \
--param from_name="Sparrow Alerts" \
--param default_recipient=ops@yourdomain.com \
--event sparrow.webhook.health_changed \
--event sparrow.webhook.delivery_failed \
--event sparrow.webhook.disabled

That registers the SendGrid webhook, subscribes it to the system events, and installs the transform in one step. The transform ranges payload.alert_recipients into SendGrid’s v3 personalizations array — one SendGrid API call delivers to every opted-in recipient, each in their own personalization.

  • GET /v1/capabilities returns "alert_delivery": {"configured": true} once an active webhook under _sparrow is subscribed to the system events. Until then, the dashboard and the portal say alert emails aren’t set up, and alert configs are stored but nothing is sent.
  • In the dashboard, set the consumer filter on Webhooks or Deliveries to _sparrow. You’ll see the alert-delivery webhook with its health, and every alert email it sent, failed or retried, like any other webhook’s deliveries.
  • Each webhook’s page has an Alerts section listing the addresses alerted about it, where you can add or remove them.

A receiver that has been down for days only collects retries that will never succeed. Sparrow pauses such a webhook itself once its receiver has failed every attempt for SPARROW_AUTO_DISABLE_AFTER (default 120h, 5 days) and at least SPARROW_AUTO_DISABLE_MIN_FAILURES attempts in a row (default 10). Any successful attempt resets the clock. Set SPARROW_AUTO_DISABLE_AFTER=0 (or SPARROW_AUTO_DISABLE_MIN_FAILURES=0) to turn this off. The check runs with each health evaluation, on a webhook’s attempts, so the pause comes with the first failed attempt after the window and within a minute of it; the health state machine has the details.

When it happens:

  • The webhook is paused like a manual pause: deliveries for new events, and any not yet sent, are held with status paused instead of being dropped.
  • The webhook shows Auto-disabled in the UI with the reason, and the API returns auto_disabled_at and auto_disabled_reason on it.
  • Sparrow emits sparrow.webhook.disabled, which alert configs can opt into like the other two events.
  • sparrow_webhooks_auto_disabled_total increases on /metrics.

Fix the receiver, then Resume the webhook. The response says how many deliveries were held (paused_deliveries); they are not sent automatically, so retry them from the UI (Retry paused deliveries) or with status=paused&webhook_id={webhook_id} and the batch retry. Resuming also clears the marker and restarts the failure window, so the webhook gets the full period again before it can be disabled a second time. Webhooks under _sparrow are never auto-disabled, because they carry the alerts that would report it.

  • Events are always recorded. Every health transition and permanent delivery failure pushes an event under _sparrow, even when no alert configs or subscriptions match — alert_recipients is then an empty list. Nothing is sent unless a webhook subscribes to these events; with the SendGrid recipe, an empty list goes to its default_recipient (typically your ops inbox) as a catch-all.
  • Health measures the receiving system only. Deliveries that fail because a subscription’s template did not render (template_error), and deliveries held by a paused subscription, are faults or choices on the sending side. They never change a webhook’s health or trigger these events.
  • Errors here never affect delivery. If emitting a system event fails, Sparrow logs it and moves on; it never retries or blocks the webhook delivery that triggered it.
  • _sparrow’s events are internal. They don’t show up in your consumer’s event history — they live under the _sparrow consumer and exist only to drive this alerting pipeline.