Webhook Health Alert Emails
Sparrow already tracks each webhook’s health (unknown / healthy / degraded /
unhealthy) and retries failed deliveries until they succeed or exhaust their
attempts. This feature lets you get emailed the moment either of those things
happens, without polling the API or wiring up your own monitor.
How it works
Section titled “How it works”Sparrow emits three events into a Sparrow-owned internal consumer, _sparrow,
using its own event/subscription/delivery pipeline — the same one your events
flow through:
| Event | Fires when |
|---|---|
sparrow.webhook.health_changed |
A webhook’s health transitions (e.g. healthy → degraded). The very first unknown → healthy transition is skipped — every webhook starts unknown and this would otherwise fire on every registration. |
sparrow.webhook.delivery_failed |
A delivery exhausts all of its retries and is marked permanently failed. |
sparrow.webhook.disabled |
Sparrow paused a webhook automatically because its receiver kept failing (see Automatic disabling). The payload carries the reason, consecutive_failures and failing_since. |
Two pieces work together:
- Who gets emailed: an alert config per email address, scoped to a consumer or one webhook.
- How the email is sent: one webhook under
_sparrow, subscribed to thesparrow.webhook.*events, usually the SendGrid recipe (see below). Without it, nothing is sent.
You register an alert config naming an email address; Sparrow embeds every matching
recipient into the event’s payload (payload.alert_recipients) at the moment
it’s pushed, and a mail-sending recipe (see below) fans that out to a real
email via SendGrid.
Register an alert config
Section titled “Register an alert config”-
Consumer-wide — every webhook under a consumer:
Terminal window curl -X POST http://localhost:8080/v1/consumers/acme/alert-configs \-H 'Content-Type: application/json' \-d '{"email": "ops@acme.example.com","event_types": ["sparrow.webhook.health_changed", "sparrow.webhook.delivery_failed", "sparrow.webhook.disabled"]}' -
One specific webhook — add
webhook_id:Terminal window curl -X POST http://localhost:8080/v1/consumers/acme/alert-configs \-H 'Content-Type: application/json' \-d '{"webhook_id": "wh_123","email": "oncall@acme.example.com","event_types": ["sparrow.webhook.delivery_failed"]}' -
List or remove configs the same way you would any other resource:
Terminal window curl http://localhost:8080/v1/consumers/acme/alert-configscurl -X DELETE http://localhost:8080/v1/consumers/acme/alert-configs/{alert_config_id}
A webhook whose own consumer is _sparrow never generates these events —
that would create a feedback loop.
Set up the email delivery channel
Section titled “Set up the email delivery channel”The system events above only become emails once you wire a mail-sending
webhook under the _sparrow consumer. Sparrow ships a bundled SendGrid
recipe for exactly this — apply it with the CLI:
sparrow --consumer _sparrow use sendgrid \ --param api_key=$SENDGRID_API_KEY \ --param from_email=alerts@yourdomain.com \ --param from_name="Sparrow Alerts" \ --param default_recipient=ops@yourdomain.com \ --event sparrow.webhook.health_changed \ --event sparrow.webhook.delivery_failed \ --event sparrow.webhook.disabledThat registers the SendGrid webhook, subscribes it to the system events, and
installs the transform in one step. The transform ranges
payload.alert_recipients into SendGrid’s v3 personalizations array — one
SendGrid API call delivers to every opted-in recipient, each in their own
personalization.
Check that alerts are being sent
Section titled “Check that alerts are being sent”GET /v1/capabilitiesreturns"alert_delivery": {"configured": true}once an active webhook under_sparrowis subscribed to the system events. Until then, the dashboard and the portal say alert emails aren’t set up, and alert configs are stored but nothing is sent.- In the dashboard, set the consumer filter on Webhooks or Deliveries to
_sparrow. You’ll see the alert-delivery webhook with its health, and every alert email it sent, failed or retried, like any other webhook’s deliveries. - Each webhook’s page has an Alerts section listing the addresses alerted about it, where you can add or remove them.
Automatic disabling
Section titled “Automatic disabling”A receiver that has been down for days only collects retries that will never
succeed. Sparrow pauses such a webhook itself once its receiver has failed
every attempt for SPARROW_AUTO_DISABLE_AFTER (default 120h, 5 days)
and at least SPARROW_AUTO_DISABLE_MIN_FAILURES attempts in a row
(default 10). Any successful attempt resets the clock. Set
SPARROW_AUTO_DISABLE_AFTER=0 (or SPARROW_AUTO_DISABLE_MIN_FAILURES=0) to
turn this off. The check runs with each health evaluation, on a webhook’s
attempts, so the pause comes with the first failed attempt after the window and
within a minute of it; the
health state machine
has the details.
When it happens:
- The webhook is paused like a manual pause: deliveries for new events, and any not yet sent, are held with status paused instead of being dropped.
- The webhook shows Auto-disabled in the UI with the reason, and the API
returns
auto_disabled_atandauto_disabled_reasonon it. - Sparrow emits
sparrow.webhook.disabled, which alert configs can opt into like the other two events. sparrow_webhooks_auto_disabled_totalincreases on/metrics.
Fix the receiver, then Resume the webhook. The response says how many
deliveries were held (paused_deliveries); they are not sent automatically, so
retry them from the UI (Retry paused deliveries) or with
status=paused&webhook_id={webhook_id} and the
batch retry. Resuming also
clears the marker and restarts the failure window, so the webhook gets the full
period again before it can be disabled a second time. Webhooks under _sparrow are never
auto-disabled, because they carry the alerts that would report it.
Good to know
Section titled “Good to know”- Events are always recorded. Every health transition and permanent
delivery failure pushes an event under
_sparrow, even when no alert configs or subscriptions match —alert_recipientsis then an empty list. Nothing is sent unless a webhook subscribes to these events; with the SendGrid recipe, an empty list goes to itsdefault_recipient(typically your ops inbox) as a catch-all. - Health measures the receiving system only. Deliveries that fail
because a subscription’s template did not render (
template_error), and deliveries held by a paused subscription, are faults or choices on the sending side. They never change a webhook’s health or trigger these events. - Errors here never affect delivery. If emitting a system event fails, Sparrow logs it and moves on; it never retries or blocks the webhook delivery that triggered it.
_sparrow’s events are internal. They don’t show up in your consumer’s event history — they live under the_sparrowconsumer and exist only to drive this alerting pipeline.