Skip to content

Architecture

graph TD
    accTitle: System overview diagram showing the Sparrow architecture
    accDescr: System overview diagram showing the Sparrow architecture: your application pushes events via HTTP REST to the API, which stores events in PostgreSQL and enqueues them. EventWorker fans out to matching subscriptions, WebhookWorker delivers via HTTP POST with HMAC signing, and the embedded SvelteKit UI reads and writes through the same REST API.
    App[Your Application] -->|POST /v1/.../events<br>HTTP / JSON| API

    subgraph Sparrow
        API[REST API<br>HTTP :8080]
        DB[(PostgreSQL)]
        EQ[Events Queue]
        WQ[Webhooks Queue]
        EW[EventWorker]
        WW[WebhookWorker]
        Health[Health Tracker]

        API -->|Store event| DB
        API -->|Enqueue| EQ
        EQ --> EW
        EW -->|Find matching<br>subscriptions| DB
        EW -->|Fan-out jobs| WQ
        WQ --> WW
        WW -->|Load config +<br>transform payload| DB
        WW -->|Record outcome| DB
        WW -->|Update| Health
    end

    WW -->|HTTP POST<br>HMAC-signed| Endpoint[External Endpoints]
    UI[Embedded SvelteKit UI] -->|REST / JSON| API

See the Layered Architecture page for an interactive diagram of the request path through middleware, services, storage, and queue workers.

  • Backend – Go 1.27
  • Database – PostgreSQL 15
  • Job Queue – River (Postgres-based)
  • API – REST/HTTP on :8080 (Huma-based)
  • API Spec – OpenAPI 3.1, committed at api/openapi.yaml
  • Web UI – SvelteKit 3 + Svelte 5 (Runes) + TypeScript + Tailwind CSS 4 (embedded static build)
  • Observability – OpenTelemetry (traces, metrics, logs via OTLP; metrics also scrapeable at /metrics in Prometheus format)
  • DB Access – pgx/v5 + sqlx (OTel-instrumented)
  • Container – Multi-stage Dockerfile (distroless nonroot)

Sparrow exposes a single HTTP REST server on :8080, built with Huma. All endpoints live under the /v1 base path and are described by an OpenAPI 3.1 spec committed at api/openapi.yaml (also api/openapi.json), generated from the Go REST definitions in internal/rest.

The API covers events, webhooks, subscriptions, deliveries, event types, and health. Consumer-scoped resources use /v1/consumers/{consumer}/... paths (for example, GET /v1/consumers/{consumer}/stats for consumer stats).

See the API Reference for the full endpoint list.


flowchart LR
    accTitle: Event processing pipeline
    accDescr: An event pushed to the API is stored and queued, EventProcessingWorker fans it out into deliveries, and WebhookWorker sends each one to its target URL, retrying failures with backoff.
    IN["POST …/events"] --> H["Push"] --> EW["EventProcessingWorker"] --> WW["WebhookWorker"] --> T["Target URL"]
Stage What it does
Push (internal/webhooks) Rejects unknown event types (404) unless auto-register is on, dedups on the idempotency key, validates the payload against the current schema (soft), stores the event with its event_version, and enqueues it.
EventProcessingWorker Matches subscriptions (consumer + event name or * + labels) and batch-inserts deliveries: paused for a paused subscription or webhook, queued for the rest.
WebhookWorker Holds the delivery as paused if its webhook or subscription is paused, renders the transform (strict by default; a failure is template_error or falls back), applies the rate limit, sends the signed request, updates the delivery and records health for receiver outcomes only. A retryable failure is re-queued with backoff; a webhook failing too long is auto-disabled.

Event-sourced health calculation with a 24-hour lookback window:

stateDiagram-v2
    accTitle: Health state machine diagram showing transitions between unknown, healthy, degraded, and unhealthy states based on success rates and consecutive failures, with a 24-hour inactivity timeout back to unknown.
    accDescr: Health state machine diagram showing transitions between unknown, healthy, degraded, and unhealthy states based on success rates and consecutive failures, with a 24-hour inactivity timeout back to unknown.
    [*] --> unknown

    unknown --> healthy : >=90% success rate\n(3+ events)
    unknown --> degraded : <90% success rate\n(5+ events)
    unknown --> unhealthy : 5+ consecutive failures\nOR <80% rate (10+ events)

    healthy --> degraded : Success rate drops <90%
    healthy --> unhealthy : 5+ consecutive failures

    degraded --> healthy : Success rate recovers >=90%
    degraded --> unhealthy : 5+ consecutive failures\nOR <80% rate (10+ events)

    unhealthy --> degraded : Partial recovery
    unhealthy --> healthy : Full recovery >=90%

    healthy --> unknown : No events in 24h
    degraded --> unknown : No events in 24h
    unhealthy --> unknown : No events in 24h

How it works:

  1. The delivery worker records each attempt as one append-only webhook_health_events row, in the same transaction as the delivery update. It never writes webhook-owned rows.
  2. HealthEvaluatorWorker runs every SPARROW_HEALTH_EVAL_INTERVAL (default 1m) and folds the events since its last run into per-minute buckets, the running 24-hour counters and webhook_health_state (consecutive failures, failing_since).
  3. When a webhook’s label changes, the same pass rewrites webhook_registrations.health and Sparrow emits sparrow.webhook.health_changed. The label therefore lags by up to the evaluation interval.
  4. The health endpoint computes its summary (attempt counts, error category breakdown, average response time) from the buckets on request.
  5. In the same evaluator pass, a webhook in a long enough failure run is auto-disabled. See Automatic disabling below.

Health labels only describe a receiver; automatic disabling acts on it. It is a separate rule from the labels above, fed by the same failure run.

stateDiagram-v2
    accTitle: Automatic disabling of a webhook
    accDescr: An active webhook enters a failure run on a failed attempt and leaves it on a success. An evaluator pass pauses a webhook whose run is long enough in both time and attempts. Resuming it restarts the run's clock.
    [*] --> active
    active --> failing : failed attempt\n(failing_since = its time)
    failing --> active : any successful attempt
    failing --> failing : failed attempt\n(consecutive_failures + 1)
    failing --> auto_disabled : evaluator pass, when\nrun >= AUTO_DISABLE_AFTER\nAND >= AUTO_DISABLE_MIN_FAILURES
    auto_disabled --> active : Resume\n(failing_since cleared)

The failure run. Each evaluator pass derives two columns of webhook_health_state from the attempts it folds in, in order: consecutive_failures (failed attempts since the last success) and failing_since (the time of the first of them). Any successful attempt ends the run and resets both. Only receiver outcomes are attempts: a delivery held as paused, or failed on Sparrow’s side (such as a template_error), records no health event and neither starts nor ends a run.

The check. After folding, the pass checks every webhook that recorded an attempt in that pass and is in a failure run. It pauses the webhook when all of these hold:

  • automatic disabling is on: both SPARROW_AUTO_DISABLE_AFTER (default 120h) and SPARROW_AUTO_DISABLE_MIN_FAILURES (default 10) are greater than 0; setting either to 0 turns it off;
  • the webhook is active (a paused webhook is never disabled again) and its consumer is not _sparrow, whose webhooks carry the alerts that would report it;
  • consecutive_failures >= SPARROW_AUTO_DISABLE_MIN_FAILURES;
  • failing_since is at least SPARROW_AUTO_DISABLE_AFTER ago.

Both conditions must hold: ten failures in a minute do not disable a webhook, and neither does one failure a day for a week.

Timing. Because only webhooks with a new attempt in the pass are checked, a webhook is disabled on its first failed attempt after the window has passed, not at the moment the window ends. A receiver that gets no traffic stays active, failing, until its next attempt. The pause lands within one SPARROW_HEALTH_EVAL_INTERVAL (default 1m) of that attempt, plus the evaluator’s few seconds of lag.

The pause. Check and pause are one UPDATE on webhook_registrations (HealthRepository.AutoDisableWebhook), so evaluator runs on several instances disable a webhook once. It sets active = false, auto_disabled_at, and auto_disabled_reason (auto-disabled: <n> failed attempts in a row since <failing_since> with no success). The evaluator then logs a warning, increments sparrow_webhooks_auto_disabled_total, and emits sparrow.webhook.disabled. From then on the webhook’s deliveries are held as described under Pausing.

Resume. Saving an auto-disabled webhook as active clears auto_disabled_at, auto_disabled_reason and failing_since in the same statement. The next failed attempt starts a new failing_since, so the webhook gets a full SPARROW_AUTO_DISABLE_AFTER before it can be disabled again. consecutive_failures is not reset: only a successful attempt resets it, so after a resume the minimum-failures condition is usually already met and the time window is what counts. A webhook paused and resumed by hand keeps its failing_since: if its run is already older than the window, the next failed attempt after resuming disables it.

For the operator’s view (UI, alerts, retrying held deliveries) see Automatic disabling in the health alerts guide.

A paused webhook (by hand or auto-disabled) and a paused subscription behave the same way: fan-out records their deliveries as paused without queuing a job, and the delivery worker turns any queued or retrying delivery for them into paused instead of sending it. Holding is not an attempt and never affects health. Resume reports how many deliveries were held (paused_deliveries); they are only sent when retried.


A centralized, OTel-instrumented HTTP client (internal/webhooks/client/):

  • Connection pooling: up to SPARROW_WEBHOOK_WORKERS connections per receiver host (default 20) and twice that idle in total, 90s idle timeout
  • Signing: Support for both HMAC-SHA256 and Ed25519 using Standard Webhooks format.
  • Leaky Bucket: Per-webhook rate limiting implemented via atomic PostgreSQL UPDATE statements. Workers “snooze” jobs when rate limits are hit.
  • Template engine: Go text/template with LRU cache (100 entries, SHA-256 keyed), 37 built-in helper functions (json, base64, urlencode, string manipulation, etc.)
  • Object pooling: sync.Pool for bytes.Buffer, []byte slices, and header maps to reduce GC pressure
  • Header merging: Subscription-level headers override webhook-level defaults
  • Metrics: OpenTelemetry instruments on the HTTP transport plus Sparrow’s own delivery-attempt counters and duration histogram, exported over OTLP and at /metrics

Every webhook delivery sends a JSON envelope with snake_case field names:

{
"version": "1",
"event_id": "550e8400-e29b-41d4-a716-446655440000",
"event_name": "user.created",
"consumer": "billing",
"webhook_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
"delivery_id": "d-123",
"timestamp": "2026-03-17T10:30:00Z",
"attempt": 1,
"payload": {
"user_id": "u-123",
"email": "alice@example.com"
}
}
  • version – Envelope schema version (currently "1").
  • event_id – UUID of the event that triggered this delivery.
  • event_name – The event type (e.g. user.created, order.paid).
  • consumer – Consumer the event belongs to.
  • webhook_id – UUID of the webhook registration receiving this delivery.
  • delivery_id – UUID of the delivery. It stays the same on every retry of that delivery; use it to deduplicate.
  • timestamp – ISO 8601 / RFC 3339 timestamp of when the delivery was sent.
  • attempt – Delivery attempt number (1 = first attempt, 2+ = retries).
  • payload – The original event payload as submitted by the producer.

When a subscription has transform_enabled = true and a transform_template, the template output replaces the entire body. If the template fails to render, the subscription’s on_transform_error decides: fail (default) marks the delivery failed with template_error and sends nothing; fallback sends this envelope instead. Either way the error is recorded on the delivery, and it never counts toward webhook health.

Every webhook delivery includes these headers:

  • Content-Type: application/json (a Content-Type in the webhook or subscription headers replaces it, e.g. for a form-encoded transform)
  • User-Agent: Sparrow-Webhook/<version> (the server version, e.g. Sparrow-Webhook/1.0.0). A User-Agent in the webhook or subscription headers replaces it. http_config.user_agent and http_config.content_type are stored but not applied yet; set those headers in headers instead.
  • X-Sparrow-Event-ID – Same as event_id in the body.
  • X-Sparrow-Delivery-ID – Same as delivery_id in the body.
  • X-Sparrow-Webhook-ID – Same as webhook_id in the body.
  • X-Sparrow-Event-Type – The event type name (event_name in the envelope). Present even when a transform replaces the body.
  • X-Sparrow-Event-Version – The event type version the event was accepted under.
  • X-Sparrow-Subscription-ID – The subscription that matched the event.
  • webhook-id – Standard Webhooks message ID (msg_<delivery_id>). Only present when the webhook has a signing secret or an Ed25519 keypair.
  • webhook-timestamp – Unix epoch seconds. Only present when the webhook has a signing secret or an Ed25519 keypair.
  • webhook-signature – Space-delimited signatures (see below). Only present when the webhook has a signing secret or an Ed25519 keypair.

Custom headers configured on the webhook or subscription are merged in, with subscription-level headers overriding webhook-level defaults. They can’t override the X-Sparrow-* headers above.

When a webhook_secret is configured, Sparrow signs every delivery using the Standard Webhooks specification.

Signature format:

Sparrow sets three headers: webhook-id, webhook-timestamp, and webhook-signature.

  • The signed message is: {webhook-id}.{timestamp}.{raw_body}
  • HMAC-SHA256 signatures are prefixed with v1, and base64-encoded.
  • Ed25519 signatures are prefixed with v1a, and base64-encoded.
  • Multiple signatures are space-delimited in the webhook-signature header.

Don’t hand-roll verification. Use the Go module pkg/signature, which is authoritative and tested against the server’s signer, or the copy-in samples for Python, TypeScript, Java, Kotlin, Ruby, PHP, Rust, and Elixir. See Verifying Webhook Signatures for per-language examples. Any Standard Webhooks library also verifies v1 signatures when the secret is in the default whsec_ format.


Webhook secrets (webhook_secret) and sensitive headers (secret_headers) are encrypted at rest using envelope encryption with AES-256-GCM.

Each encrypted value uses a unique random data encryption key (DEK). The DEK is wrapped (encrypted) by a key encryption key (KEK) and stored alongside the ciphertext:

[version:2] [kid_len:1] [key_id:kid_len] [edek_len:2 LE] [wrapped_dek:60] [nonce:12] [ciphertext+tag]
  • Version byte (0x02) identifies the envelope format
  • Key ID selects which KEK wrapped the DEK, so ciphertext decrypts deterministically during rotations
  • Wrapped DEK (60 bytes) = 12-byte nonce + 32-byte DEK + 16-byte GCM tag
  • Data ciphertext uses its own 12-byte nonce + GCM authenticated encryption

The KEK is provided by the SPARROW_ENCRYPTION_KEYS keyring plus SPARROW_ENCRYPTION_PRIMARY_KEY_ID. The server will not start without both variables. Generate keys with openssl rand -hex 32.

The key material is never stored in the database. Storing the encryption key next to the data it protects defeats the purpose of encryption at rest. Use a secrets manager, Kubernetes Secret, or .env file to provide the key. During a rotation, Sparrow encrypts new values with the primary key and continues decrypting existing values with any configured secondary keys.

See the Security Model page for API authentication, SSRF protection, HTTP hardening, secret masking, and tenant isolation.


flowchart TD
    accTitle: Package structure
    accDescr: Package structure: cmd/server/main.go is the composition root wiring internal/tenant, internal/webhooks, and internal/rest. tenant imports no other Sparrow package; webhooks imports tenant (for the default tenant ID and the reserved _sparrow consumer), internal/storage and internal/errors, and contains store/ (internal/storage, tenant), queue/ (store, client, tenant, internal/errors, internal/observability), and client/ (store models, internal/crypto); internal/rest is the transport layer over webhooks.
    Main["cmd/server/main.go<br>composition root — wires everything"]
    Main --> Tenant["internal/tenant"]
    Main --> Webhooks["internal/webhooks"]
    Main --> Rest["internal/rest<br>transport layer"]
    Webhooks --> Tenant
    Webhooks --> Storage[internal/storage]
    Webhooks --> Errors[internal/errors]
    Webhooks --> Store["store/"]
    Webhooks --> Queue["queue/"]
    Webhooks --> Client["client/"]
    Store --> Storage
    Queue --> Store
    Queue --> Client
    Queue --> Tenant
    Queue --> Errors
    Queue --> Obs[internal/observability]
    Client --> Store
    Rest --> Webhooks

internal/tenant is a leaf: it imports no other Sparrow package, so webhooks can use its constants (DefaultTenantID, SystemConsumer) without a cycle. Zero cycles.

internal/tenant – The default tenant and the reserved _sparrow consumer. At startup, main.go checks that migrations created the default tenant.

  • Tables: tenants

internal/webhooks – Core business domain: consumers, events, subscriptions, deliveries, health tracking.

  • Tables: See OKF / DB schema for full table list
  • Sub-packages:
    • store/ – Data access (repository pattern, SQL queries)
    • queue/ – Async processing (River workers: EventProcessingWorker for fan-out, WebhookWorker for delivery, BatchJobWorker for bulk re-push and retry, and the periodic HealthEvaluatorWorker, RetentionWorker, TokenPurgeWorker and BatchCleanupWorker)
    • client/ – HTTP delivery (request building, HMAC signing, templating)

internal/rest – HTTP REST handlers (transport layer) built with Huma, serving the OpenAPI-described API on :8080. Delegates to the webhook service; it does not touch storage directly.

internal/observability – OpenTelemetry setup (traces, metrics, logs via OTLP) and the Prometheus /metrics handler.

internal/ui – Embedded SvelteKit frontend (go:embed).

internal/config – Environment variable loading.

internal/health – /health (liveness: the process is up, no database check) and /ready (readiness: pings the database, 503 once shutdown starts).

  • internal/storage – DB/DBTX interfaces, WithTransaction helper, SQL error translation
  • internal/errors – Error classification with retryability determination

Composition root in main.go – cmd/server/main.go constructs repositories, services, and the HTTP stack, and wires them together. No framework – just constructor functions and explicit wiring.

Repository per domain, not per table – Each domain owns a Repository interface and implementation. The repository encapsulates all SQL for that domain’s tables.

Schema ownership – Each domain package owns its tables: internal/tenant owns tenants, internal/webhooks owns the rest.

No shared models – No shared “models” package and no ORM. Each package defines its own models matching its own SQL schemas. The only shared infrastructure is internal/storage (DB abstraction).


The web dashboard is a SvelteKit application that compiles to static files and is embedded into the Go binary via go:embed.

Build pipeline:

  1. cd web && npm run build – compiles SvelteKit to static files in internal/ui/dist/
  2. go build ./cmd/server – embeds internal/ui/dist/ via go:embed
  3. At runtime, internal/ui/embed.go serves the SPA with proper fallback routing

The Docker image builds the frontend automatically – no manual build step needed.