Architecture
Overview
Section titled “Overview”graph TD
accTitle: System overview diagram showing the Sparrow architecture
accDescr: System overview diagram showing the Sparrow architecture: your application pushes events via HTTP REST to the API, which stores events in PostgreSQL and enqueues them. EventWorker fans out to matching subscriptions, WebhookWorker delivers via HTTP POST with HMAC signing, and the embedded SvelteKit UI reads and writes through the same REST API.
App[Your Application] -->|POST /v1/.../events<br>HTTP / JSON| API
subgraph Sparrow
API[REST API<br>HTTP :8080]
DB[(PostgreSQL)]
EQ[Events Queue]
WQ[Webhooks Queue]
EW[EventWorker]
WW[WebhookWorker]
Health[Health Tracker]
API -->|Store event| DB
API -->|Enqueue| EQ
EQ --> EW
EW -->|Find matching<br>subscriptions| DB
EW -->|Fan-out jobs| WQ
WQ --> WW
WW -->|Load config +<br>transform payload| DB
WW -->|Record outcome| DB
WW -->|Update| Health
end
WW -->|HTTP POST<br>HMAC-signed| Endpoint[External Endpoints]
UI[Embedded SvelteKit UI] -->|REST / JSON| API
See the Layered Architecture page for an interactive diagram of the request path through middleware, services, storage, and queue workers.
Tech Stack
Section titled “Tech Stack”- Backend – Go 1.27
- Database – PostgreSQL 15
- Job Queue – River (Postgres-based)
- API – REST/HTTP on
:8080(Huma-based) - API Spec – OpenAPI 3.1, committed at
api/openapi.yaml - Web UI – SvelteKit 3 + Svelte 5 (Runes) + TypeScript + Tailwind CSS 4 (embedded static build)
- Observability – OpenTelemetry (traces, metrics, logs via OTLP; metrics also scrapeable at
/metricsin Prometheus format) - DB Access – pgx/v5 + sqlx (OTel-instrumented)
- Container – Multi-stage Dockerfile (distroless nonroot)
REST API
Section titled “REST API”Sparrow exposes a single HTTP REST server on :8080, built with Huma. All endpoints live under the /v1 base path and are described by an OpenAPI 3.1 spec committed at api/openapi.yaml (also api/openapi.json), generated from the Go REST definitions in internal/rest.
The API covers events, webhooks, subscriptions, deliveries, event types, and health. Consumer-scoped resources use /v1/consumers/{consumer}/... paths (for example, GET /v1/consumers/{consumer}/stats for consumer stats).
See the API Reference for the full endpoint list.
Event Processing Pipeline
Section titled “Event Processing Pipeline”flowchart LR
accTitle: Event processing pipeline
accDescr: An event pushed to the API is stored and queued, EventProcessingWorker fans it out into deliveries, and WebhookWorker sends each one to its target URL, retrying failures with backoff.
IN["POST …/events"] --> H["Push"] --> EW["EventProcessingWorker"] --> WW["WebhookWorker"] --> T["Target URL"]
| Stage | What it does |
|---|---|
Push (internal/webhooks) |
Rejects unknown event types (404) unless auto-register is on, dedups on the idempotency key, validates the payload against the current schema (soft), stores the event with its event_version, and enqueues it. |
| EventProcessingWorker | Matches subscriptions (consumer + event name or * + labels) and batch-inserts deliveries: paused for a paused subscription or webhook, queued for the rest. |
| WebhookWorker | Holds the delivery as paused if its webhook or subscription is paused, renders the transform (strict by default; a failure is template_error or falls back), applies the rate limit, sends the signed request, updates the delivery and records health for receiver outcomes only. A retryable failure is re-queued with backoff; a webhook failing too long is auto-disabled. |
Health State Machine
Section titled “Health State Machine”Event-sourced health calculation with a 24-hour lookback window:
stateDiagram-v2
accTitle: Health state machine diagram showing transitions between unknown, healthy, degraded, and unhealthy states based on success rates and consecutive failures, with a 24-hour inactivity timeout back to unknown.
accDescr: Health state machine diagram showing transitions between unknown, healthy, degraded, and unhealthy states based on success rates and consecutive failures, with a 24-hour inactivity timeout back to unknown.
[*] --> unknown
unknown --> healthy : >=90% success rate\n(3+ events)
unknown --> degraded : <90% success rate\n(5+ events)
unknown --> unhealthy : 5+ consecutive failures\nOR <80% rate (10+ events)
healthy --> degraded : Success rate drops <90%
healthy --> unhealthy : 5+ consecutive failures
degraded --> healthy : Success rate recovers >=90%
degraded --> unhealthy : 5+ consecutive failures\nOR <80% rate (10+ events)
unhealthy --> degraded : Partial recovery
unhealthy --> healthy : Full recovery >=90%
healthy --> unknown : No events in 24h
degraded --> unknown : No events in 24h
unhealthy --> unknown : No events in 24h
How it works:
- The delivery worker records each attempt as one append-only
webhook_health_eventsrow, in the same transaction as the delivery update. It never writes webhook-owned rows. HealthEvaluatorWorkerruns everySPARROW_HEALTH_EVAL_INTERVAL(default1m) and folds the events since its last run into per-minute buckets, the running 24-hour counters andwebhook_health_state(consecutive failures,failing_since).- When a webhook’s label changes, the same pass rewrites
webhook_registrations.healthand Sparrow emitssparrow.webhook.health_changed. The label therefore lags by up to the evaluation interval. - The health endpoint computes its summary (attempt counts, error category breakdown, average response time) from the buckets on request.
- In the same evaluator pass, a webhook in a long enough failure run is auto-disabled. See Automatic disabling below.
Automatic disabling
Section titled “Automatic disabling”Health labels only describe a receiver; automatic disabling acts on it. It is a separate rule from the labels above, fed by the same failure run.
stateDiagram-v2
accTitle: Automatic disabling of a webhook
accDescr: An active webhook enters a failure run on a failed attempt and leaves it on a success. An evaluator pass pauses a webhook whose run is long enough in both time and attempts. Resuming it restarts the run's clock.
[*] --> active
active --> failing : failed attempt\n(failing_since = its time)
failing --> active : any successful attempt
failing --> failing : failed attempt\n(consecutive_failures + 1)
failing --> auto_disabled : evaluator pass, when\nrun >= AUTO_DISABLE_AFTER\nAND >= AUTO_DISABLE_MIN_FAILURES
auto_disabled --> active : Resume\n(failing_since cleared)
The failure run. Each evaluator pass derives two columns of
webhook_health_state from the attempts it folds in, in order:
consecutive_failures (failed attempts since the last success) and
failing_since (the time of the first of them). Any successful attempt ends
the run and resets both. Only receiver outcomes are attempts: a delivery held
as paused, or failed on Sparrow’s side (such as a template_error), records
no health event and neither starts nor ends a run.
The check. After folding, the pass checks every webhook that recorded an attempt in that pass and is in a failure run. It pauses the webhook when all of these hold:
- automatic disabling is on: both
SPARROW_AUTO_DISABLE_AFTER(default120h) andSPARROW_AUTO_DISABLE_MIN_FAILURES(default10) are greater than0; setting either to0turns it off; - the webhook is active (a paused webhook is never disabled again) and its
consumer is not
_sparrow, whose webhooks carry the alerts that would report it; consecutive_failures >= SPARROW_AUTO_DISABLE_MIN_FAILURES;failing_sinceis at leastSPARROW_AUTO_DISABLE_AFTERago.
Both conditions must hold: ten failures in a minute do not disable a webhook, and neither does one failure a day for a week.
Timing. Because only webhooks with a new attempt in the pass are checked, a
webhook is disabled on its first failed attempt after the window has passed,
not at the moment the window ends. A receiver that gets no traffic stays
active, failing, until its next attempt. The pause lands within one
SPARROW_HEALTH_EVAL_INTERVAL (default 1m) of that attempt, plus the
evaluator’s few seconds of lag.
The pause. Check and pause are one UPDATE on webhook_registrations
(HealthRepository.AutoDisableWebhook), so evaluator runs on several instances
disable a webhook once. It sets active = false, auto_disabled_at, and
auto_disabled_reason (auto-disabled: <n> failed attempts in a row since <failing_since> with no success). The evaluator then logs a warning, increments
sparrow_webhooks_auto_disabled_total, and emits sparrow.webhook.disabled.
From then on the webhook’s deliveries are held as described under
Pausing.
Resume. Saving an auto-disabled webhook as active clears auto_disabled_at,
auto_disabled_reason and failing_since in the same statement. The next
failed attempt starts a new failing_since, so the webhook gets a full
SPARROW_AUTO_DISABLE_AFTER before it can be disabled again.
consecutive_failures is not reset: only a successful attempt resets it, so
after a resume the minimum-failures condition is usually already met and the
time window is what counts. A webhook paused and resumed by hand keeps its
failing_since: if its run is already older than the window, the next failed
attempt after resuming disables it.
For the operator’s view (UI, alerts, retrying held deliveries) see Automatic disabling in the health alerts guide.
Pausing
Section titled “Pausing”A paused webhook (by hand or auto-disabled) and a paused subscription behave the same way: fan-out records their deliveries as paused without queuing a job, and the delivery worker turns any queued or retrying delivery for them into paused instead of sending it. Holding is not an attempt and never affects health. Resume reports how many deliveries were held (paused_deliveries); they are only sent when retried.
HTTP Client Design
Section titled “HTTP Client Design”A centralized, OTel-instrumented HTTP client (internal/webhooks/client/):
- Connection pooling: up to
SPARROW_WEBHOOK_WORKERSconnections per receiver host (default 20) and twice that idle in total, 90s idle timeout - Signing: Support for both HMAC-SHA256 and Ed25519 using Standard Webhooks format.
- Leaky Bucket: Per-webhook rate limiting implemented via atomic PostgreSQL
UPDATEstatements. Workers “snooze” jobs when rate limits are hit. - Template engine: Go
text/templatewith LRU cache (100 entries, SHA-256 keyed), 37 built-in helper functions (json, base64, urlencode, string manipulation, etc.) - Object pooling:
sync.Poolforbytes.Buffer,[]byteslices, and header maps to reduce GC pressure - Header merging: Subscription-level headers override webhook-level defaults
- Metrics: OpenTelemetry instruments on the HTTP transport plus Sparrow’s own delivery-attempt counters and duration histogram, exported over OTLP and at
/metrics
Default Webhook Body
Section titled “Default Webhook Body”Every webhook delivery sends a JSON envelope with snake_case field names:
{ "version": "1", "event_id": "550e8400-e29b-41d4-a716-446655440000", "event_name": "user.created", "consumer": "billing", "webhook_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7", "delivery_id": "d-123", "timestamp": "2026-03-17T10:30:00Z", "attempt": 1, "payload": { "user_id": "u-123", "email": "alice@example.com" }}version– Envelope schema version (currently"1").event_id– UUID of the event that triggered this delivery.event_name– The event type (e.g.user.created,order.paid).consumer– Consumer the event belongs to.webhook_id– UUID of the webhook registration receiving this delivery.delivery_id– UUID of the delivery. It stays the same on every retry of that delivery; use it to deduplicate.timestamp– ISO 8601 / RFC 3339 timestamp of when the delivery was sent.attempt– Delivery attempt number (1 = first attempt, 2+ = retries).payload– The original event payload as submitted by the producer.
When a subscription has transform_enabled = true and a transform_template, the template output replaces the entire body. If the template fails to render, the subscription’s on_transform_error decides: fail (default) marks the delivery failed with template_error and sends nothing; fallback sends this envelope instead. Either way the error is recorded on the delivery, and it never counts toward webhook health.
HTTP Headers
Section titled “HTTP Headers”Every webhook delivery includes these headers:
Content-Type: application/json(aContent-Typein the webhook or subscription headers replaces it, e.g. for a form-encoded transform)User-Agent: Sparrow-Webhook/<version>(the server version, e.g.Sparrow-Webhook/1.0.0). AUser-Agentin the webhook or subscription headers replaces it.http_config.user_agentandhttp_config.content_typeare stored but not applied yet; set those headers inheadersinstead.X-Sparrow-Event-ID– Same asevent_idin the body.X-Sparrow-Delivery-ID– Same asdelivery_idin the body.X-Sparrow-Webhook-ID– Same aswebhook_idin the body.X-Sparrow-Event-Type– The event type name (event_namein the envelope). Present even when a transform replaces the body.X-Sparrow-Event-Version– The event type version the event was accepted under.X-Sparrow-Subscription-ID– The subscription that matched the event.webhook-id– Standard Webhooks message ID (msg_<delivery_id>). Only present when the webhook has a signing secret or an Ed25519 keypair.webhook-timestamp– Unix epoch seconds. Only present when the webhook has a signing secret or an Ed25519 keypair.webhook-signature– Space-delimited signatures (see below). Only present when the webhook has a signing secret or an Ed25519 keypair.
Custom headers configured on the webhook or subscription are merged in, with subscription-level headers overriding webhook-level defaults. They can’t override the X-Sparrow-* headers above.
Verifying Webhook Signatures
Section titled “Verifying Webhook Signatures”When a webhook_secret is configured, Sparrow signs every delivery using the Standard Webhooks specification.
Signature format:
Sparrow sets three headers: webhook-id, webhook-timestamp, and webhook-signature.
- The signed message is:
{webhook-id}.{timestamp}.{raw_body} - HMAC-SHA256 signatures are prefixed with
v1,and base64-encoded. - Ed25519 signatures are prefixed with
v1a,and base64-encoded. - Multiple signatures are space-delimited in the
webhook-signatureheader.
Don’t hand-roll verification. Use the Go module pkg/signature,
which is authoritative and tested against the server’s signer, or the copy-in samples for
Python, TypeScript, Java, Kotlin, Ruby, PHP, Rust, and Elixir. See
Verifying Webhook Signatures for per-language examples.
Any Standard Webhooks library also verifies v1 signatures when the secret is in the
default whsec_ format.
Encryption at Rest
Section titled “Encryption at Rest”Webhook secrets (webhook_secret) and sensitive headers (secret_headers) are encrypted at rest using envelope encryption with AES-256-GCM.
How It Works
Section titled “How It Works”Each encrypted value uses a unique random data encryption key (DEK). The DEK is wrapped (encrypted) by a key encryption key (KEK) and stored alongside the ciphertext:
[version:2] [kid_len:1] [key_id:kid_len] [edek_len:2 LE] [wrapped_dek:60] [nonce:12] [ciphertext+tag]- Version byte (
0x02) identifies the envelope format - Key ID selects which KEK wrapped the DEK, so ciphertext decrypts deterministically during rotations
- Wrapped DEK (60 bytes) = 12-byte nonce + 32-byte DEK + 16-byte GCM tag
- Data ciphertext uses its own 12-byte nonce + GCM authenticated encryption
Key Resolution
Section titled “Key Resolution”The KEK is provided by the SPARROW_ENCRYPTION_KEYS keyring plus SPARROW_ENCRYPTION_PRIMARY_KEY_ID. The server will not start without both variables. Generate keys with openssl rand -hex 32.
The key material is never stored in the database. Storing the encryption key next to the data it protects defeats the purpose of encryption at rest. Use a secrets manager, Kubernetes Secret, or .env file to provide the key. During a rotation, Sparrow encrypts new values with the primary key and continues decrypting existing values with any configured secondary keys.
See the Security Model page for API authentication, SSRF protection, HTTP hardening, secret masking, and tenant isolation.
Package Structure
Section titled “Package Structure”flowchart TD
accTitle: Package structure
accDescr: Package structure: cmd/server/main.go is the composition root wiring internal/tenant, internal/webhooks, and internal/rest. tenant imports no other Sparrow package; webhooks imports tenant (for the default tenant ID and the reserved _sparrow consumer), internal/storage and internal/errors, and contains store/ (internal/storage, tenant), queue/ (store, client, tenant, internal/errors, internal/observability), and client/ (store models, internal/crypto); internal/rest is the transport layer over webhooks.
Main["cmd/server/main.go<br>composition root — wires everything"]
Main --> Tenant["internal/tenant"]
Main --> Webhooks["internal/webhooks"]
Main --> Rest["internal/rest<br>transport layer"]
Webhooks --> Tenant
Webhooks --> Storage[internal/storage]
Webhooks --> Errors[internal/errors]
Webhooks --> Store["store/"]
Webhooks --> Queue["queue/"]
Webhooks --> Client["client/"]
Store --> Storage
Queue --> Store
Queue --> Client
Queue --> Tenant
Queue --> Errors
Queue --> Obs[internal/observability]
Client --> Store
Rest --> Webhooks
internal/tenant is a leaf: it imports no other Sparrow package, so
webhooks can use its constants (DefaultTenantID, SystemConsumer)
without a cycle. Zero cycles.
Domain Packages
Section titled “Domain Packages”internal/tenant – The default tenant and the reserved _sparrow consumer. At startup, main.go checks that migrations created the default tenant.
- Tables:
tenants
internal/webhooks – Core business domain: consumers, events, subscriptions, deliveries, health tracking.
- Tables: See OKF / DB schema for full table list
- Sub-packages:
store/– Data access (repository pattern, SQL queries)queue/– Async processing (River workers:EventProcessingWorkerfor fan-out,WebhookWorkerfor delivery,BatchJobWorkerfor bulk re-push and retry, and the periodicHealthEvaluatorWorker,RetentionWorker,TokenPurgeWorkerandBatchCleanupWorker)client/– HTTP delivery (request building, HMAC signing, templating)
Infrastructure Packages
Section titled “Infrastructure Packages”internal/rest – HTTP REST handlers (transport layer) built with Huma, serving the OpenAPI-described API on :8080. Delegates to the webhook service; it does not touch storage directly.
internal/observability – OpenTelemetry setup (traces, metrics, logs via OTLP) and the Prometheus /metrics handler.
internal/ui – Embedded SvelteKit frontend (go:embed).
internal/config – Environment variable loading.
internal/health – /health (liveness: the process is up, no database check) and /ready (readiness: pings the database, 503 once shutdown starts).
Shared Packages
Section titled “Shared Packages”internal/storage–DB/DBTXinterfaces,WithTransactionhelper, SQL error translationinternal/errors– Error classification with retryability determination
Design Principles
Section titled “Design Principles”Composition root in main.go – cmd/server/main.go constructs repositories, services, and the HTTP stack, and wires them together. No framework – just constructor functions and explicit wiring.
Repository per domain, not per table – Each domain owns a Repository interface and implementation. The repository encapsulates all SQL for that domain’s tables.
Schema ownership – Each domain package owns its tables: internal/tenant owns tenants, internal/webhooks owns the rest.
No shared models – No shared “models” package and no ORM. Each package defines its own models matching its own SQL schemas. The only shared infrastructure is internal/storage (DB abstraction).
Web UI
Section titled “Web UI”The web dashboard is a SvelteKit application that compiles to static files and is embedded into the Go binary via go:embed.
Build pipeline:
cd web && npm run build– compiles SvelteKit to static files ininternal/ui/dist/go build ./cmd/server– embedsinternal/ui/dist/viago:embed- At runtime,
internal/ui/embed.goserves the SPA with proper fallback routing
The Docker image builds the frontend automatically – no manual build step needed.