Sources
sparrow-sources is a small server that pushes events into Sparrow. It
can emit an event on a schedule, and it can receive Stripe and GitHub webhooks,
check their signatures, and republish them as Sparrow events. From there,
Sparrow fans them out to your subscribers with its usual retries, label
filters, transforms and delivery history.
Pick a source
Section titled “Pick a source”| Source | Produces | You need |
|---|---|---|
| cron | A fixed event on a schedule, e.g. report.tick every 5 minutes |
Nothing beyond Sparrow |
| stripe | stripe.<type> for each Stripe webhook, e.g. stripe.payment_intent.succeeded |
A Stripe webhook endpoint and its signing secret |
| github | github.<event>[.<action>] for each GitHub webhook, e.g. github.pull_request.opened |
A repository or organization webhook and its secret |
One process can run all three. The webhook providers share one HTTP server
(:8787 by default), each on its own path.
Install and run
Section titled “Install and run”Install the release binary (macOS and Linux) with the install script:
curl -fsSL https://raw.githubusercontent.com/sarathsp06/sparrow/main/scripts/install.sh | sh -s -- -c sourcessparrow-sources --config sources.yamlOr download the sparrow-sources-* archive for your platform from the
releases page, or
build it from a checkout with go build ./satellites/sparrow-sources.
To run it in a container, build a small image from the release binary:
FROM alpine:3.22RUN apk add --no-cache curl \ && curl -fsSL https://raw.githubusercontent.com/sarathsp06/sparrow/main/scripts/install.sh \ | sh -s -- -c sources -b /usr/local/binENTRYPOINT ["sparrow-sources", "--config", "/etc/sparrow/sources.yaml"]docker build -t sparrow-sources .docker run -v "$(pwd)/sources.yaml:/etc/sparrow/sources.yaml:ro" \ -e SPARROW_URL=http://sparrow:8080 -p 8787:8787 sparrow-sourcesUse it as a Go library
Section titled “Use it as a Go library”Each source is a Go package you can run in your own process, mounted on your
own server, instead of running sparrow-sources:
go get github.com/sarathsp06/sparrow/satellites/sparrow-sourcesImport sources for the shared pieces and the package for each source you
use:
| Package | What it gives you |
|---|---|
sources |
Pusher and Client (publish to Sparrow), Handle + MapFunc (build a webhook source), SparrowConfig |
sources/stripe |
stripe.Config (the stripe: provider section) and stripe.Handler |
sources/github |
github.Config (the github: provider section) and github.Handler |
sources/cron |
cron.Job (one cron: entry), cron.New and Emitter.Run |
Every source package keeps its own dependencies, so you only build what you
import. The library reads no files or env vars (SPARROW_URL and friends are
the binary’s), and your server owns the paths:
import ( "github.com/sarathsp06/sparrow/satellites/sparrow-sources/sources" "github.com/sarathsp06/sparrow/satellites/sparrow-sources/sources/cron" "github.com/sarathsp06/sparrow/satellites/sparrow-sources/sources/stripe")
client := sources.NewClient(sources.SparrowConfig{URL: "http://sparrow:8080", APIKey: key})
h, err := stripe.Handler(stripe.Config{SigningSecret: "whsec_..."}, client, logger)mux.Handle("POST /webhooks/stripe", h)
emitter, err := cron.New([]cron.Job{{Schedule: "0 * * * *", Event: "report.tick"}}) // validates every jobgo emitter.Run(ctx, client, logger)To add your own webhook source, pass a MapFunc to sources.Handle. You
verify the request and name the event; it pushes the raw body as the payload
and answers the provider exactly as the built-in ones do (401 for
sources.ErrBadSignature, 502 when the push fails so the provider retries):
mux.Handle("POST /webhooks/linear", sources.Handle("linear", client, logger, func(r *http.Request, body []byte) (string, map[string]string, error) { if !verifyLinear(r, body) { return "", nil, sources.ErrBadSignature } return "linear.issue.updated", map[string]string{"source": "linear"}, nil }))client.PushEvent(ctx, event, payload, labels, idempotencyKey) is also usable
on its own: it creates the event type on first use, and a non-empty
idempotencyKey makes the push idempotent (pass "" for none).
Handle sends <name>:sha256:<body hex> as the key, so a replayed request is
one event while Sparrow keeps the original (SPARROW_EVENT_RETENTION_DAYS).
Two identical bodies are always one event.
Before you start
Section titled “Before you start”-
Decide how event types get registered. Sparrow rejects a push to an unregistered event type with
404. When that happens,sparrow-sourcesregisters the type (POST /v1/event-types) and pushes again, so the firststripe.charge.refundedcreates its own type. That needs an API key allowed to register event types: the master key or a tenant-wide access token. If you’d rather control the definitions (with a schema, say), register or import them first. -
Get an API key if the Sparrow server runs with
SPARROW_API_KEY, and put it insparrow.api_keyorSPARROW_API_KEY. Without it every push gets401. -
Make the webhook server reachable by Stripe or GitHub: a public HTTPS URL in front of port 8787 (a load balancer, ingress, or a tunnel such as
cloudflaredwhile testing). Only the provider paths are served; everything else gets404. -
Subscribe to the events in Sparrow (register a webhook with
events: ["stripe.payment_intent.succeeded"], for example). An event with no matching subscription is stored but delivered nowhere.
Configuration
Section titled “Configuration”One YAML file, sources.yaml by default (--config to change it; --version prints the version). The
sparrow: section says where to push:
| Field | Required | Default | Env override | Meaning |
|---|---|---|---|---|
sparrow.url |
yes | SPARROW_URL |
Sparrow’s base URL, e.g. http://sparrow:8080. |
|
sparrow.api_key |
when the server has SPARROW_API_KEY |
none | SPARROW_API_KEY |
Sent as X-API-Key. |
sparrow.consumer |
no | default |
SPARROW_CONSUMER |
The consumer every event is pushed into. |
webhook.listen |
no | :8787 |
Address of the webhook server. Only started when a provider is configured. |
Environment variables win over the file. The provider secrets have no env override, so keep the file readable only by the user that runs it.
sparrow: url: http://sparrow:8080 api_key: "" consumer: default
cron: - schedule: "*/5 * * * *" event: report.tick payload: {kind: five_minutely}
webhook: listen: :8787 providers: stripe: signing_secret: whsec_... github: secret: your-github-webhook-secretWhat the webhook server answers
Section titled “What the webhook server answers”The provider decides whether to redeliver based on this status, and
sparrow-sources answers 200 only after Sparrow accepted the event, so a
failed push is retried by Stripe or GitHub.
| Status | When |
|---|---|
200 |
The event was pushed into Sparrow. |
400 |
The body isn’t valid JSON, a Stripe body has no type, or a GitHub request has no X-GitHub-Event header. The response says which. |
401 |
The signature is missing or doesn’t match. Also returned for a body over 1 MiB: only the first 1 MiB is read, so the signature can’t match. No response body. |
404 |
No provider is configured on that path. |
405 |
Anything but POST. |
502 |
The push to Sparrow failed: Sparrow down, 401 (bad API key), 409 (event type inactive), or auto-registration refused. Check the sparrow-sources log. |
Pushes a fixed event on a schedule.
| Field | Required | Default | Meaning |
|---|---|---|---|
schedule |
yes | Five fields: minute, hour, day of month, month, day of week. Each takes a number, *, a range (1-5), a step (*/10, 0-30/5) or a list (0,15,30,45). Day of week 0 and 7 are both Sunday. Names like MON aren’t supported. |
|
event |
yes | The Sparrow event name to push. | |
payload |
no | {} |
The event payload, the same every time. |
labels |
no | none | Labels for subscription filtering. |
cron: - schedule: "0 0 * * *" # every day at midnight event: billing.daily_renewal payload: {job: process_renewals} labels: {source: cron} - schedule: "*/5 9-17 * * 1-5" # every 5 minutes, 9:00-17:55, Monday to Friday event: telemetry.health_tickWhat arrives in Sparrow: an event named billing.daily_renewal with payload
{"job": "process_renewals"} and label source=cron, at the start of each
matching minute.
Good to know:
- Schedules run in the time zone of the
sparrow-sourcesprocess:TZon the host, UTC in most containers. - A tick is pushed once. If Sparrow is down at that moment, the push fails,
is logged, and is not retried: that tick is lost. Ticks missed while
sparrow-sourcesitself was stopped aren’t caught up either. - When both day of month and day of week are set (neither is
*), a day matching either one fires, as in standard cron. - Running two copies pushes every tick twice. Run one.
stripe
Section titled “stripe”Receives Stripe webhooks, checks the Stripe-Signature header, and pushes each
one as <prefix>.<type>.
Before you start: in the Stripe dashboard, open Developers → Webhooks →
Add endpoint, set the URL to https://<your-sources-host>/webhooks/stripe,
pick the event types, and after saving, reveal the endpoint’s Signing secret.
| Field | Required | Default | Meaning |
|---|---|---|---|
signing_secret |
yes | The endpoint’s signing secret, whsec_.... Test mode and live mode endpoints have different secrets. |
|
path |
no | /webhooks/stripe |
The path on this server. Must match the URL you gave Stripe. |
sparrow_event_prefix |
no | stripe |
Prefix of the Sparrow event name. Can’t be sparrow, which is reserved. |
What arrives in Sparrow for Stripe’s payment_intent.succeeded:
| Event name | stripe.payment_intent.succeeded |
| Labels | source=stripe, livemode=true or false |
| Payload | Stripe’s event object, unchanged: {"id": "evt_...", "type": "payment_intent.succeeded", "livemode": false, "data": {"object": {...}}, ...} |
Good to know:
- The signature must be at most 5 minutes old, so the host’s clock has to be right. Replaying an old event from the Stripe dashboard works, because Stripe signs each attempt afresh.
- Stripe retries a failed delivery for up to three days, and may deliver the
same event twice. Deduplicate downstream on the payload’s
id. - Filter test traffic out with a
livemode=truelabel filter on the subscription.
github
Section titled “github”Receives GitHub webhooks, checks the X-Hub-Signature-256 header, and pushes
each one as <prefix>.<X-GitHub-Event>, plus .<action> when the payload has
an action.
Before you start: in the repository or organization, open Settings →
Webhooks → Add webhook. Set Payload URL to
https://<your-sources-host>/webhooks/github, Content type to
application/json (the form-encoded type is rejected with 400), and a
Secret. Choose which events to send.
| Field | Required | Default | Meaning |
|---|---|---|---|
secret |
yes | The secret you set on the GitHub webhook. | |
path |
no | /webhooks/github |
The path on this server. Must match the Payload URL. |
sparrow_event_prefix |
no | github |
Prefix of the Sparrow event name. |
What arrives in Sparrow:
| GitHub sends | Sparrow event | Labels |
|---|---|---|
X-GitHub-Event: pull_request, "action": "opened" |
github.pull_request.opened |
source=github, repo=acme/api |
X-GitHub-Event: push (no action) |
github.push |
source=github, repo=acme/api |
X-GitHub-Event: ping (sent when you create the webhook) |
github.ping |
source=github, repo=acme/api (for a repository webhook) |
The payload is GitHub’s JSON body, unchanged. repo is the payload’s
repository.full_name, so you can route one repository’s events with a label
filter.
Good to know:
- Creating the webhook immediately sends a
ping, which registers agithub.pingevent type in Sparrow. That’s harmless; ignore it or subscribe to it to check the wiring. - GitHub doesn’t retry failed deliveries on its own. Redeliver from the webhook’s Recent Deliveries tab.
- A replayed delivery is recorded once. GitHub’s signature covers only the
body, with no timestamp, so each push uses the body’s SHA-256 as its
idempotency key: a byte-identical body, whatever its
X-GitHub-EventorX-GitHub-Deliveryheader, answers200and pushes nothing new while the first event is retained.X-GitHub-Deliveryisn’t signed and isn’t used. - Redelivering a delivery Sparrow already recorded therefore does nothing. To send it to a receiver again, retry the delivery in Sparrow.
Worked example: Stripe payments to your billing service
Section titled “Worked example: Stripe payments to your billing service”-
Configure the Stripe provider and start
sparrow-sources:sources.yaml sparrow:url: http://sparrow.internal:8080api_key: your-sparrow-api-keywebhook:providers:stripe:signing_secret: whsec_...Terminal window sparrow-sources --config sources.yamlIt logs
webhook listening addr=:8787. -
In Stripe, add the endpoint
https://sources.example.com/webhooks/stripeforpayment_intent.succeeded, then trigger one (with the Stripe CLI:stripe trigger payment_intent.succeeded).sparrow-sourceslogswebhook event pushed provider=stripe event=stripe.payment_intent.succeeded, and the event type now exists in Sparrow. -
Register your billing service as a webhook, then subscribe it to live payments only (label filters are set on the subscription):
Terminal window curl -X POST http://sparrow.internal:8080/v1/consumers/default/webhooks \-H 'Content-Type: application/json' -H 'X-API-Key: your-sparrow-api-key' \-d '{"url": "https://billing.internal/hooks/payments"}'# → note webhook_id and http_config.webhook_secretcurl -X POST http://sparrow.internal:8080/v1/consumers/default/subscriptions \-H 'Content-Type: application/json' -H 'X-API-Key: your-sparrow-api-key' \-d '{"webhook_id": "WEBHOOK_ID","event_name": "stripe.payment_intent.succeeded","label_filters": {"livemode": "true"}}'Each live payment is now delivered to the billing service, signed with that webhook’s own secret, so the service never needs Stripe’s.