ingestlayer/destination

Destination.

Where events land. One event in, many places out. The router decides which, the destinations decide how. When something is down, we retry. When something is your fault, we tell you.

fig. 1 — one event, many sinks. The router walks the destination list once; each clause decides write or skip. Every sink owns its own retries, dead-letters, and redaction.

Routing01

The branching step.

Every destination has a when clause. The event walks the list of destinations once; each clause is evaluated against the typed event, including labels from classify and entities from enrich. A truthy clause writes. A false one skips. always: true bypasses the check.

Fan-out is implicit: list as many destinations as you want. Each is its own retry budget, its own dead-letter queue, its own redaction policy. They do not share failure modes.

Destinations02

Where they need to land.

Three flavors. Chat for human eyes. Webhooks and APIs for other systems. Warehouses and object storage for the long tail.

Missing one? Open a destination request on GitHub. Vote on the queue with thumbs up.

Branching03

One event, shaped differently for each consumer.

The same signup event becomes a Slack ping in #growth, a Linear issue for the on-call engineer, and a row in Postgres. Three destinations, three different shapes, one set of conditions evaluated typed.

step · route
# pipelines/signup.yaml (excerpt)
destinations:
  - slack:
      channel: "#growth"
      when: $classify.intent == "hot_lead"
      template: |
        {{ $person.name }} from {{ $person.company }}
        just signed up.

  - linear.issue:
      when: $classify.priority > 80
      team:  ENG
      title: "Follow up: {{ $person.email }}"

  - warehouse.pg:
      table:  events.signup
      always: true   # write every event, branching skipped

Templates are mustache. They run after classify and enrich, so everything attached upstream is in scope.

Failure04

Retry. Dead-letter. Replay.

The wire is not the universe. Sometimes Slack is down. Sometimes the warehouse is rotating credentials. Sometimes their server returns a 500 for reasons nobody understands and fixes it in a deploy. We back off exponentially, with jitter, up to a ceiling. If we exhaust the attempts, the event lands in the dead-letter queue with the response that killed it.

retry · dead-letter
# global retry config (defaults shown)
retry:
  attempts: 5
  backoff:
    initial: 1s
    factor:  2
    max:     60s
  dead_letter:
    after_attempts: true
    after_status:   [400, 401, 403, 422]
    # 4xx that are "your fault, not ours" go to DLQ
    # without burning the rest of the retry budget
i.
retry
Exponential backoff with jitter. Per destination. Configurable per pipeline.
ii.
dead-letter
Browsable queue with the request, the response, and the failure reason. Per destination.
iii.
replay
From the DLQ on demand. Counted against your monthly cap, because the work is real.

End of pipeline05

That was the spec.

Three stages, end to end. Ingest accepts the event. Transform runs any subset of ten typed actions — enrichment, mutation, control — in the order you declared. Route fans the result out. One process, in two EU regions, your data never leaving the bloc.

← back to ingestiontransformationthe full spec

Ready to send an event somewhere it actually belongs?

sign upread the docs