ingestlayer/docs/v0.4

docs · 02 concepts

Pipelines.

A pipeline is three slots: which events trigger it, what runs on each one, and where the result lands. Composed in the dashboard, versioned on every deploy, evaluated in order on every matched event.

Shape

Sources → actions → destinations.

Every pipeline is a sequence of three lists:

i.
sources
One or more source filters. An event matches if any source's `match` equals the event type (or the match is empty — meaning any event from that source).
ii.
actions
Zero or more transforms, looked up by category in the action picker. They run in declared order. A filter / dedupe / throttle can short-circuit the run.
iii.
destinations
Zero or more egress targets. The last transform's output (or the raw event if there is no transform) gets fanned out to every destination.

Matching

When does a pipeline fire.

Only pipelines in active status fire. A pipeline matches an event when one of its source rules names that event type. An empty match is permissive — it accepts everything from that source — and a pipeline with no sources at all accepts everything. That default keeps things useful while you're still wiring up.

Execution

One delivery per matched pipeline.

When N pipelines match one event, you get N delivery rows — one per pipeline — each running independently. A single pipeline's failure doesn't block the others. See deliveries for the retry curve.

Versioning

Every deploy is a new version.

Clicking deploy saves the canvas as a new version of the pipeline. Older versions are kept for history, so you can see what was running when. A new version takes effect on the next event that matches; deliveries that are already in flight finish on the version they started under, never halfway between two.

Status

active · paused · draft.

draft is the initial state — never matched, never run. Click activate in the header to go draft → active. The toggle between active and paused is a one-click action that writes immediately (no deploy needed) — used to take a pipeline out of service without redeploying.


sourcesback to docsnext: destinations