ingestlayer/entities

Entities.

Entities are our primitive for storing and retrieving data from inside your pipeline executions. Create or upsert entities from a pipeline and let events that follow utilize that data. You decide what's worth storing.

fig. 1 — an entity is a passport. A key on the cover, stamps that accumulate over time, presented and read at every crossing. The event carries nothing; the passport carries everything.

In the pipeline01

The one thing that persists.

Every other part of the pipeline is a step that runs once and forgets. Entities are the exception. They are written and read during transform, then read again at route — the one thing that outlives a single event. An event carries only what it arrived with; the entity is the memory the rest of the pipeline draws on.

Keying02

Keyed however you like.

An entity is a kind and an external key. You choose both. A person keyed by email, a company keyed by domain, an account keyed by its id, a session keyed by token. There is no fixed schema and no enum of allowed kinds — if you can name it and point at a field that identifies it, it's an entity.

kind is free text. The key is any $-rooted field on the event or context. One event can touch several entities at once.

Enrichment03

Stamps that stay.

upsert.entity maps event fields into traits and merges them onto the row. First-seen is preserved, last-seen advances, and the merge is one level deep — a partial update never blanks a trait you set on an earlier event. Provider enrichment stacks onto the same passport, so person and company lookups land next to the traits you wrote yourself.

step · upsert.entity
# pipelines/signup.yaml (excerpt)
transform:
  # stamp the passport — merge event fields onto person:<email>.
  # first-seen is preserved; last-seen advances; partial
  # updates never blank a trait you set before.
  - upsert.entity:
      kind:            person
      externalIdField: $event.email
      traitFields:
        plan:      $event.plan
        company:   $event.company
        last_seen: $event.received_at

Traits live on the entity, not the event. Write once; every future event keyed to the same passport reads it back.

Retrieval04

Read live, every run.

enrich.entity resolves the key field, looks the row up by (kind, external_id), and merges its traits into $entity.<kind>. The lookup is live on every event — no cache — so a trait written one step earlier is already readable in the next. From there it's in scope for every later action and every route template. A missing row is a no-op: downstream templates simply read undefined, nothing fails.

step · enrich.entity → route
# pipelines/signup.yaml (excerpt, continued)
transform:
  # read it back — live, no cache, every single run.
  # merges the row's traits into $entity.person.*
  # a missing row is a no-op, never an error.
  - enrich.entity:
      kind:            person
      externalIdField: $event.email

destinations:
  - slack:
      channel: "#growth"
      template: |
        {{ $event.email }} signed up on the
        {{ $entity.person.plan }} plan
        ({{ $entity.person.company }}).

Reference traits anywhere downstream as $entity.<kind>.<path> — in when clauses, in mustache templates, in the next action's input.

Capacity05

How many passports you keep.

The entity store holds a set number of distinct entities per plan. It counts unique passports, not lookups — enriching the same person across a million events is still one entity. Need more room, move up a tier. Whatever you store stays until you delete it; the free plan keeps entities for a month.

i.
starter
100 entities. A project, a beta cohort, a first hundred users.
ii.
team
10,000 entities. A working customer book.
iii.
scale
Unlimited. Every person, company, and account you touch.
← transformationdestinationthe full spec

Ready to give your events a memory?

sign upread the docs