Data flow

Data follows a consistent path through Perpetual. A connector brings external events into Perpetual, and deployable content defines the processing stages that follow. An integration groups the parsers that extract typed fields from one source with the mappers that give those fields shared names. Detections materialize matches as signals.

Together, parsers, mappers, lookups, and detections specialize Perpetual's generic dependency and materialization system for security analytics. Parsers, mappers, and detections each define one conceptual processing stage. A Space can contain many nodes at each stage, for example, one for each distinct data source or detection. When deployed, each node consists of one dataset together with the work that maintains it.

These nodes form the Space's dependency graph, where queries define data flow by depending on other datasets. Data flows in the opposite direction of dependency: a query that depends on a dataset pulls rows from that dataset in order to process them.

A lookup can supply reference data to a detection (as shown). Use of lookups is not limited to detections, of course.

Connectors provide access to events outside Perpetual

A connector gives a Space access to raw events held in one external source. It makes those source rows available under connectors.<name>, preserving their original shape until a parser extracts typed fields.

Connector datasets are queryable, but they are not intended to be the primary way to work with source data. Direct queries let you develop and validate a parser against staged source rows before Perpetual ingests those rows into the parser's event table.

A connector's type determines its transport mechanism. Perpetual can read object storage, poll an API, consume a stream, or accept events through HTTP push.

Parsers extract typed fields from raw events

A parser is a Hamelin query that reads raw connector rows and extracts named, typed columns. A deployed parser runs as an ingest table at events.<integration>_<parser>, materializing parsed rows as source events arrive.

Changes to parser logic do not rewrite rows already in the ingest table. The new logic applies only to events that arrive afterward.

An integration can contain several parsers when its source emits several event shapes or multiplexes many distinct events through one transport. Each deployed parser materializes its own typed event table, giving later queries stable fields to address by name and type.

Mappers give source-specific fields shared names

Typed parser tables still carry the source's vocabulary. A mapper is a Hamelin query that reads one of those tables and restates its fields under shared ECS names. A mapper should only read one parser table. Its query fits that table's output shape into the normalized schema.

An integration can contain several mappers. Each deployed mapper contributes one named branch to the integration's ECS view, events_ecs.<integration>. The view is virtual: it evaluates mapper transformations at query time rather than storing rows of its own.

Mapper changes take effect on the next query against the ECS view, including for rows ingested before the change. Queries can also read a parser table directly when they need its source-specific fields.

Queries use lookups to add reference data

A lookup makes a keyed reference dataset available to queries at lookups.<name>. A query can join it to an input value, such as an IP address or user ID, to add reference data not carried by the input dataset.

Lookup data can come from a file upload or a query. A static lookup stores the uploaded rows, while a computed lookup materializes the result of a query over existing datasets. Both forms give later queries a named set of reference rows to read.

Lookups are allowed anywhere Hamelin is allowed, other than in parsers, which must operate within a tight operational envelope because they run as part of Perpetual's ingest system. This includes the continuous tables used by detections.

Detections continuously materialize matches as signals

A detection is a Hamelin query that emits a signal each time its conditions match. A detection can read parser tables, ECS views, or another detection's signals. Regardless of its query or source, it shapes every match into the shared signal schema.

A deployed detection runs as a continuous table at signals.<detection> and materializes its signals incrementally. The query must therefore be eligible for incremental analysis, which allows Perpetual to update the signal table as new rows arrive. A configurable staleness ceiling controls how current the materialized signals must remain.

Detection changes apply only to new signals. Perpetual does not re-derive the continuous table after a definition change. Signals already in the table remain unchanged, stay queryable, and can become the input to a higher-level detection.

Every dataset is queryable

Each dataset is queryable, whether as input to another transformation or interactively. Connector datasets use connectors, parser tables use events, integration ECS views use events_ecs, lookup datasets use lookups, and detection tables use signals.

Perpetual also keeps two Space-wide datasets alongside the per-resource ones. The bare events_ecs combines every integration's ECS view, while the bare signals combines every detection's signal table. These names are distinct from a per-resource name such as events_ecs.<integration> or signals.<detection>.

Both Space-wide datasets are virtual unions. They resolve against their member datasets when a query reads them and hold no rows of their own. The unions provide cross-resource query surfaces without materializing another copy of their members' rows.