Skip to main content

Data dependencies

A data dependency is the relationship between a dataset and the datasets it is defined in terms of. The dependencies across every dataset in a deployment add up to one dataset dependency graph, and that graph is how Perpetual knows what to compute and when, with nothing for you to schedule.

A dataset's definition names its sources​

Almost every dataset in Perpetual reads other datasets. A view resolves a query over tables. A union merges the datasets it names. A detection reads normalized events and lands its signals in a dataset of its own, which a second detection can read as its input. Each of those relationships is a dependency, pointing from the dataset you defined back to the datasets it reads.

The dependency lives inside the definition. For a dataset defined by a query, Perpetual reads the sources out of the query and records them on the resource, so there is no separate edge list to keep in step with the logic. A union has no query, so it names its sources directly, and that naming is the whole definition. Dataset resources documents the field each kind records those sources in.

Follow a chain of dependencies upstream and it ends at a dataset that reads nothing else, usually a connector carrying rows from an external source. Everything above that is derived, and each dependency points from a dataset back to what it was defined from, so the graph never loops. It is a directed acyclic graph, a DAG.

A node in the graph is a dataset and the work that fills it​

Each node is both a place rows live and the process that puts them there. The two arrive together. Deploying a resource stands up the dataset and the work that maintains it, and no other process writes into a dataset that something else declared.

A process fills exactly one dataset. Rows cannot fork out of one node into two destinations, so the same events in two shapes are two datasets, each declaring the upstream source it reads. That constraint is what makes the graph legible. To find out how a dataset got its rows, you read its definition, because nothing else could have written them.

Perpetual derives the evaluation order from the graph​

Transformation in Perpetual is declarative. You state what each dataset is in terms of others, and the engine derives the graph from those statements, then decides which datasets a batch of new upstream rows affects and in what order to bring them up to date. There is no place to write "run this, then that", and nothing for you to put on a schedule.

The lever you do hold over timing is a ceiling on staleness. A materialized dataset carries a maximum age its rows may reach, and the engine schedules refreshes to stay inside it. Incremental queries covers how a refresh folds new upstream rows into a dataset that already holds rows.

Tracking the graph is what keeps derived data from drifting​

The alternative to a tracked graph is a pile of scheduled searches, each writing its output where the next one reads. That is a dependency graph in effect, and nothing in the system holds it. Every search runs on its own clock over whatever sits in its input at the moment it fires. An event that arrives late, or out of order, lands after the search has already passed the window that would have caught it, and a delayed run shifts every window below it. Nothing raises an error. The rows are absent, and the only way to find out is to go looking.

Perpetual tracks the graph, so the engine knows which upstream rows each derived dataset has already accounted for. Out-of-order arrival, late arrival, and scheduling delay in the materialization system change when the engine folds a row in. Every row still reaches every dataset that depends on it.

The cost is imperative control. You cannot pin one dataset to refresh ahead of another or hand-tune the minute a particular node runs. The engine owns order and timing, in exchange for the guarantee that nothing goes missing between one dataset and the next.

The guarantee covers declared dependencies. Two datasets that describe the same events without either one reading the other share no edge, and the engine has no reason to keep them consistent with each other.