Skip to main content

Incremental queries

An incremental query has a shape the materialization engine can evaluate from a new batch of rows and bounded earlier context, without recomputing its full configured source range. This distinction matters in Perpetual because continuous tables accept only incremental queries, while materialized views avoid full-range recreation during ordinary refreshes when their queries are incremental. Simba knows what makes a query incremental and can use Perpetual's query-validation tools to check a query's eligibility, so you do not need to apply these rules by hand.

A query is incremental when time is traceable and context is bounded​

With time traceable and context bounded, the platform can determine the finite source range that can affect a given output range.

  • Traceable time. To be incremental, each output row's timestamp must retain a computable relationship to the source timestamp. A query may preserve the timestamp or transform it in a way the platform can reverse for range calculation. Dropping it or deriving it in a way the platform cannot trace breaks incremental evaluation. Function calls in an incremental query must therefore be non-volatile. For example, the query cannot contain a call to now().
  • Bounded context. Row-local transformations are incremental because they require only the changed rows. Sliding windows, event correlations, and time-bucketed aggregates can also be incremental when the platform can compute a bounded input range around the change. If the platform cannot bound the required context, as with an aggregate over the entire input, the query is not incremental.

Materialized views fully recompute when their definitions change​

The platform uses those rules to determine whether it can bound an ordinary refresh. A materialized view's definition change has a different requirement: the stored result must reflect the current definition across what its sources still retain. The platform therefore rebuilds the view's configured range, which applies the new definition consistently across all source history the view can still reach.

That rebuild can reproduce only the history the sources still retain. Full re-derivation keeps the stored result aligned with the view's definition and sources at the cost of rescanning that history.

Continuous tables preserve what they have written​

A continuous table builds its output by appending incrementally derived rows. It never fully rebuilds its existing output from its sources. The platform applies definition changes on a go-forward basis and never changes rows after writing them.

This property is what allows continuous tables to retain rows for longer than their upstream dependencies. The platform does not need an expired source row to reproduce a derived row it has already stored. Signal datasets, for example, preserve detection history long after the raw events that produced their signals have expired.

Continuous tables require incrementally maintainable queries​

Because a continuous table may retain rows that its sources no longer hold, the platform must derive an incremental-refresh strategy for every continuous-table query. It checks this whenever the continuous table's specification changes. Full re-derivation cannot serve as a fallback, so the platform rejects a query it cannot evaluate incrementally. A transformation that requires full source access belongs in a view.

The platform bounds ordinary materialized-view refreshes for incremental queries​

A materialized view remains correct with an incremental or non-incremental query. During each refresh, the materialization engine reads only the affected source range plus bounded context and replaces only the affected output range when the query is incremental. When the query is non-incremental, the platform recreates the view from its full configured source range.

Recreating the full configured range requires more compute and takes longer as that range grows, so a non-incremental materialized view should generally refresh at most daily.

Even with an incremental query, changing a materialized view's definition or intentionally rebuilding it requires full re-derivation from the configured source range.