Skip to main content

Detections

A detection is a Hamelin query that Perpetual incrementally refreshes over events, lookups, signals, or summary views and records its results as signals. A signal can feed other detections or, when the detection is configured to produce alerts, enter triage as an alert.

Most detections use one of four query shapes​

A detection's Hamelin query defines how input rows become signals. Four query shapes account for most detection logic.

Simple filter. A simple filter defines a signal from one input row that meets a condition. The condition can still be selective or complex. This fragment recognizes a long encoded PowerShell command in one process event:

...
| WHERE regexp_like(
    process.command_line,
    '(?i)-enc(?:odedcommand)?\s+[A-Za-z0-9+/]{80,}={0,2}'
  )

Time-series aggregate. A time-series aggregate defines a signal as a condition on an aggregate value over a calendar interval. Counts crossing a threshold are common, such as five push notifications for one user within an hour:

...
| AGG push_count = count()
  BY user.name, timestamp@h
| WHERE push_count >= 5

Sliding-window aggregate. The same aggregate shape can use a sliding window to catch qualifying activity that a calendar boundary splits across two intervals:

...
| WINDOW push_count = count()
  BY user.name WITHIN -1h
| WHERE push_count >= 5

Ordered correlation. An ordered correlation recognizes activities that occur in a specified sequence within a window. Here, run-dialog activity must precede one or more outbound connections on the same host:

DEF run_dialog = ...;
DEF outbound_connection = ...;

MATCH run_dialog outbound_connection+
BY host.name WITHIN 5m

Unordered correlation. An unordered correlation recognizes activities within the same window regardless of their order. This fragment requires a registry change and a process start on the same host within one second:

DEF registry_change = ...;
DEF process_start = ...;

FROM registry = registry_change,
     process = process_start
| WINDOW
    registry = last_value(registry),
    process = last_value(process)
  BY host.name WITHIN 1s
| WHERE registry IS NOT NULL
    AND process IS NOT NULL

Events, upstream signals, or both can serve as inputs to any of these shapes. Hamelin supports detection queries beyond these four common shapes. Detections can be defined by any incremental query.

Detections record results as signals​

A detection query is responsible for outputting signals. Each output row must conform to the signal schema and identify the specific detected activity along with the entities involved. These rows land in the detection's derived continuous table.

The query shape determines which evidence the signal can carry:

  • Simple filter. The output can include fields from its input row, such as the process name and command line that satisfied the filter.
  • Aggregate. The output can include group keys and calculated values, such as the user name and push count from the aggregate above.
  • Correlation. The query has access to the individual records that contribute to the correlation. Its output can embed those records, but often selects specific values with aggregates such as first_value or last_value.

Perpetual keeps each signal for the period set by the detection's retention setting. The signal schema defines the fields available during that period.

Signals can feed downstream detections and alert triage​

Other detections can read a detection's retained signals through its signal table. This allows simpler detections to create signal data for narrow observations and a more complex detection to combine those observations with other activity into a complete scenario.

Alerts are configured for a detection as a whole. Detections for narrow observations can create signals without alerts, while a detection for the complete scenario can send every signal it creates to triage.

Perpetual incrementally refreshes detections​

Perpetual creates a detection's derived continuous table after the detection is marked as deployed and the change is committed. Its incremental query allows Perpetual's query engine to refresh that table by touching only newly arrived upstream rows with temporally bounded context. Perpetual refreshes the detection as needed to keep its signals within the configured maximum materialization latency. Perpetual's query engine is designed to execute filters, windowed aggregates, and windowed correlations incrementally because the common detection shapes require those query styles.

The engine cannot incrementally refresh a continuous table whose query requires more than temporally bounded context. Such a result would depend on assumptions about how long upstream tables retain their rows, preventing the engine from determining a finite source range for each refresh.

Materialized views can cover many non-incremental query shapes by producing summaries that update occasionally. A detection can join one of those summaries with its other inputs. A materialized view used this way should refresh infrequently, generally at most daily, because recreating the view requires more compute as its configured source range grows.