Skip to main content

Enrichment

Events often lack information that security analysis needs. When they do, they need to be enriched with additional context. In Perpetual, you configure that enrichment with lookup content resources, which deploy tables or views that queries can join.

Context can change an event's significance​

An identifier tells a query which entity appears in an event. Context turns that identifier into attributes security analysis can use. A user ID can lead to account status, while a host name can lead to the sensitivity of the asset. A query can use these attributes even when the event source omits them.

Queries can use those attributes to answer questions the original event could not. Two failed logins may carry the same activity and differ only in the context joined to their user IDs. Account status turns one into a failed login from an active employee and the other into a failed login from a terminated employee. The activity looks identical until that context enters the analysis.

Lookup resources make context available for joins​

In Perpetual, you configure enrichment with a lookup content resource. The lookup defines a keyed reference dataset that queries can join to events. Once the lookup is deployed, its backing dataset becomes queryable in the lookups namespace. A static lookup deploys a table, while a computed lookup deploys a materialized view.

The Hamelin LOOKUP operator joins that dataset to the events flowing through a query. Its ON condition states how an event identifier matches a field in the lookup. A match attaches context to the event. Without a match, the event continues through the query without added context.

The event and lookup datasets enter the join, and every event leaves it. Events that match gain the context the next clause needs, letting WHERE select activity associated with terminated users. The lookup's key columns identify its rows. The ON condition independently defines how a query matches those rows to events.

Update lookup rows directly or derive them from other datasets​

You choose a static or computed variant when you create a lookup. Switching variants requires a new lookup.

  • Static. A static lookup gets its rows from an uploaded file. Its table holds those rows until another upload changes them. Choose this variant when the context should be an explicitly maintained snapshot.
  • Computed. A computed lookup gets its rows from a query over other datasets in Perpetual. Its materialized view updates as those datasets change. Choose this variant when the context should follow data that Perpetual already receives.

Queries consume both variants the same way: they read the lookup by name and apply its ON condition.

Update enrichment independently of event ingestion​

Both maintenance paths keep lookup rows separate from the events they enrich. Perpetual's design therefore encourages enriching events in downstream queries rather than during ingestion. Different downstream queries can enrich the same retained event, each with the context it needs. An investigation might join account status, while a detection joins asset sensitivity to evaluate the same activity from another angle.

Separating ingestion from enrichment also gives lookup rows time to become more precise and informative. Context need not be perfect when an event arrives because later queries can join retained events with improved lookup rows. The better context produces richer investigation results and gives detections more information for evaluating events.

This design improves enrichment by changing lookup rows instead of rewriting source events. Every downstream query that needs context must perform a join. The design therefore depends on Perpetual's query engine executing joins efficiently.