Event normalization

Event normalization is the common security analytics practice of mapping events from heterogeneous sources onto shared entity and event types so one query can analyze them together. Perpetual helps you build that shared shape while keeping its interpretation selective and reversible: you map only the fields that fit, then revise those query-time mappings across retained history. Simba can help you author the mappings and ensure that each source field is a semantic match for the shared schema field.

Shared entity and event types make cross-source analysis possible

Security sources describe the same entities and activities in different ways. One identity provider may record a person as an actor, while a cloud service records the same person as a user. Mapping both representations to a shared entity schema lets a query recognize that person across heterogeneous events. Because signals name the entities involved in detection findings, normalization also enables correlation across signals, or across events and signals.

Event types offer a narrower benefit because sources often describe activity at different levels of detail. Where the meanings do align, a detection can describe abstract behavior instead of enumerating each source's events. Once you map a new source to the same entities and behavior, existing cross-source analysis can include it without a source-specific rewrite or a separate detection. That reuse depends on every shared field preserving the source's meaning.

Perpetual encourages you to map only the fields that fit

Trustworthy normalization requires leaving some source fields unmapped. No shared schema models every detail that security sources produce, and a source field may resemble the nearest available destination while carrying a different meaning. Mapping it there would misstate what happened. Requiring every field to land somewhere rewards apparent completeness at the cost of semantic accuracy.

You should map a source field only when an ECS field intends to model exactly what the source provides. Before the mapping can take effect, Perpetual checks that the mapper uses valid ECS fields and types. That establishes structural conformance, but you still need to determine whether each source field carries the meaning ECS assigns to its target. Simba can compare source samples and field descriptions with the ECS definition, using its language model to reason about meaning that field-and-type validation cannot express.

A close-enough mapping can do more harm than leaving the field unmapped because downstream analysis will trust meaning the source field does not carry. The parser table preserves an unmapped source field in its original shape, where queries can still use it. The normalized view should contain only entity and event types that genuinely share meaning across sources.

Use the normalized view for analysis that spans sources and the parser table for source-specific detail. A good normalization earns trust through the meaning of its shared fields, regardless of how many source fields it maps.

Event normalization belongs at query time

Perpetual keeps that shared interpretation correctable by applying normalization at query time. During ingestion, Perpetual parses each source event into typed fields that retain the source's shape. A mapper applies the shared schema when a query reads the normalized view. Changing the mapper reinterprets retained history in place, which lets the same source-shaped fields support a corrected mapping or an alternative schema. Mapping errors and schema choices remain reversible because normalization does not rewrite the stored event.

Query-time normalization appears in cross-source query plans as field renames, nested struct construction, projections, and transformations between each table scan and the union that combines the sources.

The arrows trace optimizer pushdown rather than row flow. Eligible filters and projections travel from the cross-source query through the union and each branch's normalization layers toward both table scans. During execution, rows flow upward.

Perpetual's optimizer expects this normalization-heavy pattern. It pushes filters and projections back through field renames and constructed structs, across the union branches, and toward the scans. That reduces the fields and rows that later operations carry. Perpetual has focused optimizer work on plans with these layers because security analytics depends heavily on normalization. That focus makes query-time mapping practical even when a mapper builds a large shared event shape. The tradeoff remains in the query plan: performance depends on the optimizer recognizing and reducing these normalization layers.