Skip to main content

Azure Event Hubs setup

Azure Event Hubs connector setup​

Consumes events from an Azure Event Hub via a consumer group. This is a streaming connector (long-running consumption, not scheduled polling).

What you need first​

RequirementDetails
Event Hubs namespaceActive namespace with your event hub
Event hubHub receiving events from your publishers
Connection stringWith Listen permission on the event hub
Consumer groupDedicated group recommended (default $Default works)

1. Get Event Hub connection details​

  1. Sign in to the Azure portal.
  2. Open your Event Hubs Namespace → Event Hubs → select your hub.
  3. Note the Event hub name.
  4. Under Settings → Shared access policies, create or open a policy with Listen claim.
  5. Copy the Connection string–primary key (or namespace-level connection string with Listen on this hub).
  1. In the event hub, open Consumer groups.
  2. Add a dedicated group (e.g. perpetual) instead of sharing $Default with other consumers.

The connection string embeds the namespace (*.servicebus.windows.net); you do not enter the namespace separately.

2. Perpetual credential​

  1. Log in to Perpetual. Open Settings from the user/org control at the bottom left of the sidebar (click your name → Settings), then go to Credentials → New Credential.
  2. Choose Text.
  3. Fill in:
FieldValue
NameUnique identifier using letters, digits, and underscores only (e.g. azure_event_hub).
Allowed URL patterns*.servicebus.windows.net
ValueEvent Hubs connection string from step 1
  1. Save the credential.

3. Create the Azure Event Hubs connector​

  1. Go to Datasets → Connectors → Create Connector.
  2. Select Azure Event Hubs.
  3. Connector name: unique identifier (letters, numbers, underscores).
  4. Configure:
    • Event Hub connection string: select the credential from step 2.
    • Event Hub name: hub name from step 1.
    • Consumer group: group to use (default $Default, or your dedicated group).
  5. Check Deploy to start the streaming connector after save.

Partition checkpoints are stored automatically by the data plane (S3-backed). You do not configure checkpoint storage in the connector settings.

Microsoft security sources​

Use Event Hubs for Microsoft security data that Microsoft exposes as append-only logs or event streams. This avoids timestamp-window polling and uses Event Hub partition offsets for bounded, resumable delivery.

The connector emits one raw Event Hub message to azure.eventhub.<eventHubName>. The original Microsoft payload remains under the record's body; Event Hub enqueue time, sequence number, and offset are retained alongside it. Choose stable, descriptive hub names because the hub name becomes the dataset suffix.

Microsoft Entra diagnostic logs​

In the Microsoft Entra admin center, open Monitoring & health → Diagnostic settings, add a diagnostic setting, and select Stream to an event hub. Route the required categories:

Perpetual dataMicrosoft diagnostic category
Directory audit activityAuditLogs
Interactive user sign-insSignInLogs
Non-interactive user sign-insNonInteractiveUserSignInLogs
Service-principal sign-insServicePrincipalSignInLogs
Managed-identity sign-insManagedIdentitySignInLogs
Risky usersRiskyUsers
User risk detectionsUserRiskEvents
Risky service principalsRiskyServicePrincipals
Service-principal risk detectionsServicePrincipalRiskEvents

Microsoft also supports categories such as ProvisioningLogs and MicrosoftGraphActivityLogs; enable them when those records are in scope. Available categories and licensing vary by tenant.

Use Event Hub as the only ongoing collector for these categories. Running equivalent Microsoft Graph polling in parallel creates overlapping records with different raw schemas. Diagnostic export begins from the point it is enabled and should not be assumed to provide a historical snapshot of current risky identities.

See Microsoft Entra logs available for streaming and stream Microsoft Entra logs to Event Hubs.

Microsoft Defender XDR event streaming​

In the Microsoft Defender portal, configure Streaming API to send selected Advanced Hunting tables to the Event Hub. Supported tables include AlertInfo, AlertEvidence, device, email, identity, URL, and cloud-app event tables.

These events supplement the Microsoft Graph Security connector. AlertInfo and AlertEvidence are Advanced Hunting records, not the same objects as Graph /security/alerts_v2, and Event Streaming does not provide the same correlated /security/incidents resource.

See Microsoft Defender XDR Event Streaming and supported Event Streaming tables.

Delivery expectations​

  • Event Hubs and the connector provide at-least-once delivery, not exactly-once delivery.
  • The connector flushes emitted records before updating the partition checkpoint. A crash after a successful flush but before checkpointing can replay the final uncommitted batch.
  • Use a dedicated consumer group for every Perpetual connector instance.
  • Prefer separate Event Hubs when sources require different retention, access, throughput, or dataset names.

Operational notes​

  • Delivery semantics: Events are emitted to Firehose, flushed, then checkpointed per partition. A flush failure does not advance the checkpoint.
  • Multiple consumers: Use a dedicated consumer group per downstream consumer to avoid partition stealing.
  • Connection string rotation: Update the shared access policy keys in Azure, then update the Perpetual credential.

Reference​