Connectors

A connector is a dataset resource that pulls raw events from an external source into a Space and lands them as a connectors.<name> dataset. Its resource id is prefixed CON_.

A connector is the entry point of every ingest pipeline. It reads from one source (an S3 bucket, a polled API, a streaming broker, or an HTTP push endpoint) and writes the raw events it collects, without parsing them. An ingest table reads the connector's connectors.<name> dataset and shapes those raw events into typed columns. The connector's type fixes which source it reads and which config shape it takes. API-polling and streaming connectors run as Lambda functions built from the connector catalog and authenticate through credentials. An integration bundles a connector with the parser, mapper, and detections that process its output.

Variants

A connector comes in four variants, fixed at creation and determined by type. Each pairs with a matching config shape.

Config: ObjectStorageConnector

Scans an S3 bucket for new objects and ingests their contents.

Field	Type	Required	Description
type	enum	Required	Literal ObjectStorageConnector.
bucket	string	Required	S3 bucket name. Must be a valid bucket name.
prefix	string	Required	Key prefix within the bucket to scan.
region	string	Optional	AWS region. Default not-used.
objectStore.auth.roleArn	string	Optional	IAM role ARN the data plane assumes to read the bucket. Must be a valid role ARN.

Config: ApiPollingConnector

Runs a catalog connector as a Lambda function on a fixed interval.

Field	Type	Required	Description
type	enum	Required	Literal ApiPollingConnector.
connectorKind	string	Required	Catalog connector kind, such as google-workspace, or custom for a customer ECR image.
intervalMinutes	integer	Required	Minutes between polls.
settings	object	Required	Source-specific settings, validated against the catalog kind's settingsSchema.
credentialIds	object	Optional	Map of connector credential slot id to credential resource id.
ecrImageUrl	string	Optional	ECR image URL for the Lambda container. Required when connectorKind is custom. Rejected for built-in kinds, which resolve their image at invoke time.

Config: StreamingConnector

Runs a catalog connector as a Lambda function that consumes a stream continuously.

Field	Type	Required	Description
type	enum	Required	Literal StreamingConnector.
connectorKind	string	Required	Catalog connector kind, or custom for a customer ECR image.
settings	object	Required	Source-specific settings, validated against the catalog kind's settingsSchema.
credentialIds	object	Optional	Map of connector credential slot id to credential resource id.
ecrImageUrl	string	Optional	ECR image URL for the Lambda container. Required when connectorKind is custom. Rejected for built-in kinds.

Config: HttpSourceConnector

Hosts an HTTP push endpoint that the source sends events to. Perpetual generates the endpoint URL and its auth material at create time.

Field	Type	Required	Description
type	enum	Required	Literal HttpSourceConnector.
authMode	enum	Required	How the source authenticates to the endpoint. One of bearer (a generated bearer token), presigned (a generated URL token), hmac (an HMAC signature keyed by a credential).
credentialId	string	Optional	The HMAC credential resource id. Required when authMode is hmac. Rejected for bearer and presigned.

For bearer and presigned connectors, the create response returns a one-time generatedSecret or generatedIngestUrl. The stored config never exposes the token hashes.

Run status

Returned in runStatus for ApiPollingConnector and StreamingConnector when requested. Holds a lastRun and an activeRun, each nullable. Streaming connectors also carry currentLag and lagUnit.

Field	Type	Description
id	integer	Run identifier.
connectorResourceId	string	The connector the run belongs to.
startedAt	timestamp	When the run started.
finishedAt	timestamp	When the run finished. Null while running.
status	enum	running, succeeded, failed, or timed_out.
rowsEmitted	integer	Rows the run wrote. Null when unreported.
bytesEmitted	integer	Bytes the run wrote. Null when unreported.
error	string	Failure message. Null on success.
lastHeartbeatAt	timestamp	Last heartbeat the run reported. Null when unreported.
lag	number	How far the run is behind the source. Null when unreported.
lagUnit	enum	Unit of lag: milliseconds or messages. Null when unreported.
messagesConsumed	integer	Messages the streaming run consumed. Null when unreported.

Inputs

Field	Type	Required	Description
name	string	Required	Dataset name in connectors.<name> or connectors.<namespace>.<name> format. The connectors. prefix is required.
type	enum	Required	The connector type. One of ObjectStorageConnector, ApiPollingConnector, StreamingConnector, HttpSourceConnector. Fixes the shape of config.
config	object	Required	The source configuration for this connector's type. See the config tables under Variants.
deploy	boolean	Required	Whether the connector ingests. true runs the source and writes the dataset. false leaves it as saved content.

Outputs

Field	Type	Description
deploymentStatus	enum	Ingestion state: UNDEPLOYED, DEPLOY_PENDING, DEPLOY_IN_PROGRESS, DEPLOYED, UPDATE_PENDING, UPDATE_IN_PROGRESS, REFRESH_IN_PROGRESS, FAILED, ARCHIVED, CLEANUP_PENDING, CLEANUP_IN_PROGRESS.
runStatus	object	Last and active run summary for ApiPollingConnector and StreamingConnector. Returned only when the request asks for it. Absent for object-storage and HTTP-source connectors. See the run status table above.
readOnly	boolean	true when the connector cannot be edited in place, such as one owned by an integration.
isSystemCreated	boolean	true when the platform created the connector rather than a user.

Plus the common resource fields.

Related

• Connector Setup, the built-in sources an ApiPollingConnector or StreamingConnector is built from, with setup steps for each.
• Tables, the ingest table that parses a connector's raw events.
• Credentials, the auth a Lambda or HMAC connector references.
• Integrations, which bundle a connector with its parser, mapper, and detections.
• Changesets, how connector edits are staged and committed.
• Connect a built-in data source, how to configure and deploy a catalog connector.
• Ingest data over HTTP push, how to set up an HttpSourceConnector.