Data plane
A data plane is the deployed data-processing engine an
organization runs to ingest, materialize, and query its
events. Perpetual Cloud holds one resource per deployment, prefixed DPL_.
The data plane runs in the organization's own cloud account, on Kubernetes,
where it reads sources, materializes datasets,
and answers queries. Perpetual Cloud never runs that work. It holds the
authored content, tracks the deployment, and sends the data
plane the resources to deploy. The two are paired at install time through an
activation code that the running data plane exchanges
for a long-lived token, and they stay in sync through a periodic
heartbeat the data plane posts back. A data plane belongs to one
organization and is not content, so it is never staged
through a changeset. Managing it requires the
ManageDataPlane capability.
Activation and status
You create an empty data plane in Perpetual Cloud, then pair it with a
running deployment before it goes live. On create, unless the caller supplies a
token directly, Perpetual Cloud issues an activationCode that expires in
72 hours. The data plane presents that code once during install and receives a
long-lived token in exchange. The token authenticates every later request from
that deployment, including the heartbeat.
Perpetual Cloud derives the status field from the token and
activation-code state. The caller never sets it. It takes one of three values.
| Value | Meaning |
|---|---|
PENDING | Created but not yet activated. It holds an activation code and no token, so no deployment has exchanged the code yet. |
ACTIVE | Activated. It holds a token and no live activation code. This is a paired, running data plane. |
PENDING_REACTIVATION | Already activated once (it has a token) but now also holds an unused activation code, because someone regenerated one to re-pair the deployment. It returns to ACTIVE once the code is used. |
An owner can regenerate the activation code (issuing a fresh code with a new
72-hour expiry) or clear it (removing the code without pairing). Both act on the
activation state only. Regenerating a code for an already-active data plane is
how a reinstalled or migrated deployment re-pairs, which is the
PENDING_REACTIVATION case above.
Heartbeat
An active data plane posts a heartbeat to Perpetual Cloud on a schedule,
authenticated by its token. Each heartbeat reports the versions the deployment
is running (the release, Hamelin compiler, connector registry, and
Helm chart) and the cloud facts Perpetual Cloud needs to reach it (cluster
name, account, region, and the warehouse, OTEL, and exchange bucket names). The
Perpetual Cloud stores the reported values on the resource, which is where the
deployed* and infrastructure fields in Outputs come from. Two side
effects run off a heartbeat. When the warehouse and OTEL bucket names are
present, Perpetual Cloud bootstraps the organization's system ingest
connectors and
tables. When the account and region are present,
it accepts the cross-account resource share the data plane created at install so
Perpetual Cloud can reach the deployment's storage.
The deployedConnectorRegistryVersion a heartbeat reports is the CalVer of the
connector registry installed on that data plane. Perpetual Cloud uses it to
show only the connectors that deployment
supports, so an organization on an older data plane never sees a connector its
deployment cannot run.
Inputs
The fields the caller provides on create or update.
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Required | Display name of the data plane. |
Outputs
| Field | Type | Description |
|---|---|---|
status | enum | Deployment state, derived from the token and activation-code state. One of PENDING, ACTIVE, or PENDING_REACTIVATION. See Activation and status. |
activationCode | string | The one-time code a deployment exchanges for a token during install. Null once the data plane is activated with no pending re-pairing. |
activationCodeExpiresAt | timestamp | When the current activation code expires. Codes are issued with a 72-hour lifetime. Null when there is no active code. |
lastHeartbeatAt | timestamp | When the data plane last posted a heartbeat. Null before the first heartbeat. |
deployedVersion | string | The Perpetual data-plane release the deployment is running, as last reported by heartbeat. |
deployedHamelinVersion | string | The Hamelin compiler version the deployment is running. |
deployedConnectorRegistryVersion | string | The CalVer of the connector registry the deployment is running. Gates which connectors the organization can use. |
deployedChartVersion | string | The Helm chart version the deployment is running. |
clusterName | string | The Kubernetes cluster the deployment runs on, as last reported by heartbeat. |
bucketName | string | The warehouse bucket where the data plane materializes datasets. |
otelBucketName | string | The bucket the deployment writes OTEL telemetry to. |
accountId | string | The cloud account the deployment runs in. |
exchangeBucket | string | The bucket the data plane and Perpetual Cloud exchange query results through. |
httpSourceIngestBaseUrl | string | The base URL the deployment exposes for HTTP-push ingest. |
region | string | The cloud region the deployment runs in. |
Plus the common resource fields.
Related
- Organizations, the tenant a data plane belongs to. One data plane per organization.
- Connectors, the ingest sources the data plane reads. The reported connector-registry version gates which ones are available.
- Tables, the datasets the data plane materializes to its warehouse bucket.