Skip to main content

Dataset resources

A dataset is a named, queryable collection of typed rows in the catalog, addressed by either a top-level identifier or a schema-qualified name and backed by a dataset resource.

Datasets are the data side of Perpetual, the counterpart to authored content. Every dataset appears in the catalog as an entry with typed columns, a resource id, and the Space that holds it. A dataset resource defines a dataset. Connectors, tables, views, and unions are the four kinds, one per type a catalog entry can carry. A query reads datasets by name, and the data dependencies between them form the graph that carries events from a source through to signals.

Shared mechanics​

These facts hold across all four dataset resources.

  • Catalog entry. Each dataset resource produces one catalog entry keyed by its name. The entry carries the resource id, the dataset type (connector, table, view, or union), the typed columns, the fromDatasets it reads, and the spaceId.
  • Typed columns. Column types come from the Hamelin type system: scalars such as string, int, double, boolean, and timestamp, along with array, map, struct, and other composite types.
  • deploy gates materialization. A dataset resource is saved content until deploy is true. Deploying runs the source or query and writes or resolves the dataset. An undeployed resource has no queryable data.
  • Space-scoped. A dataset belongs to one Space. A query in a Space reads the datasets in that Space plus those in upstream Spaces authorized to flow data into it.
  • Version-controlled. You stage edits to a dataset resource in a changeset, and they take effect on commit.

The dataset resources​

  • Connectors pull raw events from an external source into a Space and land them as a connectors.<name> dataset.
  • Tables are datasets defined by a Hamelin query that materializes its output to storage under a catalog name you choose.
  • Views are datasets defined by a Hamelin query under a catalog name you choose, resolved at query time rather than persisted by default.
  • Unions combine source datasets with compatible, overlapping schemas into one queryable dataset. Fields absent from a member are null for that member's rows.
  • Content, the authored side that reads and writes datasets.
  • Deployable content, how datasets differ from the authored content that operates on them.
  • Data flow, how events move from a connector through tables and views to signals.
  • Data dependencies, the graph the fromDatasets of these resources describe.
  • Connector Setup, the built-in sources a connector is built from, with setup steps for each.