Skip to main content

Connector file

File nameResource
*.connector.yamlConnector

GitOps file for a single space.

Identity​

  • originKey: GitOps identity of the resource this file owns. The same value as the API's originKey.
  • id: Server-assigned id. Absent until the resource exists.

Fields​

FieldTypeRequiredDescription
configobject (four variants)Required
config[ObjectStorageConnector].bucketstringRequiredPattern: ^(?!xn--|sthree-|amzn-s3-demo-)(?!.*(?:-s3alias|--x-s3|--ol-s3|--table-s3)$)[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$. At most 63 characters.
config[ObjectStorageConnector].initialLookbackDaysintegerOptionalGreater than 0. Defaults to 14.
config[ObjectStorageConnector].manageBucketNotificationsbooleanOptionalWhether the data plane configures S3 event notifications on the bucket. Defaults to true when absent. When false the data plane ingests via periodic scans only. Defaults to true.
config[ObjectStorageConnector].objectStoreobjectRequired
config[ObjectStorageConnector].objectStore.authobjectRequired
config[ObjectStorageConnector].objectStore.auth.roleArnstringOptionalPattern: ^$|^(?:\w+:){3}([^:]*):(\d{12}):role\/(.*)$. Defaults to .
config[ObjectStorageConnector].prefixstringRequired
config[ObjectStorageConnector].regionstringOptionalDefaults to not-used.
config[ObjectStorageConnector].scanIntervalMinutesintegerOptionalHow often the data plane rescans the bucket, in minutes. Defaults to 1440 (daily) when absent. The scan is a safety net for objects that arrive without an event notification. Minimum 5. Defaults to 1440.
config[ObjectStorageConnector].typestringRequiredAlways ObjectStorageConnector.
config[ApiPollingConnector].connectorKindstringRequiredCatalog connector kind (e.g. "google-workspace", "gcp-pubsub") or "custom" for a customer ECR image. Must not be empty.
config[ApiPollingConnector].credentialIdsmap of stringOptional
config[ApiPollingConnector].ecrImageUrlstringOptionalECR image URL for a custom connector Lambda container. Pattern: ^[0-9]+\.dkr\.ecr\.[a-z0-9-]+\.amazonaws\.com\/.+$. Must not be empty.
config[ApiPollingConnector].initialLookbackDaysintegerOptionalGreater than 0.
config[ApiPollingConnector].intervalMinutesintegerRequiredMinimum 1.
config[ApiPollingConnector].settingsobjectRequired
config[ApiPollingConnector].typestringRequiredAlways ApiPollingConnector.
config[StreamingConnector].connectorKindstringRequiredCatalog connector kind (e.g. "google-workspace", "gcp-pubsub") or "custom" for a customer ECR image. Must not be empty.
config[StreamingConnector].credentialIdsmap of stringOptional
config[StreamingConnector].ecrImageUrlstringOptionalECR image URL for a custom connector Lambda container. Pattern: ^[0-9]+\.dkr\.ecr\.[a-z0-9-]+\.amazonaws\.com\/.+$. Must not be empty.
config[StreamingConnector].settingsobjectRequired
config[StreamingConnector].typestringRequiredAlways StreamingConnector.
config[HttpSourceConnector].authModeenumRequiredOne of bearer, presigned, hmac.
config[HttpSourceConnector].credentialIdstringOptionalPattern: ^CRD_[A-Z0-9]{1,7}$.
config[HttpSourceConnector].typestringRequiredAlways HttpSourceConnector.
datasetKindenumOptionalOne of event, state.
deploybooleanRequired
idstringOptionalServer-assigned id. Absent until the resource exists. Pattern: ^CON_[A-Z0-9]{1,7}$.
namestringRequiredPattern: ^connectors?\.([a-zA-Z_][a-zA-Z0-9_]{0,255})(\.([a-zA-Z_][a-zA-Z0-9_]{0,255}))?$.
originKeystringRequiredGitOps identity of the resource this file owns. The same value as the API's originKey. Pattern: ^(?!perpetual:)[a-z][a-z0-9-]{0,31}:[A-Za-z0-9._~\-/:@]{1,256}$.
spaceIdstringRequiredSpace the resource belongs to. Pattern: ^SPC_[A-Z0-9]{1,7}$.
typeenumRequiredOne of ObjectStorageConnector, ApiPollingConnector, StreamingConnector, HttpSourceConnector.

Perpetual CLI 0.5.0