Skip to main content

Google Cloud Pub/Sub connector setup

Consumes messages from a Google Cloud Pub/Sub subscription and emits them to your dataset. This is a streaming connector (long-running consumption, not scheduled polling).

What you need first​

RequirementDetails
GCP projectProject containing the Pub/Sub subscription
SubscriptionExisting subscription attached to your topic (create the subscription before deploying)
Service accountWith permission to consume from that subscription
Message sourcePublisher writing events to the topic (log forwarder, log router, custom app, etc.)

1. Prepare Pub/Sub resources​

  1. Open the Google Cloud Console and select your project.
  2. Go to Pub/Sub → Subscriptions.
  3. Create or identify the subscription Perpetual will consume. Note:
    • Project ID
    • Subscription ID (short name, not the full path)
  4. Confirm messages are arriving (check Messages metrics or pull a test message).

Service account and IAM​

  1. Go to IAM & Admin → Service accounts → Create service account.
  2. Name it (e.g. perpetual-pubsub-consumer).
  3. Grant the service account Pub/Sub Subscriber (roles/pubsub.subscriber) on the project, or only on the specific subscription for least privilege.
  4. Create a JSON key for the service account (or copy the private_key and client_email fields).
  5. Record the service account email.

2. Perpetual credential​

  1. Log in to Perpetual. Open Settings from the user/org control at the bottom left of the sidebar (click your name → Settings), then go to Credentials → New Credential.
  2. Choose Text.
  3. Fill in:
FieldValue
NameUnique identifier using letters, digits, and underscores only (e.g. gcp_pubsub).
Allowed URL patterns*.googleapis.com
ValueService account private key PEM (the private_key field from the JSON key)
  1. Save the credential.

3. Create the Google Cloud Pub/Sub connector​

  1. Go to Datasets → Connectors → Create Connector.

  2. Select Google Cloud Pub/Sub.

  3. Connector name: unique identifier (letters, numbers, underscores).

  4. Configure:

    • Service account private key: select the credential from step 2.
    • Project ID: GCP project containing the subscription.
    • Subscription ID: subscription name to pull from.
    • Service account email: email of the service account (must match the private key).
    • Max messages per batch (optional): cap messages per pull (default 10).
    • Max bytes per batch (optional): cap total bytes per pull.

  5. Check Deploy to start the streaming connector after save.

The connector pulls messages, emits JSON records (id, data, publishTime), flushes to Firehose, then acks messages only after a successful flush.

Operational notes​

  • Cross-project subscriptions: If the subscription lives in a different project than the topic, grant roles/pubsub.subscriber in the subscription's project.
  • At-least-once delivery: Messages are acked after Firehose flush; failures nack for redelivery.
  • Lag metric: Heartbeats report consumer lag based on publishTime.

Reference​