airflow.gcp.sensors.pubsub

This module contains a Google PubSub sensor.

Module Contents

class airflow.gcp.sensors.pubsub.PubSubPullSensor(project, subscription, max_messages=5, return_immediately=False, ack_messages=False, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)[source]

Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator

Pulls messages from a PubSub subscription and passes them through XCom.

This sensor operator will pull up to max_messages messages from the specified PubSub subscription. When the subscription returns messages, the poke method’s criteria will be fulfilled and the messages will be returned from the operator and passed through XCom for downstream tasks.

If ack_messages is set to True, messages will be immediately acknowledged before being returned, otherwise, downstream tasks will be responsible for acknowledging them.

project and subscription are templated so you can use variables in them.

Parameters
  • project (str) – the GCP project ID for the subscription (templated)

  • subscription (str) – the Pub/Sub subscription name. Do not include the full subscription path.

  • max_messages (int) – The maximum number of messages to retrieve per PubSub pull request

  • return_immediately (bool) – If True, instruct the PubSub API to return immediately if no messages are available for delivery.

  • ack_messages (bool) – If True, each message will be acknowledged immediately rather than by any downstream tasks

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud Platform.

  • delegate_to (str) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

template_fields = ['project', 'subscription'][source]
ui_color = #ff7f50[source]
execute(self, context)[source]

Overridden to allow messages to be passed

poke(self, context)[source]