google_dataflow_job

Creates a job on Dataflow, which is an implementation of Apache Beam running on Google Compute Engine. For more information see the official documentation for Beam and Dataflow.

Example Usage

resource "google_dataflow_job" "big_data_job" {
  name              = "dataflow-job"
  template_gcs_path = "gs://my-bucket/templates/template_file"
  temp_gcs_location = "gs://my-bucket/tmp_dir"
  parameters = {
    foo = "bar"
    baz = "qux"
  }
}

Example Usage - Streaming Job

resource "google_pubsub_topic" "topic" {
    name     = "dataflow-job1"
}
resource "google_storage_bucket" "bucket1" {
    name          = "tf-test-bucket1"
    location      = "US"
    force_destroy = true
}
resource "google_storage_bucket" "bucket2" {
    name          = "tf-test-bucket2"
    location      = "US"
    force_destroy = true
}
resource "google_dataflow_job" "pubsub_stream" {
    name = "tf-test-dataflow-job1"
    template_gcs_path = "gs://my-bucket/templates/template_file"
    temp_gcs_location = "gs://my-bucket/tmp_dir"
    enable_streaming_engine = true
    parameters = {
      inputFilePattern = "${google_storage_bucket.bucket1.url}/*.json"
      outputTopic    = google_pubsub_topic.topic.id
    }
    transform_name_mapping = {
        name = "test_job"
        env = "test"
    }
    on_delete = "cancel"
}

Note on "destroy" / "apply"

There are many types of Dataflow jobs. Some Dataflow jobs run constantly, getting new data from (e.g.) a GCS bucket, and outputting data continuously. Some jobs process a set amount of data then terminate. All jobs can fail while running due to programming errors or other issues. In this way, Dataflow jobs are different from most other Terraform / Google resources.

The Dataflow resource is considered 'existing' while it is in a nonterminal state. If it reaches a terminal state (e.g. 'FAILED', 'COMPLETE', 'CANCELLED'), it will be recreated on the next 'apply'. This is as expected for jobs which run continuously, but may surprise users who use this resource for other kinds of Dataflow jobs.

A Dataflow job which is 'destroyed' may be "cancelled" or "drained". If "cancelled", the job terminates - any data written remains where it is, but no new data will be processed. If "drained", no new data will enter the pipeline, but any data currently in the pipeline will finish being processed. The default is "drain". When on_delete is set to "drain" in the configuration, you may experience a long wait for your terraform destroy to complete.

You can potentially short-circuit the wait by setting skip_wait_on_job_termination to true, but beware that unless you take active steps to ensure that the job name parameter changes between instances, the name will conflict and the launch of the new job will fail. One way to do this is with a random_id resource, for example:

variable "big_data_job_subscription_id" {
  type    = string
  default = "projects/myproject/subscriptions/messages"
}

resource "random_id" "big_data_job_name_suffix" {
  byte_length = 4
  keepers = {
    region          = var.region
    subscription_id = var.big_data_job_subscription_id
  }
}
resource "google_dataflow_flex_template_job" "big_data_job" {
  provider                      = google-beta
  name                          = "dataflow-flextemplates-job-${random_id.big_data_job_name_suffix.dec}"
  region                        = var.region
  container_spec_gcs_path       = "gs://my-bucket/templates/template.json"
  skip_wait_on_job_termination = true
  parameters = {
    inputSubscription = var.big_data_job_subscription_id
  }
}

Argument Reference

The following arguments are supported:


Attributes Reference

Import

Dataflow jobs can be imported using the job id e.g.

In Terraform v1.5.0 and later, use an import block to import dataflow jobs using one of the formats above. For example:

import {
  id = "{{id}}"
  to = google_dataflow_job.default
}

When using the terraform import command, dataflow jobs can be imported using one of the formats above. For example:

$ terraform import google_dataflow_job.default {{id}}