google_data_pipeline_pipeline

The main pipeline entity and all the necessary metadata for launching and managing linked jobs.

To get more information about Pipeline, see:

Open in Cloud Shell

Example Usage - Data Pipeline Pipeline

resource "google_service_account" "service_account" {
  account_id   = "my-account"
  display_name = "Service Account"
}

resource "google_data_pipeline_pipeline" "primary" {
  name         = "my-pipeline"
  display_name = "my-pipeline"
  type         = "PIPELINE_TYPE_BATCH"
  state        = "STATE_ACTIVE"
  region       = "us-central1"

  workload {
    dataflow_launch_template_request {
      project_id = "my-project"
      gcs_path   = "gs://my-bucket/path"
      launch_parameters {
        job_name = "my-job"
        parameters = {
          "name" : "wrench"
        }
        environment {
          num_workers                = 5
          max_workers                = 5
          zone                       = "us-centra1-a"
          service_account_email      = google_service_account.service_account.email
          network                    = "default"
          temp_location              = "gs://my-bucket/tmp_dir"
          bypass_temp_dir_validation = false
          machine_type               = "E2"
          additional_user_labels = {
            "context" : "test"
          }
          worker_region    = "us-central1"
          worker_zone      = "us-central1-a"

          enable_streaming_engine = "false"
        }
        update                 = false
        transform_name_mapping = { "name" : "wrench" }
      }
      location = "us-central1"
    }
  }
  schedule_info {
    schedule = "* */2 * * *"
  }
}

Argument Reference

The following arguments are supported:


The workload block supports:

The dataflow_launch_template_request block supports:

The launch_parameters block supports:

The environment block supports:

The dataflow_flex_template_request block supports:

The launch_parameter block supports:

The environment block supports:

The schedule_info block supports:

Attributes Reference

In addition to the arguments listed above, the following computed attributes are exported:

Timeouts

This resource provides the following Timeouts configuration options:

Import

Pipeline can be imported using any of these accepted formats:

In Terraform v1.5.0 and later, use an import block to import Pipeline using one of the formats above. For example:

import {
  id = "projects/{{project}}/locations/{{region}}/pipelines/{{name}}"
  to = google_data_pipeline_pipeline.default
}

When using the terraform import command, Pipeline can be imported using one of the formats above. For example:

$ terraform import google_data_pipeline_pipeline.default projects/{{project}}/locations/{{region}}/pipelines/{{name}}
$ terraform import google_data_pipeline_pipeline.default {{project}}/{{region}}/{{name}}
$ terraform import google_data_pipeline_pipeline.default {{region}}/{{name}}
$ terraform import google_data_pipeline_pipeline.default {{name}}

User Project Overrides

This resource supports User Project Overrides.