google_composer_environment

An environment for running orchestration tasks.

Environments run Apache Airflow software on Google infrastructure.

To get more information about Environments, see:

We STRONGLY recommend you read the GCP guides as the Environment resource requires a long deployment process and involves several layers of GCP infrastructure, including a Kubernetes Engine cluster, Cloud Storage, and Compute networking resources. Due to limitations of the API, Terraform will not be able to find or manage many of these underlying resources automatically. In particular:

Example Usage

Basic Usage (Cloud Composer 1)

resource "google_composer_environment" "test" {
  name   = "example-composer-env"
  region = "us-central1"
}

Basic Usage (Cloud Composer 2)

resource "google_composer_environment" "test" {
  name   = "example-composer-env"
  region = "us-central1"
 config {
    software_config {
      image_version = "composer-2-airflow-2"
    }
  }
}

With GKE and Compute Resource Dependencies

NOTE To use custom service accounts, you must give at least role/composer.worker to the service account used by the GKE Nodes on the Composer project. For more information, see the Access Control page in the Cloud Composer documentation. You may need to assign additional roles depending on what the Airflow DAGs will be running.

GKE and Compute Resource Dependencies (Cloud Composer 1)

resource "google_composer_environment" "test" {
  name   = "example-composer-env"
  region = "us-central1"
  config {
    node_count = 4

    node_config {
      zone         = "us-central1-a"
      machine_type = "n1-standard-1"

      network    = google_compute_network.test.id
      subnetwork = google_compute_subnetwork.test.id

      service_account = google_service_account.test.name
    }

    database_config {
      machine_type = "db-n1-standard-2"
    }

    web_server_config {
      machine_type = "composer-n1-webserver-2"
    }
  }
}

resource "google_compute_network" "test" {
  name                    = "composer-test-network"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "test" {
  name          = "composer-test-subnetwork"
  ip_cidr_range = "10.2.0.0/16"
  region        = "us-central1"
  network       = google_compute_network.test.id
}

resource "google_service_account" "test" {
  account_id   = "composer-env-account"
  display_name = "Test Service Account for Composer Environment"
}

resource "google_project_iam_member" "composer-worker" {
  role   = "roles/composer.worker"
  member = "serviceAccount:${google_service_account.test.email}"
}

GKE and Compute Resource Dependencies (Cloud Composer 2)

provider "google" {
  project = "bigdata-writers"
}

resource "google_composer_environment" "test" {
  name   = "example-composer-env-tf-c2"
  region = "us-central1"
  config {

    software_config {
      image_version = "composer-2-airflow-2"
    }

    workloads_config {
      scheduler {
        cpu        = 0.5
        memory_gb  = 1.875
        storage_gb = 1
        count      = 1
      }
      web_server {
        cpu        = 0.5
        memory_gb  = 1.875
        storage_gb = 1
      }
      worker {
        cpu = 0.5
        memory_gb  = 1.875
        storage_gb = 1
        min_count  = 1
        max_count  = 3
      }


    }
    environment_size = "ENVIRONMENT_SIZE_SMALL"

    node_config {
      network    = google_compute_network.test.id
      subnetwork = google_compute_subnetwork.test.id
      service_account = google_service_account.test.name
    }
  }
}

resource "google_compute_network" "test" {
  name                    = "composer-test-network3"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "test" {
  name          = "composer-test-subnetwork"
  ip_cidr_range = "10.2.0.0/16"
  region        = "us-central1"
  network       = google_compute_network.test.id
}

resource "google_service_account" "test" {
  account_id   = "composer-env-account"
  display_name = "Test Service Account for Composer Environment"
}

resource "google_project_iam_member" "composer-worker" {
  project = "your-project-id"
  role    = "roles/composer.worker"
  member  = "serviceAccount:${google_service_account.test.email}"
}

With Software (Airflow) Config

resource "google_composer_environment" "test" {
  name   = "mycomposer"
  region = "us-central1"

  config {
    software_config {
      scheduler_count = 2 // only in Composer 1 with Airflow 2, use workloads_config in Composer 2
      airflow_config_overrides = {
        core-dags_are_paused_at_creation = "True"
      }

      pypi_packages = {
        numpy = ""
        scipy = "==1.1.0"
      }

      env_variables = {
        FOO = "bar"
      }
    }
  }
}

Argument Reference - Cloud Composer 1

The following arguments are supported:

The config block supports:

The node_config block supports:

The software_config block supports:

  AIRFLOW_HOME
  C_FORCE_ROOT
  CONTAINER_NAME
  DAGS_FOLDER
  GCP_PROJECT
  GCS_BUCKET
  GKE_CLUSTER_NAME
  SQL_DATABASE
  SQL_INSTANCE
  SQL_PASSWORD
  SQL_PROJECT
  SQL_REGION
  SQL_USER

See documentation for setting up private environments. The private_environment_config block supports:

The web_server_network_access_control supports:

The allowed_ip_range supports:

The ip_allocation_policy block supports:

The database_config block supports:

The web_server_config block supports:

The encryption_config block supports:

The maintenance_window block supports:

The master_authorized_networks_config block supports:

The cidr_blocks supports:

Argument Reference - Cloud Composer 2

The following arguments are supported:

The config block supports:

The data_retention_config block supports:

The task_logs_retention_config block supports:

The storage_config block supports:

The node_config block supports:

The software_config block supports:

  AIRFLOW_HOME
  C_FORCE_ROOT
  CONTAINER_NAME
  DAGS_FOLDER
  GCP_PROJECT
  GCS_BUCKET
  GKE_CLUSTER_NAME
  SQL_DATABASE
  SQL_INSTANCE
  SQL_PASSWORD
  SQL_PROJECT
  SQL_REGION
  SQL_USER

The cloud_data_lineage_integration block supports:

See documentation for setting up private environments. The private_environment_config block supports:

The ip_allocation_policy block supports:

The encryption_config block supports:

The maintenance_window block supports:

The recovery_config block supports:

The scheduled_snapshots_config block supports:

The workloads_config block supports:

The scheduler block supports:

The triggerer block supports:

The web_server block supports:

The worker block supports:

Argument Reference - Cloud Composer 3

Please note: This documentation corresponds to Composer 3, which is not yet released.

The following arguments are supported:

The config block supports:

The data_retention_config block supports:

The task_logs_retention_config block supports:

The storage_config block supports:

The node_config block supports:

The software_config block supports:

  AIRFLOW_HOME
  C_FORCE_ROOT
  CONTAINER_NAME
  DAGS_FOLDER
  GCP_PROJECT
  GCS_BUCKET
  GKE_CLUSTER_NAME
  SQL_DATABASE
  SQL_INSTANCE
  SQL_PASSWORD
  SQL_PROJECT
  SQL_REGION
  SQL_USER

The cloud_data_lineage_integration block supports:

The encryption_config block supports:

The maintenance_window block supports:

The workloads_config block supports:

The scheduler block supports:

The triggerer block supports:

The web_server block supports:

The worker block supports:

The dag_processor block supports:

Attributes Reference

In addition to the arguments listed above, the following computed attributes are exported:

Timeouts

This resource provides the following Timeouts configuration options: configuration options:

Import

Environment can be imported using any of these accepted formats:

In Terraform v1.5.0 and later, use an import block to import Environment using one of the formats above. For example:

import {
  id = "projects/{{project}}/locations/{{region}}/environments/{{name}}"
  to = google_composer_environment.default
}

When using the terraform import command, Environment can be imported using one of the formats above. For example:

$ terraform import google_composer_environment.default projects/{{project}}/locations/{{region}}/environments/{{name}}
$ terraform import google_composer_environment.default {{project}}/{{region}}/{{name}}
$ terraform import google_composer_environment.default {{name}}