Use this data source to generate a Glue script from a Directed Acyclic Graph (DAG).
data "aws_glue_script" "example" {
language = "PYTHON"
dag_edge {
source = "datasource0"
target = "applymapping1"
}
dag_edge {
source = "applymapping1"
target = "selectfields2"
}
dag_edge {
source = "selectfields2"
target = "resolvechoice3"
}
dag_edge {
source = "resolvechoice3"
target = "datasink4"
}
dag_node {
id = "datasource0"
node_type = "DataSource"
args {
name = "database"
value = "\"${aws_glue_catalog_database.source.name}\""
}
args {
name = "table_name"
value = "\"${aws_glue_catalog_table.source.name}\""
}
}
dag_node {
id = "applymapping1"
node_type = "ApplyMapping"
args {
name = "mapping"
value = "[(\"column1\", \"string\", \"column1\", \"string\")]"
}
}
dag_node {
id = "selectfields2"
node_type = "SelectFields"
args {
name = "paths"
value = "[\"column1\"]"
}
}
dag_node {
id = "resolvechoice3"
node_type = "ResolveChoice"
args {
name = "choice"
value = "\"MATCH_CATALOG\""
}
args {
name = "database"
value = "\"${aws_glue_catalog_database.destination.name}\""
}
args {
name = "table_name"
value = "\"${aws_glue_catalog_table.destination.name}\""
}
}
dag_node {
id = "datasink4"
node_type = "DataSink"
args {
name = "database"
value = "\"${aws_glue_catalog_database.destination.name}\""
}
args {
name = "table_name"
value = "\"${aws_glue_catalog_table.destination.name}\""
}
}
}
output "python_script" {
value = data.aws_glue_script.example.python_script
}
data "aws_glue_script" "example" {
language = "SCALA"
dag_edge {
source = "datasource0"
target = "applymapping1"
}
dag_edge {
source = "applymapping1"
target = "selectfields2"
}
dag_edge {
source = "selectfields2"
target = "resolvechoice3"
}
dag_edge {
source = "resolvechoice3"
target = "datasink4"
}
dag_node {
id = "datasource0"
node_type = "DataSource"
args {
name = "database"
value = "\"${aws_glue_catalog_database.source.name}\""
}
args {
name = "table_name"
value = "\"${aws_glue_catalog_table.source.name}\""
}
}
dag_node {
id = "applymapping1"
node_type = "ApplyMapping"
args {
name = "mappings"
value = "[(\"column1\", \"string\", \"column1\", \"string\")]"
}
}
dag_node {
id = "selectfields2"
node_type = "SelectFields"
args {
name = "paths"
value = "[\"column1\"]"
}
}
dag_node {
id = "resolvechoice3"
node_type = "ResolveChoice"
args {
name = "choice"
value = "\"MATCH_CATALOG\""
}
args {
name = "database"
value = "\"${aws_glue_catalog_database.destination.name}\""
}
args {
name = "table_name"
value = "\"${aws_glue_catalog_table.destination.name}\""
}
}
dag_node {
id = "datasink4"
node_type = "DataSink"
args {
name = "database"
value = "\"${aws_glue_catalog_database.destination.name}\""
}
args {
name = "table_name"
value = "\"${aws_glue_catalog_table.destination.name}\""
}
}
}
output "scala_code" {
value = data.aws_glue_script.example.scala_code
}
dag_edge
- (Required) List of the edges in the DAG. Defined below.dag_node
- (Required) List of the nodes in the DAG. Defined below.language
- (Optional) Programming language of the resulting code from the DAG. Defaults to PYTHON
. Valid values are PYTHON
and SCALA
.source
- (Required) ID of the node at which the edge starts.target
- (Required) ID of the node at which the edge ends.target_parameter
- (Optional) Target of the edge.args
- (Required) Nested configuration an argument or property of a node. Defined below.id
- (Required) Node identifier that is unique within the node's graph.node_type
- (Required) Type of node this is.line_number
- (Optional) Line number of the node.name
- (Required) Name of the argument or property.value
- (Required) Value of the argument or property.param
- (Optional) Boolean if the value is used as a parameter. Defaults to false
.This data source exports the following attributes in addition to the arguments above:
id
- AWS Region.python_script
- Python script generated from the DAG when the language
argument is set to PYTHON
.scala_code
- Scala code generated from the DAG when the language
argument is set to SCALA
.