Manages an Amazon MSK cluster.
resource "aws_vpc" "vpc" {
cidr_block = "192.168.0.0/22"
}
data "aws_availability_zones" "azs" {
state = "available"
}
resource "aws_subnet" "subnet_az1" {
availability_zone = data.aws_availability_zones.azs.names[0]
cidr_block = "192.168.0.0/24"
vpc_id = aws_vpc.vpc.id
}
resource "aws_subnet" "subnet_az2" {
availability_zone = data.aws_availability_zones.azs.names[1]
cidr_block = "192.168.1.0/24"
vpc_id = aws_vpc.vpc.id
}
resource "aws_subnet" "subnet_az3" {
availability_zone = data.aws_availability_zones.azs.names[2]
cidr_block = "192.168.2.0/24"
vpc_id = aws_vpc.vpc.id
}
resource "aws_security_group" "sg" {
vpc_id = aws_vpc.vpc.id
}
resource "aws_kms_key" "kms" {
description = "example"
}
resource "aws_cloudwatch_log_group" "test" {
name = "msk_broker_logs"
}
resource "aws_s3_bucket" "bucket" {
bucket = "msk-broker-logs-bucket"
}
resource "aws_s3_bucket_acl" "bucket_acl" {
bucket = aws_s3_bucket.bucket.id
acl = "private"
}
data "aws_iam_policy_document" "assume_role" {
statement {
effect = "Allow"
principals {
type = "Service"
identifiers = ["firehose.amazonaws.com"]
}
actions = ["sts:AssumeRole"]
}
}
resource "aws_iam_role" "firehose_role" {
name = "firehose_test_role"
assume_role_policy = data.aws_iam_policy_document.assume_role.json
}
resource "aws_kinesis_firehose_delivery_stream" "test_stream" {
name = "terraform-kinesis-firehose-msk-broker-logs-stream"
destination = "extended_s3"
extended_s3_configuration {
role_arn = aws_iam_role.firehose_role.arn
bucket_arn = aws_s3_bucket.bucket.arn
}
tags = {
LogDeliveryEnabled = "placeholder"
}
lifecycle {
ignore_changes = [
tags["LogDeliveryEnabled"],
]
}
}
resource "aws_msk_cluster" "example" {
cluster_name = "example"
kafka_version = "3.2.0"
number_of_broker_nodes = 3
broker_node_group_info {
instance_type = "kafka.m5.large"
client_subnets = [
aws_subnet.subnet_az1.id,
aws_subnet.subnet_az2.id,
aws_subnet.subnet_az3.id,
]
storage_info {
ebs_storage_info {
volume_size = 1000
}
}
security_groups = [aws_security_group.sg.id]
}
encryption_info {
encryption_at_rest_kms_key_arn = aws_kms_key.kms.arn
}
open_monitoring {
prometheus {
jmx_exporter {
enabled_in_broker = true
}
node_exporter {
enabled_in_broker = true
}
}
}
logging_info {
broker_logs {
cloudwatch_logs {
enabled = true
log_group = aws_cloudwatch_log_group.test.name
}
firehose {
enabled = true
delivery_stream = aws_kinesis_firehose_delivery_stream.test_stream.name
}
s3 {
enabled = true
bucket = aws_s3_bucket.bucket.id
prefix = "logs/msk-"
}
}
}
tags = {
foo = "bar"
}
}
output "zookeeper_connect_string" {
value = aws_msk_cluster.example.zookeeper_connect_string
}
output "bootstrap_brokers_tls" {
description = "TLS connection host:port pairs"
value = aws_msk_cluster.example.bootstrap_brokers_tls
}
resource "aws_msk_cluster" "example" {
cluster_name = "example"
kafka_version = "2.7.1"
number_of_broker_nodes = 3
broker_node_group_info {
instance_type = "kafka.m5.4xlarge"
client_subnets = [
aws_subnet.subnet_az1.id,
aws_subnet.subnet_az2.id,
aws_subnet.subnet_az3.id,
]
storage_info {
ebs_storage_info {
provisioned_throughput {
enabled = true
volume_throughput = 250
}
volume_size = 1000
}
}
security_groups = [aws_security_group.sg.id]
}
}
This resource supports the following arguments:
broker_node_group_info
- (Required) Configuration block for the broker nodes of the Kafka cluster.cluster_name
- (Required) Name of the MSK cluster.kafka_version
- (Required) Specify the desired Kafka software version.number_of_broker_nodes
- (Required) The desired total number of broker nodes in the kafka cluster. It must be a multiple of the number of specified client subnets.client_authentication
- (Optional) Configuration block for specifying a client authentication. See below.configuration_info
- (Optional) Configuration block for specifying a MSK Configuration to attach to Kafka brokers. See below.encryption_info
- (Optional) Configuration block for specifying encryption. See below.enhanced_monitoring
- (Optional) Specify the desired enhanced MSK CloudWatch monitoring level. See Monitoring Amazon MSK with Amazon CloudWatchopen_monitoring
- (Optional) Configuration block for JMX and Node monitoring for the MSK cluster. See below.logging_info
- (Optional) Configuration block for streaming broker logs to Cloudwatch/S3/Kinesis Firehose. See below.storage_mode
- (Optional) Controls storage mode for supported storage tiers. Valid values are: LOCAL
or TIERED
.tags
- (Optional) A map of tags to assign to the resource. If configured with a provider default_tags
configuration block present, tags with matching keys will overwrite those defined at the provider-level.client_subnets
- (Required) A list of subnets to connect to in client VPC (documentation).instance_type
- (Required) Specify the instance type to use for the kafka brokersE.g., kafka.m5.large. (Pricing info)security_groups
- (Required) A list of the security groups to associate with the elastic network interfaces to control who can communicate with the cluster.az_distribution
- (Optional) The distribution of broker nodes across availability zones (documentation). Currently the only valid value is DEFAULT
.connectivity_info
- (Optional) Information about the cluster access configuration. See below. For security reasons, you can't turn on public access while creating an MSK cluster. However, you can update an existing cluster to make it publicly accessible. You can also create a new cluster and then update it to make it publicly accessible (documentation).storage_info
- (Optional) A block that contains information about storage volumes attached to MSK broker nodes. See below.public_access
- (Optional) Access control settings for brokers. See below.vpc_connectivity
- (Optional) VPC connectivity access control for brokers. See below.type
- (Optional) Public access type. Valid values: DISABLED
, SERVICE_PROVIDED_EIPS
.client_authentication
- (Optional) Includes all client authentication information for VPC connectivity. See below.sasl
- (Optional) SASL authentication type details for VPC connectivity. See below.tls
- (Optional) Enables TLS authentication for VPC connectivity.iam
- (Optional) Enables SASL/IAM authentication for VPC connectivity.scram
- (Optional) Enables SASL/SCRAM authentication for VPC connectivity.ebs_storage_info
- (Optional) A block that contains EBS volume information. See below.provisioned_throughput
- (Optional) A block that contains EBS volume provisioned throughput information. To provision storage throughput, you must choose broker type kafka.m5.4xlarge or larger. See below.volume_size
- (Optional) The size in GiB of the EBS volume for the data drive on each broker node. Minimum value of 1
and maximum value of 16384
.enabled
- (Optional) Controls whether provisioned throughput is enabled or not. Default value: false
.volume_throughput
- (Optional) Throughput value of the EBS volumes for the data drive on each kafka broker node in MiB per second. The minimum value is 250
. The maximum value varies between broker type. You can refer to the valid values for the maximum volume throughput at the following documentation on throughput bottleneckssasl
- (Optional) Configuration block for specifying SASL client authentication. See below.tls
- (Optional) Configuration block for specifying TLS client authentication. See below.unauthenticated
- (Optional) Enables unauthenticated access.iam
- (Optional) Enables IAM client authentication. Defaults to false
.scram
- (Optional) Enables SCRAM client authentication via AWS Secrets Manager. Defaults to false
.certificate_authority_arns
- (Optional) List of ACM Certificate Authority Amazon Resource Names (ARNs).arn
- (Required) Amazon Resource Name (ARN) of the MSK Configuration to use in the cluster.revision
- (Required) Revision of the MSK Configuration to use in the cluster.encryption_in_transit
- (Optional) Configuration block to specify encryption in transit. See below.encryption_at_rest_kms_key_arn
- (Optional) You may specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If no key is specified, an AWS managed KMS ('aws/msk' managed service) key will be used for encrypting the data at rest.client_broker
- (Optional) Encryption setting for data in transit between clients and brokers. Valid values: TLS
, TLS_PLAINTEXT
, and PLAINTEXT
. Default value is TLS
.in_cluster
- (Optional) Whether data communication among broker nodes is encrypted. Default value: true
.prometheus
- (Required) Configuration block for Prometheus settings for open monitoring. See below.jmx_exporter
- (Optional) Configuration block for JMX Exporter. See below.node_exporter
- (Optional) Configuration block for Node Exporter. See below.enabled_in_broker
- (Required) Indicates whether you want to enable or disable the JMX Exporter.enabled_in_broker
- (Required) Indicates whether you want to enable or disable the Node Exporter.broker_logs
- (Required) Configuration block for Broker Logs settings for logging info. See below.enabled
- (Optional) Indicates whether you want to enable or disable streaming broker logs to Cloudwatch Logs.log_group
- (Optional) Name of the Cloudwatch Log Group to deliver logs to.enabled
- (Optional) Indicates whether you want to enable or disable streaming broker logs to Kinesis Data Firehose.delivery_stream
- (Optional) Name of the Kinesis Data Firehose delivery stream to deliver logs to.enabled
- (Optional) Indicates whether you want to enable or disable streaming broker logs to S3.bucket
- (Optional) Name of the S3 bucket to deliver logs to.prefix
- (Optional) Prefix to append to the folder name.This resource exports the following attributes in addition to the arguments above:
arn
- Amazon Resource Name (ARN) of the MSK cluster.bootstrap_brokers
- Comma separated list of one or more hostname:port pairs of kafka brokers suitable to bootstrap connectivity to the kafka cluster. Contains a value if encryption_info.0.encryption_in_transit.0.client_broker
is set to PLAINTEXT
or TLS_PLAINTEXT
. The resource sorts values alphabetically. AWS may not always return all endpoints so this value is not guaranteed to be stable across applies.bootstrap_brokers_public_sasl_iam
- One or more DNS names (or IP addresses) and SASL IAM port pairs. For example, b-1-public.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9198,b-2-public.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9198,b-3-public.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9198
. This attribute will have a value if encryption_info.0.encryption_in_transit.0.client_broker
is set to TLS_PLAINTEXT
or TLS
and client_authentication.0.sasl.0.iam
is set to true
and broker_node_group_info.0.connectivity_info.0.public_access.0.type
is set to SERVICE_PROVIDED_EIPS
and the cluster fulfill all other requirements for public access. The resource sorts the list alphabetically. AWS may not always return all endpoints so the values may not be stable across applies.bootstrap_brokers_public_sasl_scram
- One or more DNS names (or IP addresses) and SASL SCRAM port pairs. For example, b-1-public.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9196,b-2-public.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9196,b-3-public.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9196
. This attribute will have a value if encryption_info.0.encryption_in_transit.0.client_broker
is set to TLS_PLAINTEXT
or TLS
and client_authentication.0.sasl.0.scram
is set to true
and broker_node_group_info.0.connectivity_info.0.public_access.0.type
is set to SERVICE_PROVIDED_EIPS
and the cluster fulfill all other requirements for public access. The resource sorts the list alphabetically. AWS may not always return all endpoints so the values may not be stable across applies.bootstrap_brokers_public_tls
- One or more DNS names (or IP addresses) and TLS port pairs. For example, b-1-public.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9194,b-2-public.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9194,b-3-public.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9194
. This attribute will have a value if encryption_info.0.encryption_in_transit.0.client_broker
is set to TLS_PLAINTEXT
or TLS
and broker_node_group_info.0.connectivity_info.0.public_access.0.type
is set to SERVICE_PROVIDED_EIPS
and the cluster fulfill all other requirements for public access. The resource sorts the list alphabetically. AWS may not always return all endpoints so the values may not be stable across applies.bootstrap_brokers_sasl_iam
- One or more DNS names (or IP addresses) and SASL IAM port pairs. For example, b-1.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9098,b-2.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9098,b-3.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9098
. This attribute will have a value if encryption_info.0.encryption_in_transit.0.client_broker
is set to TLS_PLAINTEXT
or TLS
and client_authentication.0.sasl.0.iam
is set to true
. The resource sorts the list alphabetically. AWS may not always return all endpoints so the values may not be stable across applies.bootstrap_brokers_sasl_scram
- One or more DNS names (or IP addresses) and SASL SCRAM port pairs. For example, b-1.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9096,b-2.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9096,b-3.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9096
. This attribute will have a value if encryption_info.0.encryption_in_transit.0.client_broker
is set to TLS_PLAINTEXT
or TLS
and client_authentication.0.sasl.0.scram
is set to true
. The resource sorts the list alphabetically. AWS may not always return all endpoints so the values may not be stable across applies.bootstrap_brokers_tls
- One or more DNS names (or IP addresses) and TLS port pairs. For example, b-1.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9094,b-2.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9094,b-3.exampleClusterName.abcde.c2.kafka.us-east-1.amazonaws.com:9094
. This attribute will have a value if encryption_info.0.encryption_in_transit.0.client_broker
is set to TLS_PLAINTEXT
or TLS
. The resource sorts the list alphabetically. AWS may not always return all endpoints so the values may not be stable across applies.bootstrap_brokers_vpc_connectivity_sasl_iam
- A string containing one or more DNS names (or IP addresses) and SASL IAM port pairs for VPC connectivity. AWS may not always return all endpoints so the values may not be stable across applies.bootstrap_brokers_vpc_connectivity_sasl_scram
- A string containing one or more DNS names (or IP addresses) and SASL SCRAM port pairs for VPC connectivity. AWS may not always return all endpoints so the values may not be stable across applies.bootstrap_brokers_vpc_connectivity_tls
- A string containing one or more DNS names (or IP addresses) and TLS port pairs for VPC connectivity. AWS may not always return all endpoints so the values may not be stable across applies.cluster_uuid
- UUID of the MSK cluster, for use in IAM policies.current_version
- Current version of the MSK Cluster used for updates, e.g., K13V1IB3VIYZZH
encryption_info.0.encryption_at_rest_kms_key_arn
- The ARN of the KMS key used for encryption at rest of the broker data volumes.tags_all
- A map of tags assigned to the resource, including those inherited from the provider default_tags
configuration block.zookeeper_connect_string
- A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster. The returned values are sorted alphabetically. The AWS API may not return all endpoints, so this value is not guaranteed to be stable across applies.zookeeper_connect_string_tls
- A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster via TLS. The returned values are sorted alphabetically. The AWS API may not return all endpoints, so this value is not guaranteed to be stable across applies.create
- (Default 120m
)update
- (Default 120m
)
Note that the update
timeout is used separately for storage_info
, instance_type
, number_of_broker_nodes
, configuration_info
, kafka_version
and monitoring and logging update timeouts.delete
- (Default 120m
)In Terraform v1.5.0 and later, use an import
block to import MSK clusters using the cluster arn
. For example:
import {
to = aws_msk_cluster.example
id = "arn:aws:kafka:us-west-2:123456789012:cluster/example/279c0212-d057-4dba-9aa9-1c4e5a25bfc7-3"
}
Using terraform import
, import MSK clusters using the cluster arn
. For example:
% terraform import aws_msk_cluster.example arn:aws:kafka:us-west-2:123456789012:cluster/example/279c0212-d057-4dba-9aa9-1c4e5a25bfc7-3