Provides a Kinesis Analytics Application resource. Kinesis Analytics is a managed service that allows processing and analyzing streaming data using standard SQL.
For more details, see the Amazon Kinesis Analytics Documentation.
resource "aws_kinesis_stream" "test_stream" {
name = "terraform-kinesis-test"
shard_count = 1
}
resource "aws_kinesis_analytics_application" "test_application" {
name = "kinesis-analytics-application-test"
inputs {
name_prefix = "test_prefix"
kinesis_stream {
resource_arn = aws_kinesis_stream.test_stream.arn
role_arn = aws_iam_role.test.arn
}
parallelism {
count = 1
}
schema {
record_columns {
mapping = "$.test"
name = "test"
sql_type = "VARCHAR(8)"
}
record_encoding = "UTF-8"
record_format {
mapping_parameters {
json {
record_row_path = "$"
}
}
}
}
}
}
resource "aws_cloudwatch_log_group" "example" {
name = "analytics"
}
resource "aws_cloudwatch_log_stream" "example" {
name = "example-kinesis-application"
log_group_name = aws_cloudwatch_log_group.example.name
}
resource "aws_kinesis_stream" "example" {
name = "example-kinesis-stream"
shard_count = 1
}
resource "aws_kinesis_firehose_delivery_stream" "example" {
name = "example-kinesis-delivery-stream"
destination = "extended_s3"
extended_s3_configuration {
bucket_arn = aws_s3_bucket.example.arn
role_arn = aws_iam_role.example.arn
}
}
resource "aws_kinesis_analytics_application" "test" {
name = "example-application"
cloudwatch_logging_options {
log_stream_arn = aws_cloudwatch_log_stream.example.arn
role_arn = aws_iam_role.example.arn
}
inputs {
name_prefix = "example_prefix"
schema {
record_columns {
name = "COLUMN_1"
sql_type = "INTEGER"
}
record_format {
mapping_parameters {
csv {
record_column_delimiter = ","
record_row_delimiter = "|"
}
}
}
}
kinesis_stream {
resource_arn = aws_kinesis_stream.example.arn
role_arn = aws_iam_role.example.arn
}
starting_position_configuration {
starting_position = "NOW"
}
}
outputs {
name = "OUTPUT_1"
schema {
record_format_type = "CSV"
}
kinesis_firehose {
resource_arn = aws_kinesis_firehose_delivery_stream.example.arn
role_arn = aws_iam_role.example.arn
}
}
start_application = true
}
This resource supports the following arguments:
name
- (Required) Name of the Kinesis Analytics Application.code
- (Optional) SQL Code to transform input data, and generate output.description
- (Optional) Description of the application.cloudwatch_logging_options
- (Optional) The CloudWatch log stream options to monitor application errors.
See CloudWatch Logging Options below for more details.inputs
- (Optional) Input configuration of the application. See Inputs below for more details.outputs
- (Optional) Output destination configuration of the application. See Outputs below for more details.reference_data_sources
- (Optional) An S3 Reference Data Source for the application.
See Reference Data Sources below for more details.start_application
- (Optional) Whether to start or stop the Kinesis Analytics Application. To start an application, an input with a defined starting_position
must be configured.
To modify an application's starting position, first stop the application by setting start_application = false
, then update starting_position
and set start_application = true
.tags
- Key-value map of tags for the Kinesis Analytics Application. If configured with a provider default_tags
configuration block present, tags with matching keys will overwrite those defined at the provider-level.Configure a CloudWatch Log Stream to monitor application errors.
The cloudwatch_logging_options
block supports the following:
log_stream_arn
- (Required) The ARN of the CloudWatch Log Stream.role_arn
- (Required) The ARN of the IAM Role used to send application messages.Configure an Input for the Kinesis Analytics Application. You can only have 1 Input configured.
The inputs
block supports the following:
name_prefix
- (Required) The Name Prefix to use when creating an in-application stream.schema
- (Required) The Schema format of the data in the streaming source. See Source Schema below for more details.kinesis_firehose
- (Optional) The Kinesis Firehose configuration for the streaming source. Conflicts with kinesis_stream
.
See Kinesis Firehose below for more details.kinesis_stream
- (Optional) The Kinesis Stream configuration for the streaming source. Conflicts with kinesis_firehose
.
See Kinesis Stream below for more details.parallelism
- (Optional) The number of Parallel in-application streams to create.
See Parallelism below for more details.processing_configuration
- (Optional) The Processing Configuration to transform records as they are received from the stream.
See Processing Configuration below for more details.starting_position_configuration
(Optional) The point at which the application starts processing records from the streaming source.
See Starting Position Configuration below for more details.Configure Output destinations for the Kinesis Analytics Application. You can have a maximum of 3 destinations configured.
The outputs
block supports the following:
name
- (Required) The Name of the in-application stream.schema
- (Required) The Schema format of the data written to the destination. See Destination Schema below for more details.kinesis_firehose
- (Optional) The Kinesis Firehose configuration for the destination stream. Conflicts with kinesis_stream
.
See Kinesis Firehose below for more details.kinesis_stream
- (Optional) The Kinesis Stream configuration for the destination stream. Conflicts with kinesis_firehose
.
See Kinesis Stream below for more details.lambda
- (Optional) The Lambda function destination. See Lambda below for more details.Add a Reference Data Source to the Kinesis Analytics Application. You can only have 1 Reference Data Source.
The reference_data_sources
block supports the following:
schema
- (Required) The Schema format of the data in the streaming source. See Source Schema below for more details.table_name
- (Required) The in-application Table Name.s3
- (Optional) The S3 configuration for the reference data source. See S3 Reference below for more details.Configuration for a Kinesis Firehose delivery stream.
The kinesis_firehose
block supports the following:
resource_arn
- (Required) The ARN of the Kinesis Firehose delivery stream.role_arn
- (Required) The ARN of the IAM Role used to access the stream.Configuration for a Kinesis Stream.
The kinesis_stream
block supports the following:
resource_arn
- (Required) The ARN of the Kinesis Stream.role_arn
- (Required) The ARN of the IAM Role used to access the stream.The Schema format of the data in the destination.
The schema
block supports the following:
record_format_type
- (Required) The Format Type of the records on the output stream. Can be CSV
or JSON
.The Schema format of the data in the streaming source.
The schema
block supports the following:
record_columns
- (Required) The Record Column mapping for the streaming source data element.
See Record Columns below for more details.record_format
- (Required) The Record Format and mapping information to schematize a record.
See Record Format below for more details.record_encoding
- (Optional) The Encoding of the record in the streaming source.Configures the number of Parallel in-application streams to create.
The parallelism
block supports the following:
count
- (Required) The Count of streams.The Processing Configuration to transform records as they are received from the stream.
The processing_configuration
block supports the following:
lambda
- (Required) The Lambda function configuration. See Lambda below for more details.The Lambda function that pre-processes records in the stream.
The lambda
block supports the following:
resource_arn
- (Required) The ARN of the Lambda function.role_arn
- (Required) The ARN of the IAM Role used to access the Lambda function.The point at which the application reads from the streaming source.
The starting_position_configuration
block supports the following:
starting_position
- (Required) The starting position on the stream. Valid values: LAST_STOPPED_POINT
, NOW
, TRIM_HORIZON
.The Column mapping of each data element in the streaming source to the corresponding column in the in-application stream.
The record_columns
block supports the following:
name
- (Required) Name of the column.sql_type
- (Required) The SQL Type of the column.mapping
- (Optional) The Mapping reference to the data element.The Record Format and relevant mapping information that should be applied to schematize the records on the stream.
The record_format
block supports the following:
record_format_type
- (Required) The type of Record Format. Can be CSV
or JSON
.mapping_parameters
- (Optional) The Mapping Information for the record format.
See Mapping Parameters below for more details.Provides Mapping information specific to the record format on the streaming source.
The mapping_parameters
block supports the following:
csv
- (Optional) Mapping information when the record format uses delimiters.
See CSV Mapping Parameters below for more details.json
- (Optional) Mapping information when JSON is the record format on the streaming source.
See JSON Mapping Parameters below for more details.Mapping information when the record format uses delimiters.
The csv
block supports the following:
record_column_delimiter
- (Required) The Column Delimiter.record_row_delimiter
- (Required) The Row Delimiter.Mapping information when JSON is the record format on the streaming source.
The json
block supports the following:
record_row_path
- (Required) Path to the top-level parent that contains the records.Identifies the S3 bucket and object that contains the reference data.
The s3
blcok supports the following:
bucket_arn
- (Required) The S3 Bucket ARN.file_key
- (Required) The File Key name containing reference data.role_arn
- (Required) The IAM Role ARN to read the data.This resource exports the following attributes in addition to the arguments above:
id
- The ARN of the Kinesis Analytics Application.arn
- The ARN of the Kinesis Analytics Appliation.create_timestamp
- The Timestamp when the application version was created.last_update_timestamp
- The Timestamp when the application was last updated.status
- The Status of the application.version
- The Version of the application.tags_all
- A map of tags assigned to the resource, including those inherited from the provider default_tags
configuration block.In Terraform v1.5.0 and later, use an import
block to import Kinesis Analytics Application using ARN. For example:
import {
to = aws_kinesis_analytics_application.example
id = "arn:aws:kinesisanalytics:us-west-2:1234567890:application/example"
}
Using terraform import
, import Kinesis Analytics Application using ARN. For example:
% terraform import aws_kinesis_analytics_application.example arn:aws:kinesisanalytics:us-west-2:1234567890:application/example