Provides a Datadog Logs Pipeline API resource, which is used to create and manage Datadog logs custom pipelines. Each datadog_logs_custom_pipeline
resource defines a complete pipeline. The order of the pipelines is maintained in a different resource: datadog_logs_pipeline_order
. When creating a new pipeline, you need to explicitly add this pipeline to the datadog_logs_pipeline_order
resource to track the pipeline. Similarly, when a pipeline needs to be destroyed, remove its references from the datadog_logs_pipeline_order
resource.
resource "datadog_logs_custom_pipeline" "sample_pipeline" {
filter {
query = "source:foo"
}
name = "sample pipeline"
is_enabled = true
processor {
arithmetic_processor {
expression = "(time1 - time2)*1000"
target = "my_arithmetic"
is_replace_missing = true
name = "sample arithmetic processor"
is_enabled = true
}
}
processor {
attribute_remapper {
sources = ["db.instance"]
source_type = "tag"
target = "db"
target_type = "attribute"
target_format = "string"
preserve_source = true
override_on_conflict = false
name = "sample attribute processor"
is_enabled = true
}
}
processor {
category_processor {
target = "foo.severity"
category {
name = "debug"
filter {
query = "@severity: \".\""
}
}
category {
name = "verbose"
filter {
query = "@severity: \"-\""
}
}
name = "sample category processor"
is_enabled = true
}
}
processor {
date_remapper {
sources = ["_timestamp", "published_date"]
name = "sample date remapper"
is_enabled = true
}
}
processor {
geo_ip_parser {
sources = ["network.client.ip"]
target = "network.client.geoip"
name = "sample geo ip parser"
is_enabled = true
}
}
processor {
grok_parser {
samples = ["sample log 1"]
source = "message"
grok {
support_rules = ""
match_rules = "Rule %%{word:my_word2} %%{number:my_float2}"
}
name = "sample grok parser"
is_enabled = true
}
}
processor {
lookup_processor {
source = "service_id"
target = "service_name"
lookup_table = ["1,my service"]
default_lookup = "unknown service"
name = "sample lookup processor"
is_enabled = true
}
}
processor {
message_remapper {
sources = ["msg"]
name = "sample message remapper"
is_enabled = true
}
}
processor {
pipeline {
filter {
query = "source:foo"
}
processor {
url_parser {
name = "sample url parser"
sources = ["url", "extra"]
target = "http_url"
normalize_ending_slashes = true
}
}
name = "nested pipeline"
is_enabled = true
}
}
processor {
service_remapper {
sources = ["service"]
name = "sample service remapper"
is_enabled = true
}
}
processor {
status_remapper {
sources = ["info", "trace"]
name = "sample status remapper"
is_enabled = true
}
}
processor {
string_builder_processor {
target = "user_activity"
template = "%%{user.name} logged in at %%{timestamp}"
name = "sample string builder processor"
is_enabled = true
is_replace_missing = false
}
}
processor {
trace_id_remapper {
sources = ["dd.trace_id"]
name = "sample trace id remapper"
is_enabled = true
}
}
processor {
user_agent_parser {
sources = ["user", "agent"]
target = "http_agent"
is_encoded = false
name = "sample user agent parser"
is_enabled = true
}
}
}
filter
(Block List, Min: 1) (see below for nested schema)name
(String)is_enabled
(Boolean)processor
(Block List) (see below for nested schema)id
(String) The ID of this resource.filter
Required:
query
(String) Filter criteria of the category.processor
Optional:
arithmetic_processor
(Block List, Max: 1) Arithmetic Processor. More information can be found in the official docs (see below for nested schema)attribute_remapper
(Block List, Max: 1) Attribute Remapper Processor. More information can be found in the official docs (see below for nested schema)category_processor
(Block List, Max: 1) Category Processor. More information can be found in the official docs (see below for nested schema)date_remapper
(Block List, Max: 1) Date Remapper Processor. More information can be found in the official docs (see below for nested schema)geo_ip_parser
(Block List, Max: 1) Date GeoIP Processor. More information can be found in the official docs (see below for nested schema)grok_parser
(Block List, Max: 1) Grok Processor. More information can be found in the official docs (see below for nested schema)lookup_processor
(Block List, Max: 1) Lookup Processor. More information can be found in the official docs (see below for nested schema)message_remapper
(Block List, Max: 1) Message Remapper Processor. More information can be found in the official docs (see below for nested schema)pipeline
(Block List, Max: 1) (see below for nested schema)reference_table_lookup_processor
(Block List, Max: 1) Reference Table Lookup Processor. Reference Tables are in public beta. More information can be found in the official docs (see below for nested schema)service_remapper
(Block List, Max: 1) Service Remapper Processor. More information can be found in the official docs (see below for nested schema)status_remapper
(Block List, Max: 1) Status Remapper Processor. More information can be found in the official docs (see below for nested schema)string_builder_processor
(Block List, Max: 1) String Builder Processor. More information can be found in the official docs (see below for nested schema)trace_id_remapper
(Block List, Max: 1) Trace ID Remapper Processor. More information can be found in the official docs (see below for nested schema)url_parser
(Block List, Max: 1) URL Parser Processor. More information can be found in the official docs (see below for nested schema)user_agent_parser
(Block List, Max: 1) User-Agent Parser Processor. More information can be found in the official docs (see below for nested schema)processor.arithmetic_processor
Required:
expression
(String) Arithmetic operation between one or more log attributes.target
(String) Name of the attribute that contains the result of the arithmetic operation.Optional:
is_enabled
(Boolean) Boolean value to enable your pipeline.is_replace_missing
(Boolean) If true, it replaces all missing attributes of expression by 0, false skips the operation if an attribute is missing.name
(String) Your pipeline name.processor.attribute_remapper
Required:
source_type
(String) Defines where the sources are from (log attribute
or tag
).sources
(List of String) List of source attributes or tags.target
(String) Final attribute or tag name to remap the sources.target_type
(String) Defines if the target is a log attribute
or tag
.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processoroverride_on_conflict
(Boolean) Override the target element if already set.preserve_source
(Boolean) Remove or preserve the remapped source element.target_format
(String) If the target_type
of the remapper is attribute
, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string
, integer
, or double
are the possible types. If the target_type
is tag
, this parameter may not be specified.processor.category_processor
Required:
category
(Block List, Min: 1) List of filters to match or exclude a log with their corresponding name to assign a custom value to the log. (see below for nested schema)target
(String) Name of the target attribute whose value is defined by the matching category.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the categoryprocessor.category_processor.category
Required:
filter
(Block List, Min: 1, Max: 1) (see below for nested schema)name
(String)processor.category_processor.category.filter
Required:
query
(String) Filter criteria of the category.processor.date_remapper
Required:
sources
(List of String) List of source attributes.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.geo_ip_parser
Required:
sources
(List of String) List of source attributes.target
(String) Name of the parent attribute that contains all the extracted details from the sources.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.grok_parser
Required:
grok
(Block List, Min: 1, Max: 1) (see below for nested schema)source
(String) Name of the log attribute to parse.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processorsamples
(List of String) List of sample logs for this parser. It can save up to 5 samples. Each sample takes up to 5000 characters.processor.grok_parser.grok
Required:
match_rules
(String) Match rules for your grok parser.support_rules
(String) Support rules for your grok parser.processor.lookup_processor
Required:
lookup_table
(List of String) List of entries of the lookup table using key,value
format.source
(String) Name of the source attribute used to do the lookup.target
(String) Name of the attribute that contains the result of the lookup.Optional:
default_lookup
(String) Default lookup value to use if there is no entry in the lookup table for the value of the source attribute.is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processorprocessor.message_remapper
Required:
sources
(List of String) List of source attributes.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.pipeline
Required:
filter
(Block List, Min: 1) (see below for nested schema)name
(String)Optional:
is_enabled
(Boolean)processor
(Block List) (see below for nested schema)processor.pipeline.filter
Required:
query
(String) Filter criteria of the category.processor.pipeline.processor
Optional:
arithmetic_processor
(Block List, Max: 1) Arithmetic Processor. More information can be found in the official docs (see below for nested schema)attribute_remapper
(Block List, Max: 1) Attribute Remapper Processor. More information can be found in the official docs (see below for nested schema)category_processor
(Block List, Max: 1) Category Processor. More information can be found in the official docs (see below for nested schema)date_remapper
(Block List, Max: 1) Date Remapper Processor. More information can be found in the official docs (see below for nested schema)geo_ip_parser
(Block List, Max: 1) Date GeoIP Processor. More information can be found in the official docs (see below for nested schema)grok_parser
(Block List, Max: 1) Grok Processor. More information can be found in the official docs (see below for nested schema)lookup_processor
(Block List, Max: 1) Lookup Processor. More information can be found in the official docs (see below for nested schema)message_remapper
(Block List, Max: 1) Message Remapper Processor. More information can be found in the official docs (see below for nested schema)reference_table_lookup_processor
(Block List, Max: 1) Reference Table Lookup Processor. Reference Tables are in public beta. More information can be found in the official docs (see below for nested schema)service_remapper
(Block List, Max: 1) Service Remapper Processor. More information can be found in the official docs (see below for nested schema)status_remapper
(Block List, Max: 1) Status Remapper Processor. More information can be found in the official docs (see below for nested schema)string_builder_processor
(Block List, Max: 1) String Builder Processor. More information can be found in the official docs (see below for nested schema)trace_id_remapper
(Block List, Max: 1) Trace ID Remapper Processor. More information can be found in the official docs (see below for nested schema)url_parser
(Block List, Max: 1) URL Parser Processor. More information can be found in the official docs (see below for nested schema)user_agent_parser
(Block List, Max: 1) User-Agent Parser Processor. More information can be found in the official docs (see below for nested schema)processor.pipeline.processor.arithmetic_processor
Required:
expression
(String) Arithmetic operation between one or more log attributes.target
(String) Name of the attribute that contains the result of the arithmetic operation.Optional:
is_enabled
(Boolean) Boolean value to enable your pipeline.is_replace_missing
(Boolean) If true, it replaces all missing attributes of expression by 0, false skips the operation if an attribute is missing.name
(String) Your pipeline name.processor.pipeline.processor.attribute_remapper
Required:
source_type
(String) Defines where the sources are from (log attribute
or tag
).sources
(List of String) List of source attributes or tags.target
(String) Final attribute or tag name to remap the sources.target_type
(String) Defines if the target is a log attribute
or tag
.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processoroverride_on_conflict
(Boolean) Override the target element if already set.preserve_source
(Boolean) Remove or preserve the remapped source element.target_format
(String) If the target_type
of the remapper is attribute
, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string
, integer
, or double
are the possible types. If the target_type
is tag
, this parameter may not be specified.processor.pipeline.processor.category_processor
Required:
category
(Block List, Min: 1) List of filters to match or exclude a log with their corresponding name to assign a custom value to the log. (see below for nested schema)target
(String) Name of the target attribute whose value is defined by the matching category.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the categoryprocessor.pipeline.processor.category_processor.category
Required:
filter
(Block List, Min: 1, Max: 1) (see below for nested schema)name
(String)processor.pipeline.processor.category_processor.category.filter
Required:
query
(String) Filter criteria of the category.processor.pipeline.processor.date_remapper
Required:
sources
(List of String) List of source attributes.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.pipeline.processor.geo_ip_parser
Required:
sources
(List of String) List of source attributes.target
(String) Name of the parent attribute that contains all the extracted details from the sources.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.pipeline.processor.grok_parser
Required:
grok
(Block List, Min: 1, Max: 1) (see below for nested schema)source
(String) Name of the log attribute to parse.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processorsamples
(List of String) List of sample logs for this parser. It can save up to 5 samples. Each sample takes up to 5000 characters.processor.pipeline.processor.grok_parser.grok
Required:
match_rules
(String) Match rules for your grok parser.support_rules
(String) Support rules for your grok parser.processor.pipeline.processor.lookup_processor
Required:
lookup_table
(List of String) List of entries of the lookup table using key,value
format.source
(String) Name of the source attribute used to do the lookup.target
(String) Name of the attribute that contains the result of the lookup.Optional:
default_lookup
(String) Default lookup value to use if there is no entry in the lookup table for the value of the source attribute.is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processorprocessor.pipeline.processor.message_remapper
Required:
sources
(List of String) List of source attributes.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.pipeline.processor.reference_table_lookup_processor
Required:
lookup_enrichment_table
(String) Name of the Reference Table for the source attribute and their associated target attribute values.source
(String) Name of the source attribute used to do the lookup.target
(String) Name of the attribute that contains the result of the lookup.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processorprocessor.pipeline.processor.service_remapper
Required:
sources
(List of String) List of source attributes.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.pipeline.processor.status_remapper
Required:
sources
(List of String) List of source attributes.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.pipeline.processor.string_builder_processor
Required:
target
(String) The name of the attribute that contains the result of the template.template
(String) The formula with one or more attributes and raw text.Optional:
is_enabled
(Boolean) If the processor is enabled or not.is_replace_missing
(Boolean) If it replaces all missing attributes of template by an empty string.name
(String) The name of the processor.processor.pipeline.processor.trace_id_remapper
Required:
sources
(List of String) List of source attributes.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.pipeline.processor.url_parser
Required:
sources
(List of String) List of source attributes.target
(String) Name of the parent attribute that contains all the extracted details from the sources.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processornormalize_ending_slashes
(Boolean) Normalize the ending slashes or not.processor.pipeline.processor.user_agent_parser
Required:
sources
(List of String) List of source attributes.target
(String) Name of the parent attribute that contains all the extracted details from the sources.Optional:
is_enabled
(Boolean) If the processor is enabled or not.is_encoded
(Boolean) If the source attribute is URL encoded or not.name
(String) Name of the processorprocessor.reference_table_lookup_processor
Required:
lookup_enrichment_table
(String) Name of the Reference Table for the source attribute and their associated target attribute values.source
(String) Name of the source attribute used to do the lookup.target
(String) Name of the attribute that contains the result of the lookup.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processorprocessor.service_remapper
Required:
sources
(List of String) List of source attributes.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.status_remapper
Required:
sources
(List of String) List of source attributes.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.string_builder_processor
Required:
target
(String) The name of the attribute that contains the result of the template.template
(String) The formula with one or more attributes and raw text.Optional:
is_enabled
(Boolean) If the processor is enabled or not.is_replace_missing
(Boolean) If it replaces all missing attributes of template by an empty string.name
(String) The name of the processor.processor.trace_id_remapper
Required:
sources
(List of String) List of source attributes.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processor.processor.url_parser
Required:
sources
(List of String) List of source attributes.target
(String) Name of the parent attribute that contains all the extracted details from the sources.Optional:
is_enabled
(Boolean) If the processor is enabled or not.name
(String) Name of the processornormalize_ending_slashes
(Boolean) Normalize the ending slashes or not.processor.user_agent_parser
Required:
sources
(List of String) List of source attributes.target
(String) Name of the parent attribute that contains all the extracted details from the sources.Optional:
is_enabled
(Boolean) If the processor is enabled or not.is_encoded
(Boolean) If the source attribute is URL encoded or not.name
(String) Name of the processorImport is supported using the following syntax:
# To find the pipeline ID, click the "edit" button in the UI to open the pipeline details.
# The pipeline ID is the last part of the URL.
terraform import <resource.name> <pipelineID>