pipelines

Creates, updates, deletes, gets or lists a pipelines resource.

Overview

Name	pipelines
Type	Resource
Id	datadog.logs.pipelines

Fields

The following fields are returned by SELECT queries:

get_logs_pipeline

Name	Datatype	Description
id	string	ID of the pipeline.
name	string	Name of the pipeline. (example: )
description	string	A description of the pipeline.
filter	object	Filter for logs.
is_enabled	boolean	Whether or not the pipeline is enabled.
is_read_only	boolean	Whether or not the pipeline can be edited.
processors	array	Ordered list of processors in this pipeline.
tags	array	A list of tags associated with the pipeline.
type	string	Type of pipeline. (example: pipeline)

list_logs_pipelines

Name	Datatype	Description
id	string	ID of the pipeline.
name	string	Name of the pipeline. (example: )
description	string	A description of the pipeline.
filter	object	Filter for logs.
is_enabled	boolean	Whether or not the pipeline is enabled.
is_read_only	boolean	Whether or not the pipeline can be edited.
processors	array	Ordered list of processors in this pipeline.
tags	array	A list of tags associated with the pipeline.
type	string	Type of pipeline. (example: pipeline)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get_logs_pipeline	select	pipeline_id		Get a specific pipeline from your organization.<br />This endpoint takes no JSON arguments.
list_logs_pipelines	select			Get all pipelines from your organization.<br />This endpoint takes no JSON arguments.
create_logs_pipeline	insert	name		Create a pipeline in your organization.
update_logs_pipeline	replace	pipeline_id, name		Update a given pipeline configuration to change it’s processors or their order.<br /><br />Note: Using this method updates your pipeline configuration by replacing<br />your current configuration with the new one sent to your Datadog organization.
delete_logs_pipeline	delete	pipeline_id		Delete a given pipeline from your organization.<br />This endpoint takes no JSON arguments.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
pipeline_id	string	ID of the pipeline to delete.
site	string	The Datadog site (region) for your organization, for example datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, ap1.datadoghq.com, ap2.datadoghq.com, datadoghq.eu, ddog-gov.com. Resolved from the DD_SITE environment variable when set. Optional: defaults to datadoghq.com, or the value of the DD_SITE environment variable when set; a WHERE value overrides both.

SELECT examples

get_logs_pipeline

Get a specific pipeline from your organization.<br />This endpoint takes no JSON arguments.

SELECT
id,
name,
description,
filter,
is_enabled,
is_read_only,
processors,
tags,
type
FROM datadog.logs.pipelines
WHERE pipeline_id = '{{ pipeline_id }}' -- required
;

list_logs_pipelines

Get all pipelines from your organization.<br />This endpoint takes no JSON arguments.

SELECT
id,
name,
description,
filter,
is_enabled,
is_read_only,
processors,
tags,
type
FROM datadog.logs.pipelines
;

INSERT examples

create_logs_pipeline

Create a pipeline in your organization.

INSERT INTO datadog.logs.pipelines (
description,
filter,
is_enabled,
name,
processors,
tags
)
SELECT 
'{{ description }}',
'{{ filter }}',
{{ is_enabled }},
'{{ name }}' /* required */,
'{{ processors }}',
'{{ tags }}'
RETURNING
id,
name,
description,
filter,
is_enabled,
is_read_only,
processors,
tags,
type
;

Manifest

# Description fields are for documentation purposes
- name: pipelines
props:
  - name: description
    value: "{{ description }}"
    description: |
      A description of the pipeline.
  - name: filter
    description: |
      Filter for logs.
    value:
      query: "{{ query }}"
  - name: is_enabled
    value: {{ is_enabled }}
    description: |
      Whether or not the pipeline is enabled.
  - name: name
    value: "{{ name }}"
    description: |
      Name of the pipeline.
  - name: processors
    description: |
      Ordered list of processors in this pipeline.
    value:
      - grok:
          match_rules: "{{ match_rules }}"
          support_rules: "{{ support_rules }}"
        is_enabled: {{ is_enabled }}
        name: "{{ name }}"
        samples: "{{ samples }}"
        source: "{{ source }}"
        type: "{{ type }}"
        sources: "{{ sources }}"
        override_on_conflict: {{ override_on_conflict }}
        preserve_source: {{ preserve_source }}
        source_type: "{{ source_type }}"
        target: "{{ target }}"
        target_format: "{{ target_format }}"
        target_type: "{{ target_type }}"
        normalize_ending_slashes: {{ normalize_ending_slashes }}
        is_encoded: {{ is_encoded }}
        categories: "{{ categories }}"
        expression: "{{ expression }}"
        is_replace_missing: {{ is_replace_missing }}
        template: "{{ template }}"
        description: "{{ description }}"
        filter:
          query: "{{ query }}"
        processors: "{{ processors }}"
        tags: "{{ tags }}"
        default_lookup: "{{ default_lookup }}"
        lookup_table: "{{ lookup_table }}"
        lookup_enrichment_table: "{{ lookup_enrichment_table }}"
        operation:
          preserve_source: {{ preserve_source }}
          source: "{{ source }}"
          target: "{{ target }}"
          type: "{{ type }}"
          filter: "{{ filter }}"
          value_to_extract: "{{ value_to_extract }}"
          key_to_extract: "{{ key_to_extract }}"
          override_on_conflict: {{ override_on_conflict }}
        binary_to_text_encoding: "{{ binary_to_text_encoding }}"
        input_representation: "{{ input_representation }}"
        mappers: "{{ mappers }}"
        schema:
          class_name: "{{ class_name }}"
          class_uid: {{ class_uid }}
          profiles:
            - "{{ profiles }}"
          schema_type: "{{ schema_type }}"
          version: "{{ version }}"
        attribute_to_exclude: "{{ attribute_to_exclude }}"
  - name: tags
    value:
      - "{{ tags }}"
    description: |
      A list of tags associated with the pipeline.

REPLACE examples

update_logs_pipeline

Update a given pipeline configuration to change it’s processors or their order.<br /><br />Note: Using this method updates your pipeline configuration by replacing<br />your current configuration with the new one sent to your Datadog organization.

REPLACE datadog.logs.pipelines
SET 
description = '{{ description }}',
filter = '{{ filter }}',
is_enabled = {{ is_enabled }},
name = '{{ name }}',
processors = '{{ processors }}',
tags = '{{ tags }}'
WHERE 
pipeline_id = '{{ pipeline_id }}' --required
AND name = '{{ name }}' --required
RETURNING
id,
name,
description,
filter,
is_enabled,
is_read_only,
processors,
tags,
type;

DELETE examples

delete_logs_pipeline

Delete a given pipeline from your organization.<br />This endpoint takes no JSON arguments.

DELETE FROM datadog.logs.pipelines
WHERE pipeline_id = '{{ pipeline_id }}' --required
;