Skip to content

Adding custom mapper support to Observability Pipelines OCSF Mapper #3175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
api-clients-generation-pipeline merged 1 commit into from
Feb 18, 2026
Merged

Adding custom mapper support to Observability Pipelines OCSF Mapper #3175

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions .generator/schemas/v2/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41365,6 +41365,7 @@ components:
example: CloudTrail Account Change
oneOf:
- $ref: '#/components/schemas/ObservabilityPipelineOcsfMappingLibrary'
- $ref: '#/components/schemas/ObservabilityPipelineOcsfMappingCustom'
ObservabilityPipelineOcsfMapperProcessorType:
default: ocsf_mapper
description: The processor type. The value should always be `ocsf_mapper`.
Expand All @@ -41374,6 +41375,116 @@ components:
type: string
x-enum-varnames:
- OCSF_MAPPER
ObservabilityPipelineOcsfMappingCustom:
description: Custom OCSF mapping configuration for transforming logs.
properties:
mapping:
description: A list of field mapping rules for transforming log fields to
OCSF schema fields.
items:
$ref: '#/components/schemas/ObservabilityPipelineOcsfMappingCustomFieldMapping'
type: array
metadata:
$ref: '#/components/schemas/ObservabilityPipelineOcsfMappingCustomMetadata'
version:
description: The version of the custom mapping configuration.
example: 1
format: int64
type: integer
required:
- mapping
- metadata
- version
type: object
ObservabilityPipelineOcsfMappingCustomFieldMapping:
description: Defines a single field mapping rule for transforming a source field
to an OCSF destination field.
properties:
default:
description: The default value to use if the source field is missing or
empty.
example: ''
dest:
description: The destination OCSF field path.
example: device.type
type: string
lookup:
$ref: '#/components/schemas/ObservabilityPipelineOcsfMappingCustomLookup'
source:
description: The source field path from the log event.
example: host.type
sources:
description: Multiple source field paths for combined mapping.
example:
- field1
- field2
value:
description: A static value to use for the destination field.
example: static_value
required:
- dest
type: object
ObservabilityPipelineOcsfMappingCustomLookup:
description: Lookup table configuration for mapping source values to destination
values.
properties:
default:
description: The default value to use if no lookup match is found.
example: unknown
table:
description: A list of lookup table entries for value transformation.
items:
$ref: '#/components/schemas/ObservabilityPipelineOcsfMappingCustomLookupTableEntry'
type: array
type: object
ObservabilityPipelineOcsfMappingCustomLookupTableEntry:
description: A single entry in a lookup table for value transformation.
properties:
contains:
description: The substring to match in the source value.
example: Desktop
type: string
equals:
description: The exact value to match in the source.
example: desktop
equals_source:
description: The source field to match against.
example: device_type
type: string
matches:
description: A regex pattern to match in the source value.
example: ^Desktop.*
type: string
not_matches:
description: A regex pattern that must not match the source value.
example: ^Mobile.*
type: string
value:
description: The value to use when a match is found.
example: desktop
type: object
ObservabilityPipelineOcsfMappingCustomMetadata:
description: Metadata for the custom OCSF mapping.
properties:
class:
description: The OCSF event class name.
example: Device Inventory Info
type: string
profiles:
description: A list of OCSF profiles to apply.
example:
- container
items:
type: string
type: array
version:
description: The OCSF schema version.
example: 1.3.0
type: string
required:
- class
- version
type: object
ObservabilityPipelineOcsfMappingLibrary:
description: Predefined library mappings for common log formats.
enum:
Expand Down
35 changes: 35 additions & 0 deletions docs/datadog_api_client.v2.model.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17861,6 +17861,41 @@ datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapper\_processor\_
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_custom module
-----------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_custom\_field\_mapping module
---------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_field_mapping
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_custom\_lookup module
-------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_lookup
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_custom\_lookup\_table\_entry module
---------------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_lookup_table_entry
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_custom\_metadata module
---------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_metadata
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_ocsf\_mapping\_library module
------------------------------------------------------------------------------------

Expand Down
140 changes: 140 additions & 0 deletions examples/v2/observability-pipelines/ValidatePipeline_3024756866.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
"""
Validate an observability pipeline with OCSF mapper custom mapping returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi
from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig
from datadog_api_client.v2.model.observability_pipeline_config_processor_group import (
ObservabilityPipelineConfigProcessorGroup,
)
from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import (
ObservabilityPipelineDatadogAgentSource,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import (
ObservabilityPipelineDatadogAgentSourceType,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
ObservabilityPipelineDatadogLogsDestination,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import (
ObservabilityPipelineDatadogLogsDestinationType,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor import (
ObservabilityPipelineOcsfMapperProcessor,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor_mapping import (
ObservabilityPipelineOcsfMapperProcessorMapping,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor_type import (
ObservabilityPipelineOcsfMapperProcessorType,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom import (
ObservabilityPipelineOcsfMappingCustom,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_field_mapping import (
ObservabilityPipelineOcsfMappingCustomFieldMapping,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_lookup import (
ObservabilityPipelineOcsfMappingCustomLookup,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_lookup_table_entry import (
ObservabilityPipelineOcsfMappingCustomLookupTableEntry,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapping_custom_metadata import (
ObservabilityPipelineOcsfMappingCustomMetadata,
)
from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec
from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData

body = ObservabilityPipelineSpec(
data=ObservabilityPipelineSpecData(
attributes=ObservabilityPipelineDataAttributes(
config=ObservabilityPipelineConfig(
destinations=[
ObservabilityPipelineDatadogLogsDestination(
id="datadog-logs-destination",
inputs=[
"my-processor-group",
],
type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS,
),
],
processor_groups=[
ObservabilityPipelineConfigProcessorGroup(
enabled=True,
id="my-processor-group",
include="service:my-service",
inputs=[
"datadog-agent-source",
],
processors=[
ObservabilityPipelineOcsfMapperProcessor(
enabled=True,
id="ocsf-mapper-processor",
include="service:my-service",
mappings=[
ObservabilityPipelineOcsfMapperProcessorMapping(
include="source:custom",
mapping=ObservabilityPipelineOcsfMappingCustom(
mapping=[
ObservabilityPipelineOcsfMappingCustomFieldMapping(
default="",
dest="time",
source="timestamp",
),
ObservabilityPipelineOcsfMappingCustomFieldMapping(
default="",
dest="severity",
source="level",
),
ObservabilityPipelineOcsfMappingCustomFieldMapping(
default="",
dest="device.type",
lookup=ObservabilityPipelineOcsfMappingCustomLookup(
table=[
ObservabilityPipelineOcsfMappingCustomLookupTableEntry(
contains="Desktop",
value="desktop",
),
],
),
source="host.type",
),
],
metadata=ObservabilityPipelineOcsfMappingCustomMetadata(
_class="Device Inventory Info",
profiles=[
"container",
],
version="1.3.0",
),
version=1,
),
),
],
type=ObservabilityPipelineOcsfMapperProcessorType.OCSF_MAPPER,
),
],
),
],
sources=[
ObservabilityPipelineDatadogAgentSource(
id="datadog-agent-source",
type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT,
),
],
),
name="OCSF Custom Mapper Pipeline",
),
type="pipelines",
),
)

configuration = Configuration()
with ApiClient(configuration) as api_client:
api_instance = ObservabilityPipelinesApi(api_client)
response = api_instance.validate_pipeline(body=body)

print(response)
91 changes: 91 additions & 0 deletions examples/v2/observability-pipelines/ValidatePipeline_3565101276.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
"""
Validate an observability pipeline with OCSF mapper library mapping returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi
from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig
from datadog_api_client.v2.model.observability_pipeline_config_processor_group import (
ObservabilityPipelineConfigProcessorGroup,
)
from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import (
ObservabilityPipelineDatadogAgentSource,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import (
ObservabilityPipelineDatadogAgentSourceType,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
ObservabilityPipelineDatadogLogsDestination,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import (
ObservabilityPipelineDatadogLogsDestinationType,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor import (
ObservabilityPipelineOcsfMapperProcessor,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor_mapping import (
ObservabilityPipelineOcsfMapperProcessorMapping,
)
from datadog_api_client.v2.model.observability_pipeline_ocsf_mapper_processor_type import (
ObservabilityPipelineOcsfMapperProcessorType,
)
from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec
from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData

body = ObservabilityPipelineSpec(
data=ObservabilityPipelineSpecData(
attributes=ObservabilityPipelineDataAttributes(
config=ObservabilityPipelineConfig(
destinations=[
ObservabilityPipelineDatadogLogsDestination(
id="datadog-logs-destination",
inputs=[
"my-processor-group",
],
type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS,
),
],
processor_groups=[
ObservabilityPipelineConfigProcessorGroup(
enabled=True,
id="my-processor-group",
include="service:my-service",
inputs=[
"datadog-agent-source",
],
processors=[
ObservabilityPipelineOcsfMapperProcessor(
enabled=True,
id="ocsf-mapper-processor",
include="service:my-service",
type=ObservabilityPipelineOcsfMapperProcessorType.OCSF_MAPPER,
mappings=[
ObservabilityPipelineOcsfMapperProcessorMapping(
include="source:cloudtrail",
mapping="CloudTrail Account Change",
),
],
),
],
),
],
sources=[
ObservabilityPipelineDatadogAgentSource(
id="datadog-agent-source",
type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT,
),
],
),
name="OCSF Mapper Pipeline",
),
type="pipelines",
),
)

configuration = Configuration()
with ApiClient(configuration) as api_client:
api_instance = ObservabilityPipelinesApi(api_client)
response = api_instance.validate_pipeline(body=body)

print(response)
Loading