[DOCS-13523] Add anomaly detection guide for Data Observability monitors by OliviaShoup · Pull Request #39969 · DataDog/documentation

OliviaShoup · 2026-09-15T18:27:27Z

What does this PR do? What is the motivation?

Fixes DOCS-13523

Adds a new guide explaining how the anomaly detection model works for Data Observability monitors. It covers:

the training period
model states
per-metric-type alert conditions
seasonality
trends
annotations

Merge readiness

Ready for merge

For Datadog employees:

AI assistance

Claude Code helped port and adapt the content from the original closed PR (#34877), including applying open editorial feedback from that PR's review.

Additional notes

This PR revives content from a previously closed PR (#34877). Follow-up after a run of the docs-status-sweep skill that found #34877 had been autoclosed.

Committed With Claude

github-actions · 2026-09-15T18:31:12Z

Preview links (active after the `build_preview` check completes)

New or renamed files

DavidBraslow · 2026-09-15T20:06:31Z

+
+When you create an anomaly detection monitor, it enters a training period. During training, the monitor collects historical values to learn the metric's baseline behavior. It does not trigger alerts during this period, and the monitor chart appears in blue.
+
+Training typically takes between 3 and 7 days, depending on how frequently the underlying metric updates. Because many data pipelines behave differently on weekends, the model needs to observe both weekday and weekend behavior.


Suggested change

Training typically takes between 3 and 7 days, depending on how frequently the underlying metric updates. Because many data pipelines behave differently on weekends, the model needs to observe both weekday and weekend behavior.

Training typically takes between 3 and 9 days. Because many data pipelines behave differently on weekends, the model needs to observe both weekday and weekend behavior.

DavidBraslow · 2026-09-15T20:10:45Z

+
+Training typically takes between 3 and 7 days, depending on how frequently the underlying metric updates. Because many data pipelines behave differently on weekends, the model needs to observe both weekday and weekend behavior.
+
+After training completes, the monitor begins alerting. The chart uses color to indicate the current evaluation state:


Suggested change

After training completes, the monitor begins alerting. The chart uses color to indicate the current evaluation state:

After training completes, the monitor can alert. The chart shows the observed values over time, with bounds above and below the line as a shaded area showing expected values. The color shows the current state:

DavidBraslow · 2026-09-15T20:16:18Z

+
+## Model states
+
+The model often reuses prior predictions rather than generating new ones, such as when a value hasn't changed or when the monitor is in an alerting state. As a result, adding [annotations][2] is the primary way to get an alerting monitor back to a normal state and resume learning from new values.


Suggested change

The model often reuses prior predictions rather than generating new ones, such as when a value hasn't changed or when the monitor is in an alerting state. As a result, adding [annotations][2] is the primary way to get an alerting monitor back to a normal state and resume learning from new values.

When an anomaly is found, the model will remain in alert state for some time, until either the observed value returns to the original expected bounds or the anomaly has persisted long enough that the model considers it a new normal state. To resolve the monitor manually and return it to normal state, use [annotations][2].

DavidBraslow · 2026-09-15T20:16:31Z

+| Green | Normal | The observed value is within expected bounds. |
+| Red | Alerting | The observed value fell outside expected bounds. |
+
+## Model states


Suggested change

## Model states

## Alert state

DavidBraslow · 2026-09-15T20:18:19Z

+
+## Metric-specific behavior
+
+While the core model logic is shared across metric types, each type has additional rules:


Suggested change

While the core model logic is shared across metric types, each type has additional rules:

The model works differently for different metrics:

DavidBraslow · 2026-09-15T20:30:25Z

+| Day of month | Metrics with recurring patterns tied to the calendar month, such as end-of-month spikes. |
+| Weekday vs. weekend | Metrics with systematically different behavior on weekends versus weekdays. |
+
+Not all seasonal patterns are available for all metric types. Additionally, the model requires multiple complete cycles of normal ("green") history before it can detect a given pattern.


Suggested change

Not all seasonal patterns are available for all metric types. Additionally, the model requires multiple complete cycles of normal ("green") history before it can detect a given pattern.

Not all seasonal patterns are available for all metric types. Additionally, the model requires multiple complete cycles of normal history before it can detect a given pattern.

DavidBraslow · 2026-09-15T20:33:18Z

+
+## Annotations
+
+Annotations let you provide feedback to the model when it misclassifies a point, either by missing an alert or by generating a false alert. Because data quality expectations are often business-specific, annotations are the primary way to tune the model to your team's needs over time.


Suggested change

Annotations let you provide feedback to the model when it misclassifies a point, either by missing an alert or by generating a false alert. Because data quality expectations are often business-specific, annotations are the primary way to tune the model to your team's needs over time.

Annotations let you immediately retrain the model when it misclassifies a point, either by missing an alert or by generating a false alert. Because data quality expectations are often metric-specific, annotations are the primary way to tune the model to your team's needs.

DavidBraslow · 2026-09-15T20:35:01Z

+Annotations let you provide feedback to the model when it misclassifies a point, either by missing an alert or by generating a false alert. Because data quality expectations are often business-specific, annotations are the primary way to tune the model to your team's needs over time.
+
+Annotations have two effects:
+- **Correcting the current state**: Marking a flagged point as expected moves the monitor out of the alerting state and resumes normal learning.


Suggested change

- **Correcting the current state**: Marking a flagged point as expected moves the monitor out of the alerting state and resumes normal learning.

- **Correcting the current state**: Marking a flagged point as expected moves the monitor out of the alerting state to normal state at the next observation, so it can then alert on new anomalies.

DavidBraslow · 2026-09-15T20:35:56Z

+
+Annotations have two effects:
+- **Correcting the current state**: Marking a flagged point as expected moves the monitor out of the alerting state and resumes normal learning.
+- **Shaping future predictions**: The model weights annotated points when generating future bounds, so repeated feedback improves accuracy over time.


Suggested change

- **Shaping future predictions**: The model weights annotated points when generating future bounds, so repeated feedback improves accuracy over time.

- **Shaping future predictions**: The model uses annotated points to adjust future bounds, so feedback immediately improves accuracy.

DavidBraslow · 2026-09-15T20:36:36Z

+
+Row count monitors alert when a table's row count has flatlined, meaning it has not changed for longer than normal. A stalled row count may indicate a broken pipeline.
+
+### Percentage (nullness, uniqueness)


Suggested change

### Percentage (nullness, uniqueness)

### Percentage (e.g. nullness, uniqueness)

[DOCS-13523] Add anomaly detection guide for Data Observability monitors

4d80cef

Committed With Claude

OliviaShoup requested a review from a team as a code owner September 15, 2026 18:27

github-actions Bot added the Architecture Everything related to the Doc backend label Sep 15, 2026

OliviaShoup added the WORK IN PROGRESS No review needed, it's a wip ;) label Sep 15, 2026

DavidBraslow approved these changes Sep 15, 2026

View reviewed changes

Navigation Menu

Uh oh!

[DOCS-13523] Add anomaly detection guide for Data Observability monitors - #39969

[DOCS-13523] Add anomaly detection guide for Data Observability monitors#39969
OliviaShoup wants to merge 1 commit into
masterfrom
olivia.shoup/docs-13523-do-anomaly-detection

OliviaShoup commented Sep 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

DavidBraslow Sep 15, 2026

Uh oh!

DavidBraslow Sep 15, 2026 •

edited

Loading

Uh oh!

DavidBraslow Sep 15, 2026

Uh oh!

DavidBraslow Sep 15, 2026

Uh oh!

DavidBraslow Sep 15, 2026

Uh oh!

DavidBraslow Sep 15, 2026

Uh oh!

DavidBraslow Sep 15, 2026

Uh oh!

DavidBraslow Sep 15, 2026

Uh oh!

DavidBraslow Sep 15, 2026

Uh oh!

DavidBraslow Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		When you create an anomaly detection monitor, it enters a training period. During training, the monitor collects historical values to learn the metric's baseline behavior. It does not trigger alerts during this period, and the monitor chart appears in blue.

		Training typically takes between 3 and 7 days, depending on how frequently the underlying metric updates. Because many data pipelines behave differently on weekends, the model needs to observe both weekday and weekend behavior.

	Training typically takes between 3 and 7 days, depending on how frequently the underlying metric updates. Because many data pipelines behave differently on weekends, the model needs to observe both weekday and weekend behavior.
	Training typically takes between 3 and 9 days. Because many data pipelines behave differently on weekends, the model needs to observe both weekday and weekend behavior.


		Training typically takes between 3 and 7 days, depending on how frequently the underlying metric updates. Because many data pipelines behave differently on weekends, the model needs to observe both weekday and weekend behavior.

		After training completes, the monitor begins alerting. The chart uses color to indicate the current evaluation state:

	After training completes, the monitor begins alerting. The chart uses color to indicate the current evaluation state:
	After training completes, the monitor can alert. The chart shows the observed values over time, with bounds above and below the line as a shaded area showing expected values. The color shows the current state:


		## Model states

		The model often reuses prior predictions rather than generating new ones, such as when a value hasn't changed or when the monitor is in an alerting state. As a result, adding [annotations][2] is the primary way to get an alerting monitor back to a normal state and resume learning from new values.

	The model often reuses prior predictions rather than generating new ones, such as when a value hasn't changed or when the monitor is in an alerting state. As a result, adding [annotations][2] is the primary way to get an alerting monitor back to a normal state and resume learning from new values.
	When an anomaly is found, the model will remain in alert state for some time, until either the observed value returns to the original expected bounds or the anomaly has persisted long enough that the model considers it a new normal state. To resolve the monitor manually and return it to normal state, use [annotations][2].


		## Metric-specific behavior

		While the core model logic is shared across metric types, each type has additional rules:

	While the core model logic is shared across metric types, each type has additional rules:
	The model works differently for different metrics:

	Not all seasonal patterns are available for all metric types. Additionally, the model requires multiple complete cycles of normal ("green") history before it can detect a given pattern.
	Not all seasonal patterns are available for all metric types. Additionally, the model requires multiple complete cycles of normal history before it can detect a given pattern.


		## Annotations

		Annotations let you provide feedback to the model when it misclassifies a point, either by missing an alert or by generating a false alert. Because data quality expectations are often business-specific, annotations are the primary way to tune the model to your team's needs over time.

	Annotations let you provide feedback to the model when it misclassifies a point, either by missing an alert or by generating a false alert. Because data quality expectations are often business-specific, annotations are the primary way to tune the model to your team's needs over time.
	Annotations let you immediately retrain the model when it misclassifies a point, either by missing an alert or by generating a false alert. Because data quality expectations are often metric-specific, annotations are the primary way to tune the model to your team's needs.

	- Correcting the current state: Marking a flagged point as expected moves the monitor out of the alerting state and resumes normal learning.
	- Correcting the current state: Marking a flagged point as expected moves the monitor out of the alerting state to normal state at the next observation, so it can then alert on new anomalies.

	- Shaping future predictions: The model weights annotated points when generating future bounds, so repeated feedback improves accuracy over time.
	- Shaping future predictions: The model uses annotated points to adjust future bounds, so feedback immediately improves accuracy.


		Row count monitors alert when a table's row count has flatlined, meaning it has not changed for longer than normal. A stalled row count may indicate a broken pipeline.

		### Percentage (nullness, uniqueness)

	### Percentage (nullness, uniqueness)
	### Percentage (e.g. nullness, uniqueness)

Uh oh!

Conversation

OliviaShoup commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do? What is the motivation?

Merge readiness

For Datadog employees:

AI assistance

Additional notes

Uh oh!

github-actions Bot commented Sep 15, 2026

Preview links (active after the build_preview check completes)

New or renamed files

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DavidBraslow Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

OliviaShoup commented Sep 15, 2026 •

edited

Loading

Preview links (active after the `build_preview` check completes)

DavidBraslow Sep 15, 2026 •

edited

Loading