Repository navigation
Add data_quality_monitors bundle resource (direct engine) - #7094
Merged
Merged
Conversation
Adds the data-quality/v1 Monitor API as a direct-engine bundle resource, alongside the legacy quality_monitors resource it supersedes. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: 5a32e8e
Top 14 slowest tests (at least 2 minutes):
|
…ceptance test Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 13 changed tests (limit=20, 0 not selected) 658 tests: regression=14 unreleased=0 coverage=644 failing=0
14 more failure details omitted, see the full report in the run. |
…y monitors Co-authored-by: Isaac <no-reply@databricks.com>
…n alert Co-authored-by: Isaac <no-reply@databricks.com>
kevinlian2it
marked this pull request as ready for review
October 29, 2026 18:23
kevinlian2it
requested review from
janniklasrose and
lennartkats-db
and removed request for
lennartkats-db
October 29, 2026 18:23
Member
There was a problem hiding this comment.
what's the purpose of this folder vs resources/data_quality_monitors/*? let's remove/fold it into the others if this test offers additional coverage
Contributor
Author
There was a problem hiding this comment.
Folded it into resources/data_quality_monitors/data_profiling: it now has a dev target and the script starts by validating it (schedule dropped), while the default target's deploy keeps it. Removed the separate folder.
| # Create:REQ Update:IGN in the data-quality API: the update endpoint ignores it. | ||
| # Same as quality_monitors.assets_dir. | ||
| - field: data_profiling_config.assets_dir | ||
| reason: immutable |
Member
There was a problem hiding this comment.
can this be annotated in the upstream proto to get it out of the box (it'd show up under data_quality_monitors.generated.yml as spec:immutable). Hand-writing it is a smell
Contributor
Author
There was a problem hiding this comment.
Yes: databricks-eng/universe#2963330 marks assets_dir IMMUTABLE in the OpenAPI spec (spec-only, since the backend's ApiUtil.update rejects * masks touching a google.api.field_behavior IMMUTABLE field). I'll swap the hand-written entry for the generated one once an SDK bump picks it up.
Member
There was a problem hiding this comment.
I'll swap the hand-written entry for the generated one once an SDK bump picks it up.
I'm pretty sure the SDK bump will do this automatically (DABs should be set up to error on the field conflict) so no action from you besides merging that upstream PR :-)
Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds a
data_quality_monitorsbundle resource for the data-quality/v1 Monitor API (anomaly detection on schemas, data profiling on tables). Direct engine only. The legacyquality_monitorsresource is unchanged; deprecating it is a separate follow-up.bundle/config/resources/data_quality_monitor.goembedsdataquality.Monitor. The ID is<object_type>,<object_id>, the same as the Terraform import ID fordatabricks_data_quality_monitor.bundle/direct/dresources/data_quality_monitor.go: CRUD onw.DataQuality. Update sendsupdate_mask: *.configs/data_quality_monitors.yml:object_typeandobject_idare ID fields;data_profiling_config.assets_dirrecreates on change (the update API ignores it). Output-only fields and the input-onlyskip_builtin_dashboard/warehouse_id(never returned on GET) come from the generated config.quality_monitors:run_asis rejected (the API has none), and presets that pause triggers dropdata_profiling_config.schedule(itspause_statusis read-only). The monitor has no name, so there is no name prefix. It has no permissions and no workspace URL, becauseobject_idis a UUID.Why
Data quality monitoring is moving to GA on the data-quality/v1 API, and
quality_monitorsonly covers the legacy table-monitor API. This resource lets bundles manage v1 monitors, including schema-level anomaly detection, which the legacy resource can't express.Notes for reviewers:
ListMonitoris unimplemented server-side, so lookup variables aren't supported for this resource.snapshot: {}is lost when a monitor is defined in Python.quality_monitorshas the same pre-existing limitation; it's visible in thedata_quality_monitors-supportgolden.*,anomaly_detection_config.notification_settingsis replaced on every update, so omitting it removes an alert set in the Data Quality Monitoring UI. The bundle is the source of truth; this is documented on the resource.data_quality_monitorsto the deployment metadata service, and databricks-eng/universe#2867273 marked the input-only fields in the spec (in SDK v0.191.0).Tests
all_test.go,state_load_test.go,resources_test.go, and the mutator tests.apply_target_mode_testasserts the dev-mode schedule drop.resources/data_quality_monitors/basic: schema monitor that references${resources.schemas.x.schema_id}. Covers create, in-place update, no drift, recreate on an identity change, and destroy.data_profiling: table monitor. Covers output-only fields not drifting, in-place schedule update, andassets_dirrecreate.remote-delete: an out-of-band delete is detected.lifecycle: cloud-capable.bundle/data_quality_monitor: dev-mode target drops the schedule; production keeps it.deployment/bind/data_quality_monitor: bind and unbind, and a malformed ID is rejected.python/data_quality_monitors-support.golangci-lint run ./...reports 0 issues. tidy, whitespace, deadcode, codeowners and lockfile checks pass, and all generators are idempotent. 505 PyDABs tests pass.lifecyclepasses on all clouds, with and without deployment history (DMS).This pull request and its description were written by Isaac.