Add bundle bitmap for per-field telemetry - #6104
Draft
denik wants to merge 11 commits into
Draft
Conversation
denik
force-pushed
the
denik/bundle-bitmap
branch
from
July 30, 2026 10:11
221d417 to
39faace
Compare
Collaborator
Integration test reportCommit: 125b797
8 interesting tests: 3 SKIP, 3 KNOWN, 2 RECOVERED
Top 35 slowest tests (at least 2 minutes):
|
Co-authored-by: Isaac
Encode presence of every bundle config field as one bit. The schema is derived by reflecting over config.Root, recorded in bundle/bitmap/schema.txt, and embedded into the binary. Append-only so bit positions stay stable. Adds hidden `databricks bundle bitmap` group: schema, update-schema [--validate], bitmap-text, bitmap. Wires generate-bitmap-schema into the Taskfile and runs the encoder over all no_drift invariant configs. Co-authored-by: Isaac
Kept in PR history for context; not landed. Co-authored-by: Isaac
- Every bool/*bool field now gets a "#true" value bit in addition to its presence bit, so a *bool encodes unset/false/true as 00/10/11. - decode_bitmap.py decodes the base64 bitmap against the schema; the dedicated acceptance test asserts it round-trips to bitmap-text. - Record the encoded bitmap and set-bit list for the default-python template. Co-authored-by: Isaac
Replace the string-keyed BoolValues map with a typed bundle.Telemetry struct whose fields feed both the deploy telemetry wire (via BoolValues()) and the bundle bitmap (namespaced under telemetry.*). Value-bearing flags whose false is meaningful are stored as True/False pairs so the bitmap can represent both outcomes; BoolValues() collapses each pair back to its single wire key. libs/cache gets a concrete Metrics struct (embedded into Telemetry) instead of the old interface. The wire is unchanged except that redundant false entries of one-hot groups (state_path_*, dms_undeclared_*) are no longer emitted; their true member is preserved. Co-authored-by: Isaac
Co-authored-by: Isaac
decode_bitmap.py was committed without the executable bit, so the round-trip step failed with "permission denied" (exit code differed across OSes, breaking the golden). Mark it executable. bitmap-text and bitmap are separate invocations sharing a cache dir, so the second recorded a cache hit instead of a miss and the round-trip check diverged. Give each its own cache dir so both observe a cold miss. Co-authored-by: Isaac
On a CRLF checkout (Windows) the embedded schema.txt lines kept a trailing \r, so they never matched the clean paths from WalkType. Merge's dedup then failed and the schema doubled, breaking bundle bitmap schema/bitmap-text on Windows. splitLines now normalizes CRLF. Co-authored-by: Isaac
The schema had telemetry fields appended at the tail (from append-only merges across a rebase). Since the PR has not landed, bit positions need not be stable yet, so regenerate from scratch to put telemetry fields in natural walk order. generate-bitmap-schema now also regenerates the bitmap acceptance goldens (out.schema.txt, out.bitmap-text.txt, encoded bitmap), which embed the schema and depend on bit positions; otherwise they go stale when the schema changes. Co-authored-by: Isaac
Existing telemetry (bundle/metrics AddBoolValue/SetBoolValue and the wire) is unchanged. Add a Telemetry struct on Bundle with a field per bool flag; each metrics call site additionally sets the matching field so the same signal reaches the bitmap. Value-bearing flags use True/False pairs so the presence-only bitmap can distinguish measured-false from not-measured. Fields have no json tags, so they are invisible to the config schema and walked into the bitmap under their Go field names (telemetry.*). Verified the telemetry and resource_deps goldens are byte-identical to main: the change is strictly additive to the existing wire. Co-authored-by: Isaac
The four dms_undeclared_* flags are independent booleans recorded with their real value on every deploy, so a measured false is meaningful (it is the common "no undeclared writers" case). As single presence bits a false collapsed to 0, indistinguishable from "not measured", losing the signal. Store each as a True/False pair like the other value-bearing flags. The telemetry wire (BoolValues) is unchanged. Co-authored-by: Isaac
denik
force-pushed
the
denik/bundle-bitmap
branch
from
July 31, 2026 14:59
095720d to
125b797
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Today each tracked DABs feature needs custom code (a key constant plus a SetBoolValue call from a mutator), so only hand-picked features are tracked. The bundle bitmap encodes the presence of every bundle config field as one bit, derived by reflecting over config.Root, so all current and future fields are covered for free (http://go/dabs-telemetry).
Changes
databricks bundle bitmapgroup:schema,bitmap-text,bitmap(compressed + base64), andupdate-schema [--validate]. The schema is recorded in bundle/bitmap/schema.txt, embedded into the binary, and append-only so bit positions stay stable across releases.task generate/generate-check, and the encoded bitmap is exercised over every no_drift invariant config.The deploy telemetry wire is unchanged except that redundant false entries of one-hot groups (state_path_, dms_undeclared_) are no longer emitted; their true member is preserved.