experimental/air: upload incremental code snapshots - #6691
Draft
ben-hansen-db wants to merge 7 commits into
Draft
ben-hansen-db wants to merge 7 commits into
ben-hansen-db wants to merge 7 commits into
Conversation
git_archive snapshots are already content-addressed: a repeat submission at the same commit reuses the uploaded tarball and skips packaging + upload. plain_tar (dirty working tree) used a timestamped name, so it re-packaged and re-uploaded the full tarball on every submission, even when nothing changed. Name the plain_tar tarball by a working-tree fingerprint (sha256 over each file's path, size and mtime) and run the same snapshotExists skip for both modes. An unchanged resubmit now reuses the remote object and moves no bytes. The listing is captured once and threaded into packaging, so the tree is walked only once. The fingerprint is size+mtime, not content, matching DABs file-sync. Verified on df1: a second submission of an unchanged tree logs "snapshot upload skipped; reusing ..." and returns the identical remote path. Co-authored-by: Isaac <no-reply@databricks.com>
- Strengthen the dedup test: count import-file calls and assert the second
(unchanged) submit adds zero, instead of asserting a path-keyed set has one
entry. The set couldn't distinguish a skipped submit from a re-upload to the
same content-addressed name; the counter can (verified it fails when the skip
is disabled).
- Fold snapshotPackagingVersion into computePlainTarKey so a packaging-logic
bump invalidates plain_tar keys too, not just plainTarKeyVersion.
- Fix stale comments now that plain_tar is content-addressed: modePlainTar
("not cacheable") and snapshotExists ("git_archive" only).
Co-authored-by: Isaac <no-reply@databricks.com>
Add a local warm cache for the plain_tar snapshot path, keyed by (repo, config, include_paths) under $TMPDIR/databricks/.air/<key>: an uncompressed snapshot.tar plus a manifest of each file's size+mtime and byte range. Later runs stat the file set, copy unchanged members verbatim from the warm tar, and re-read only changed files before gzipping the upload. --no-cache bypasses it and re-packs from scratch. The cache engages only above 64 MiB. The cache's payoff is largest when the working set does not fit the OS page cache: cold, scattered small-file reads cost seconds for a large folder versus ~ms to read the warm tar sequentially. When the tree is already warm in RAM, parallel gzip accounts for most of the gain and the cache adds little. Co-authored-by: Isaac <no-reply@databricks.com>
Match the parent PR: DefaultCompression rather than BestSpeed in newGzFile, so the cached tarball is re-gzipped at the same level as the --no-cache path and the upload stays small. Parallel compression makes the higher level nearly free. Co-authored-by: Isaac <no-reply@databricks.com>
The new --no-cache flag on `air run` adds a line to its --help output, which the experimental/air/config-help acceptance test pins. Regenerate the golden. Co-authored-by: Isaac <no-reply@databricks.com>
Isaac Review flagged two MAJOR correctness bugs in the warm cache: - Concurrent `air run` on the same cache key wrote the same snapshot.tar.tmp and raced the rename plus a non-atomic manifest write, interleaving into a corrupt tar/manifest pair. - rebuildWarmSnapshot renamed the new tar into place before saving the manifest, so a crash between the two left a manifest whose byte offsets described a different tar layout -- silently corrupting a later verbatim-reuse rebuild. Fix both by binding the manifest to a per-build, uniquely named tar (snapshot.<id>.tar) that is never overwritten, and installing the manifest atomically (unique temp + rename) only after its tar is durable. A manifest and the tar it indexes are therefore always a consistent pair: there is no window where offsets describe a mismatched tar, and concurrent rebuilds are last-writer-wins on the manifest rather than interleaving, so no lock is needed. Superseded and orphaned tars are cleaned up best-effort. Adds a rotation test. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: d40316b
Top 3 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds incremental anchor-and-overlay uploads for repeated local
databricks experimental air runsubmissions with largeplain_tarcode sources.--no-cache.Node-side materialization is implemented by the paired draft: https://github.com/databricks-eng/universe/pull/2617554
Validation
go test ./experimental/air/cmd -run "Snapshot|PlainTar|Overlay"go tool -modfile=tools/go.mod golangci-lint run --allow-parallel-runners -j=4 ./experimental/air/cmd— 0 issues./task buildEnd-to-end on
e2-dogfoodwith a 96 MiB incompressible fixture:The complete
go test ./experimental/air/cmdrun reaches one current-mainfailure inTestSubmitWorkload: the SDK now populatesComputeSpec.ForceSendFields, while the base test compares the full struct against an expected value without that internal field. The snapshot-focused suite passes.Notes
This is intentionally WSFS-only initially. Remote snapshot garbage collection is out of scope; selecting a newer anchor does not delete historical objects. Initial eligibility and re-anchor thresholds are experiment inputs and should be tuned from end-to-end measurements.