air run: package code_source tarball via DABs artifact upload (#6015) - #6109
Open
vinchenzo-db wants to merge 1 commit into
Open
air run: package code_source tarball via DABs artifact upload (#6015)#6109vinchenzo-db wants to merge 1 commit into
vinchenzo-db wants to merge 1 commit into
Conversation
Package the code_source working tree into a tarball and upload it
through DABs' artifact-upload plumbing (libraries.ReplaceWithRemotePath
+ libraries.Upload over a minimal in-memory bundle), rewriting
ai_runtime_task.code_source_path to the uploaded remote path. The
packaging + upload orchestration is CLI-owned (experimental/air/cmd,
OWNERS = us); it only reuses DABs' uploader so we don't reimplement
workspace/volume upload.
snapshot_dabs.go: build the plain-tar tarball (createPlainTarball),
carry it as a file-valued code_source_path on a minimal bundle, and
drive the DABs upload. runsubmit.go swaps the old raw-filer snapshot
upload for this. Removes the retired raw-filer upload path (snapshot.go
uploader, snapshot_test.go).
Tar snapshotting only; git pinning follows in the next PR (its git
helpers are removed here and reintroduced there).
Co-authored-by: Isaac
<!-- Brief summary of your changes that is easy to understand -->
<!-- Why are these changes needed? Provide the context that the reviewer
might be missing.
For example, were there any decisions behind the change that are not
reflected in the code itself? -->
`experimental/air/cmd/...` and `acceptance/experimental/air/run-submit`
— working-tree,
git-pinned, and remote-Volume submits each assert the tarball lands
under `.internal/`
and the rewritten `code_source_path` rides the submitted
`ai_runtime_task`; plus the tar
builders (`.gitignore`, `.git` exclusion, `include_paths`) and the
no-`code_source`
nil-guard. All green.
5/5 runs SUCCESS, one per packaging mode. All runs are `CAN_VIEW` for
the workspace
`users` group, so every link below is openable by anyone in the
workspace.
**Setup** — a tiny project with a gitignored file (`debug.log`) to prove
exclusion:
```bash
mkdir -p /tmp/air-demo/proj/src/pkg && cd /tmp/air-demo/proj
printf 'import os\nprint("train ran; cwd:", os.getcwd(), "CODE_SOURCE_PATH:", os.environ.get("CODE_SOURCE_PATH"))\n' > src/train.py
echo 'def helper(): return 1' > src/pkg/util.py
echo '*.log' > src/.gitignore
echo 'noise' > src/debug.log # gitignored — must never be uploaded
cat > wt.yaml <<'YAML'
experiment_name: vchen_demo_wt
command: cd $CODE_SOURCE_PATH && python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
environment: {version: "4", dependencies: []}
code_source: {type: snapshot, snapshot: {root_path: src}}
YAML
```
**1. Working-tree tarball** — plain tar of the working tree, honoring
`.gitignore`.
Run:
[994765091508414](https://dbc-04ac0685-8857.staging.cloud.databricks.com/jobs/runs/994765091508414)
```bash
dbcli experimental air run -f wt.yaml -p dbc-04ac0685-8857
dbcli workspace export /Workspace/Users/v.chen@databricks.com/.air/repo_snapshots/.internal/src.tar.gz --file /tmp/wt.tar.gz -p dbc-04ac0685-8857
tar tzf /tmp/wt.tar.gz
```
**2. Git-pinned commit** — `git archive` of a pinned SHA. An uncommitted
file is created
*after* the commit to prove the archive captures the commit, not the
dirty working tree.
Run:
[304463075281818](https://dbc-04ac0685-8857.staging.cloud.databricks.com/jobs/runs/304463075281818)
```bash
git init -q && git add -A && git commit -qm init
SHA=$(git rev-parse HEAD)
echo "print('uncommitted')" > src/uncommitted.py # created AFTER the commit
cat > git.yaml <<YAML
experiment_name: vchen_demo_git
command: cd \$CODE_SOURCE_PATH && python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
environment: {version: "4", dependencies: []}
code_source: {type: snapshot, snapshot: {root_path: src, git: {commit: $SHA}}}
YAML
dbcli experimental air run -f git.yaml -p dbc-04ac0685-8857
dbcli workspace export /Workspace/Users/v.chen@databricks.com/.air/repo_snapshots/.internal/src.tar.gz --file /tmp/git.tar.gz -p dbc-04ac0685-8857
tar tzf /tmp/git.tar.gz
```
**3. UC Volume destination** — `remote_volume` routes the upload to a UC
Volume via the
Files API (`/api/2.0/fs/files/...`), natively, with no special-casing in
the CLI.
Run:
[438683652713410](https://dbc-04ac0685-8857.staging.cloud.databricks.com/jobs/runs/438683652713410)
```bash
dbcli volumes create main default vchen_demo MANAGED -p dbc-04ac0685-8857
cat > vol.yaml <<'YAML'
experiment_name: vchen_demo_vol
command: cd $CODE_SOURCE_PATH && python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
environment: {version: "4", dependencies: []}
code_source: {type: snapshot, snapshot: {root_path: src, remote_volume: /Volumes/main/default/vchen_demo}}
YAML
dbcli experimental air run -f vol.yaml -p dbc-04ac0685-8857 --debug 2>&1 | grep -iE "submitted|code_source_path|/api/2.0/fs/files"
dbcli fs ls dbfs:/Volumes/main/default/vchen_demo/.internal -p dbc-04ac0685-8857
```
**4. `include_paths` subset** — only the listed paths are packaged.
Run:
[261250771126835](https://dbc-04ac0685-8857.staging.cloud.databricks.com/jobs/runs/261250771126835)
```bash
cat > inc.yaml <<'YAML'
experiment_name: vchen_demo_inc
command: cd $CODE_SOURCE_PATH && python train.py
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
environment: {version: "4", dependencies: []}
code_source: {type: snapshot, snapshot: {root_path: src, include_paths: [pkg]}}
YAML
dbcli experimental air run -f inc.yaml -p dbc-04ac0685-8857
dbcli workspace export /Workspace/Users/v.chen@databricks.com/.air/repo_snapshots/.internal/src.tar.gz --file /tmp/inc.tar.gz -p dbc-04ac0685-8857
tar tzf /tmp/inc.tar.gz
```
**5. No `code_source`** — nothing is uploaded; `code_source_path` is
left empty (nil-guard).
Run:
[138286541552104](https://dbc-04ac0685-8857.staging.cloud.databricks.com/jobs/runs/138286541552104)
```bash
cat > none.yaml <<'YAML'
experiment_name: vchen_demo_none
command: echo hello
compute: {accelerator_type: GPU_1xA10, num_accelerators: 1}
environment: {version: "4", dependencies: []}
YAML
dbcli experimental air run -f none.yaml -p dbc-04ac0685-8857
```
Contributor
Approval status: pending
|
Collaborator
Integration test reportCommit: 0d872ed
8 interesting tests: 4 SKIP, 3 KNOWN, 1 RECOVERED
Top 3 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Package the code_source working tree into a tarball and upload it through DABs' artifact-upload plumbing (libraries.ReplaceWithRemotePath
snapshot_dabs.go: build the plain-tar tarball (createPlainTarball), carry it as a file-valued code_source_path on a minimal bundle, and drive the DABs upload. runsubmit.go swaps the old raw-filer snapshot upload for this. Removes the retired raw-filer upload path (snapshot.go uploader, snapshot_test.go).
Tar snapshotting only; git pinning follows in the next PR (its git helpers are removed here and reintroduced there).
Co-authored-by: Isaac
experimental/air/cmd/...andacceptance/experimental/air/run-submit— working-tree,git-pinned, and remote-Volume submits each assert the tarball lands under
.internal/and the rewritten
code_source_pathrides the submittedai_runtime_task; plus the tarbuilders (
.gitignore,.gitexclusion,include_paths) and the no-code_sourcenil-guard. All green.
5/5 runs SUCCESS, one per packaging mode. All runs are
CAN_VIEWfor the workspaceusersgroup, so every link below is openable by anyone in the workspace.Setup — a tiny project with a gitignored file (
debug.log) to prove exclusion:1. Working-tree tarball — plain tar of the working tree, honoring
.gitignore.Run:
994765091508414
dbcli experimental air run -f wt.yaml -p dbc-04ac0685-8857 dbcli workspace export /Workspace/Users/v.chen@databricks.com/.air/repo_snapshots/.internal/src.tar.gz --file /tmp/wt.tar.gz -p dbc-04ac0685-8857 tar tzf /tmp/wt.tar.gz2. Git-pinned commit —
git archiveof a pinned SHA. An uncommitted file is createdafter the commit to prove the archive captures the commit, not the dirty working tree.
Run:
304463075281818
3. UC Volume destination —
remote_volumeroutes the upload to a UC Volume via theFiles API (
/api/2.0/fs/files/...), natively, with no special-casing in the CLI.Run:
438683652713410
4.
include_pathssubset — only the listed paths are packaged. Run:261250771126835
5. No
code_source— nothing is uploaded;code_source_pathis left empty (nil-guard).Run:
138286541552104
Changes
Why
Tests