Skip to content

bundle: upload files after plan approval and DMS version creation - #6704

Draft
shreyas-goenka wants to merge 5 commits into
mainfrom
shreyas-goenka_data/shreyas-goenka/upload-after-dms-version
Draft

shreyas-goenka wants to merge 5 commits into
mainfrom
shreyas-goenka_data/shreyas-goenka/upload-after-dms-version

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Changes

Move artifact upload and file sync, their inventory update/push, and directory permissions after planning, approval, and DMS version creation. Open the resource WAL only after uploads succeed. Renew the DMS lease during deployment and drain the operation buffer even when uploads fail before resource apply.

Why

Previously a failed plan, declined deployment, or rejected DMS CreateVersion could leave uploaded files in the workspace. These failures now stop before uploading. Keeping inventory persistence after sync preserves file deletion tracking on fresh CI checkouts; permissions follow creation of custom artifact/file directories.

File-only deployments retain the existing behavior: an empty resource plan does not create a DMS version. Tracking those uploads as a versioned operation is a separate follow-up.

Tests

Local DMS/deploy acceptance tests cover saved plans and fresh-checkout file recovery. Added a failed-upload/retry test with a custom artifact directory and disabled workspace locking, assertions that rejected versions perform no content writes, and heartbeat renewal/failure/shutdown tests. Phase tests pass with the race detector; changed Go packages pass lint. Periodic heartbeat requests are normalized in request goldens, with an explicit option to inspect them.

CI is green on f8112747fab: direct, DMS, and Terraform suites on Linux/macOS/Windows, plus all six AWS/Azure/GCP Linux/Windows integration jobs.

This PR was written with Codex.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: f811274

Run: 35112370286

Env 🔄​flaky ✅​pass 🙈​skip Time
🔄​ aws linux 3 331 42 26:44
🔄​ aws windows 4 332 40 30:21
🔄​ azure linux 1 308 48 20:41
🔄​ azure windows 6 305 46 28:45
🔄​ gcp linux 3 307 48 22:22
🔄​ gcp windows 1 311 46 17:06
16 interesting tests: 16 flaky
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
🔄​ TestAccept 🔄​f ✅​p ✅​p 🔄​f 🔄​f ✅​p
🔄​ TestAccept/bundle/dms/successful-recreate ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/dms/successful-recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/resources/dashboards/change-parent-path ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/resources/dashboards/change-parent-path/DATABRICKS_BUNDLE_ENGINE=terraform/DMS= ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/resources/postgres_branches/recreate ✅​p 🔄​f 🙈​s 🙈​s 🙈​s 🙈​s
🔄​ TestAccept/bundle/resources/postgres_branches/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS= ✅​p 🔄​f
🔄​ TestAccept/bundle/resources/postgres_databases/recreate 🔄​f ✅​p 🙈​s 🙈​s 🙈​s 🙈​s
🔄​ TestAccept/bundle/resources/postgres_databases/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS= 🔄​f ✅​p
🔄​ TestAccept/bundle/resources/postgres_projects/recreate ✅​p 🔄​f 🙈​s 🙈​s 🙈​s 🙈​s
🔄​ TestAccept/bundle/resources/postgres_projects/recreate/DATABRICKS_BUNDLE_ENGINE=terraform/DMS= ✅​p 🔄​f
🔄​ TestFsCpDir ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestFsCpDir/uc-volumes_to_dbfs ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestSyncFullFileSync ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestSyncIncrementalFileSync ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
🔄​ TestSyncIncrementalSyncPythonNotebookDelete ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
Top 50 slowest tests (at least 2 minutes):
duration env testname
6:33 gcp linux TestFilerWorkspaceFilesExtensionsReadDir
5:06 aws windows TestAccept/bundle/deployment/bind/dashboard/recreation/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=/READPLAN=
4:48 aws linux TestAccept/bundle/dms/successful-recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
4:34 gcp linux TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:28 gcp windows TestAccept
4:14 gcp windows TestFilerRecursiveDelete/workspace_files
4:02 azure windows TestAccept/bundle/resources/schemas/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
4:02 aws linux TestExportDir
3:55 aws windows TestAccept/bundle/resources/schemas/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:48 aws linux TestAccept/bundle/deployment/bind/dashboard/recreation/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/READPLAN=
3:44 azure windows TestImportDirWithOverwriteFlag
3:43 azure linux TestAccept/bundle/resources/catalogs/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:37 gcp windows TestAccept/bundle/deployment/bind/dashboard/recreation/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=/READPLAN=
3:34 azure linux TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:34 azure linux TestFilerWorkspaceFilesExtensionsReadDir
3:29 aws windows TestAccept/bundle/resources/schemas/auto-approve/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
3:26 gcp linux TestAccept/bundle/resources/schemas/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:25 azure windows TestAccept/bundle/deployment/bind/dashboard/recreation/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/READPLAN=1
3:23 aws linux TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:21 gcp linux TestAccept/bundle/resources/pipelines/auto-approve/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
3:20 aws windows TestAccept/bundle/deployment/bind/dashboard/recreation/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/READPLAN=
3:13 azure windows TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:11 aws windows TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:07 gcp linux TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:06 azure windows TestAccept/bundle/dms/successful-recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:05 azure windows TestAccept/bundle/resources/dashboards/change-parent-path/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:04 aws windows TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
3:03 azure linux TestAccept/bundle/resources/pipelines/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
3:00 aws windows TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
2:58 azure linux TestImportDirWithOverwriteFlag
2:57 azure linux TestAccept/bundle/resources/pipelines/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:56 gcp windows TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
2:55 gcp windows TestAccept/bundle/resources/dashboards/change-parent-path/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
2:55 gcp windows TestAccept/bundle/resources/pipelines/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:51 aws windows TestAccept/bundle/resources/postgres_branches/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:50 aws linux TestAccept/bundle/resources/catalogs/auto-approve/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:49 gcp linux TestAccept/bundle/resources/schemas/auto-approve/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
2:44 azure windows TestFilerWorkspaceFilesExtensionsReadDir
2:44 gcp windows TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
2:43 gcp linux TestAccept/bundle/deployment/bind/dashboard/recreation/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/READPLAN=
2:42 aws linux TestAccept/bundle/resources/postgres_branches/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
2:42 azure windows TestAccept/bundle/resources/dashboards/change-parent-path/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
2:42 gcp linux TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
2:41 azure windows TestAccept/bundle/resources/pipelines/recreate/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
2:40 azure windows TestAccept/bundle/resources/volumes/recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:40 gcp linux TestAccept/bundle/deployment/bind/dashboard/recreation/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/READPLAN=1
2:40 azure windows TestAccept/bundle/deployment/bind/pipelines/recreate/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
2:39 gcp windows TestAccept/bundle/resources/dashboards/change-parent-path/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
2:39 azure linux TestAccept/bundle/resources/dashboards/change-parent-path/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
2:38 aws windows TestAccept/bundle/deployment/bind/pipelines/recreate/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants