[POC] DABs AIR native fields by vinchenzo-db · Pull Request #6428 · databricks/cli

vinchenzo-db · 2026-08-28T18:52:06Z

Proposal for AIR-on-DABs code_source parity (see DABs x AIR CLI Alignment). Extends the existing artifacts block so DABs can build the code tarball itself instead of only uploading a user build command's output:

  artifacts:
    code_source:
      type: tgz            # new ArtifactType (was whl|jar)
      include: [src/...]   # subpaths to pack, gitignore-honored
      git: {branch|commit} # snapshot a ref instead of the working tree
      files: [{source: ./dist/code.tgz}]

build and git/include are mutually exclusive: either the user's command produces the tarball (today's behavior) or DABs does. The produced file flows through the existing artifact upload path, so an ai_runtime_task's code_source_path pointing at it is uploaded and rewritten to the remote path with no new reference syntax.

include: reuses the bundle sync walker (matches file-sync filtering)
git: shells out to git archive Verified end-to-end on staging (both modes deploy; code_source_path rewritten).

Open for review: this is a proposal to react to, not a finished feature. jsonschema annotations for the new fields are a follow-up.

Co-authored-by: Isaac

Changes

Why

Tests

Show that the include field is respected, we notice the ./dist/code.tgz is properly uploaded.

 v.chen at https://dbc-04ac0685-8857 in ~/.worktrees/cli-artifacts (git:air-artifacts-code-source) [20:04:39]
$ rm -rf /tmp/demo-include && mkdir -p /tmp/demo-include/src && cd /tmp/demo-include
printf 'print("hello from packaged code")\n' > src/train.py
printf '#!/bin/bash\ncd "$(dirname "$0")"\npython src/train.py\n' > command.sh
cat > databricks.yml <<YAML
bundle: {name: demo-include}
targets: {dev: {mode: development, default: true, workspace: {host: $HOST}}}
artifacts:
  code_source:
    type: tgz
    include: [src]
    files: [{source: ./dist/code.tgz}]
resources:
  jobs:
    demo_include:
      name: demo-include
      tasks:
        - task_key: train
          environment_key: default
          ai_runtime_task:
            experiment: demo-include
            deployments: [{command_path: ./command.sh, compute: {accelerator_type: GPU_1xA10, accelerator_count: 1}}]
            code_source_path: ./dist/code.tgz
      environments: [{environment_key: default, spec: {environment_version: "5", dependencies: []}}]
YAML

$CLI bundle deploy -t dev                       # → "Building code_source..." → "Uploading dist/code.tgz..."
tar tzf dist/code.tgz                            # → src/train.py   (DABs built it from ./src)
JID=$($CLI bundle summary -t dev | grep -oE 'jobs/[0-9]+' | head -1 | cut -d/ -f2)
$CLI jobs get $JID | grep code_source_path       # → rewritten to /Workspace/.../artifacts/.internal/code.tgz
$CLI bundle destroy -t dev --auto-approve

Building code_source...
Uploading dist/code.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/demo-include/dev/files...
Created jobs.demo_include
Files: 4 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
src/train.py
          "code_source_path": "/Workspace/Users/v.chen@databricks.com/.bundle/demo-include/dev/artifacts/.internal/code.tgz",
The following resources will be deleted:
  delete resources.jobs.demo_include

All files and directories at the following location will be deleted: /Workspace/Users/v.chen@databricks.com/.bundle/demo-include/dev

Destroy: 1 deleted

Show that git refs are respected:

# v.chen at https://dbc-04ac0685-8857 in /tmp/demo-include (git:) [20:04:57]
$ rm -rf /tmp/demo-git && mkdir -p /tmp/demo-git/src && cd /tmp/demo-git
git init -q && git config user.email t@t.co && git config user.name t
printf 'print("committed code")\n' > src/train.py
printf '#!/bin/bash\ncd "$(dirname "$0")"\npython src/train.py\n' > command.sh
git add -A && git commit -qm init
BR=$(git rev-parse --abbrev-ref HEAD)
cat > databricks.yml <<YAML
bundle: {name: demo-git}
targets: {dev: {mode: development, default: true, workspace: {host: $HOST}}}
artifacts:
  code_source:
    type: tgz
    git: {branch: $BR}          # or  commit: <sha>
    include: [src]
    files: [{source: ./dist/code.tgz}]
resources:
  jobs:
    demo_git:
      name: demo-git
      tasks:
        - task_key: train
          environment_key: default
          ai_runtime_task:
            experiment: demo-git
            deployments: [{command_path: ./command.sh, compute: {accelerator_type: GPU_1xA10, accelerator_count: 1}}]
            code_source_path: ./dist/code.tgz
      environments: [{environment_key: default, spec: {environment_version: "5", dependencies: []}}]
YAML

$CLI bundle deploy -t dev
tar tzf dist/code.tgz                            # → src/train.py   (from `git archive` of the ref)
$CLI bundle destroy -t dev --auto-approve
Databricks pre-commit Git Hook V2.5.0
Running secret scanning on changes staged for commit.
secret-scan hook completed in 126 ms
Unknown project name: None, skipping linting.
pre-commit-total hook completed in 153 ms
Databricks commit-msg Git Hook V2.5.0
Running secret scanning on commit message.
secret-scan hook completed in 40 ms
commit-msg-total hook completed in 50 ms
Building code_source...
Uploading dist/code.tgz...
Uploading bundle files to /Workspace/Users/v.chen@databricks.com/.bundle/demo-git/dev/files...
Created jobs.demo_git
Files: 4 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
src/
src/train.py
The following resources will be deleted:
  delete resources.jobs.demo_git

All files and directories at the following location will be deleted: /Workspace/Users/v.chen@databricks.com/.bundle/demo-git/dev

Destroy: 1 deleted

…git ref Proposal for AIR-on-DABs code_source parity (see DABs x AIR CLI Alignment). Extends the existing `artifacts` block so DABs can build the code tarball itself instead of only uploading a user build command's output: artifacts: code_source: type: tgz # new ArtifactType (was whl|jar) include: [src/...] # subpaths to pack, gitignore-honored git: {branch|commit} # snapshot a ref instead of the working tree files: [{source: ./dist/code.tgz}] `build` and `git`/`include` are mutually exclusive: either the user's command produces the tarball (today's behavior) or DABs does. The produced file flows through the existing artifact upload path, so an ai_runtime_task's code_source_path pointing at it is uploaded and rewritten to the remote path with no new reference syntax. - include: reuses the bundle sync walker (matches file-sync filtering) - git: shells out to `git archive` Verified end-to-end on staging (both modes deploy; code_source_path rewritten). Open for review: this is a proposal to react to, not a finished feature. jsonschema annotations for the new fields are a follow-up. Co-authored-by: Isaac

github-actions · 2026-08-28T18:52:43Z

Approval status: pending

`/acceptance/bundle/` - needs approval

Files: acceptance/bundle/validate/strict/output.txt
Suggested: @janniklasrose
Also eligible: @pietern, @denik, @shreyas-goenka, @andrewnester, @anton-107, @lennartkats-db

`/bundle/` - needs approval

11 files changed
Suggested: @janniklasrose
Also eligible: @pietern, @denik, @shreyas-goenka, @andrewnester, @anton-107, @lennartkats-db

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.

See OWNERS for ownership rules.}

- annotations.yml: descriptions for the new `include`/`git` (+ branch/commit) fields and `tgz` type, so TestRequiredAnnotationsForNewFields passes. - jsonschema.json: regenerated (adds include/git/ArtifactGit/tgz; scoped diff). - validate/strict acceptance golden: artifact type enum now [whl jar tgz]. All bundle/... unit tests, go vet, and the artifacts+validate acceptance subsets pass. Co-authored-by: Isaac

…urce

- tarball.go: use errors.New for the no-args git error (perfsprint lint). - jsonschema.json: regenerated post-merge; picks up the variable-ref regex the newer generator emits (validate-generated). Co-authored-by: Isaac

eng-dev-ecosystem-bot · 2026-08-28T19:41:26Z

Integration test report

Commit: f2a8fd9

Run: 33203627766

	Env	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
💚	aws linux	1	1	274	1207	3:59
💚	aws windows	1	1	276	1205	4:06
💚	azure linux	1	1	273	1207	3:59
💚	azure windows	1	1	275	1205	3:38
💚	gcp linux	1	1	274	1207	4:18
💚	gcp windows	1	1	276	1205	3:44

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S

Top 3 slowest tests (at least 2 minutes):

duration	env	testname
4:01	aws windows	TestAccept
3:39	gcp windows	TestAccept
3:32	azure windows	TestAccept

ben-hansen-db

Looks good, let's follow the file generation flow for the autogen files

ben-hansen-db · 2026-08-31T20:54:33Z

+
+// tarballFromGit snapshots a git ref via `git archive`, so the archive reflects the
+// committed tree at that ref rather than the working tree. Commit wins over Branch.
+func tarballFromGit(ctx context.Context, b *bundle.Bundle, a *config.Artifact, w io.Writer) error {


have we tested equivalence of tarballs from here vs CLI? Note also the tarball changes that landed in CLI recently to respect gitignore

Made them share code instead of testing equivalence

ben-hansen-db · 2026-08-31T20:56:15Z

 		b.Metrics.AddBoolValue(metrics.ArtifactFilesIsSet, len(artifact.Files) != 0)

+		// A `tgz` artifact with `include`/`git` is built by DABs itself in the build
+		// phase (see artifacts.Build). `build` and `git`/`include` are mutually


is semantics for include the same as in air cli? There it is relative to code source root, is include relative to bundle root in DABs? Can we make sure that is documented well somewhere?

include is now relative to the code-source root, same as the air CLI. The reason I decided this is so that convert-to-dabs can map air CLI's include_paths straight across later.

ben-hansen-db · 2026-08-31T20:57:22Z

 var EnumFields = map[string][]string{
 	"artifacts.*.executable": {"bash", "sh", "cmd"},
-	"artifacts.*.type":       {"whl", "jar"},
+	"artifacts.*.type":       {"whl", "jar", "tgz"},


above it says this is autogenerated. Did claude update this file or was it infact autogenerated?

there's probably a flow for autogenerating this file to follow

^ claude ran the autogeneration script

- buildTarballArtifact writes to a temp file and renames on success, so a failed git archive / pack no longer leaves a partial tarball at the output path. - Drop the unreachable `.` fallback in tarballFromInclude; include mode is only entered with a non-empty a.Include. Co-authored-by: Isaac <no-reply@databricks.com>

Clarify that DABs `include` is bundle-root-relative and composes files from anywhere in the bundle (e.g. a code dir plus a sibling env file), intentionally broader than the air CLI's code-source-root-relative include. Update the field doc and schema annotation, and add a packing-layer test showing a tarball composed across two directories keeps both entries at bundle-root-relative paths. Co-authored-by: Isaac <no-reply@databricks.com>

The AI Runtime extracts a code_source tarball to /databricks/code_source/<dir>, so the archive must have a single load-bearing top-level directory. The previous include implementation packed bundle-root-relative entries with no such prefix, diverging from both the air CLI and DABs' own aicode directory packer. - Export aicode.BuildCodeSnapshot and reuse it for include mode, so entries nest under the code-source root's directory name (relBase + prefix), gitignore-aware and reproducible — identical layout to a directory code_source_path. - include now selects subpaths of the code-source root (`path`), relative to it, matching the air CLI's include_paths (so convert-to-dabs can translate 1:1). - git mode adds --prefix=<dir>/ so git archive produces the same top-level layout. - Replace the bundle-root-relative field docs/test from the prior commit with the code-source-root semantics and a git-mode prefix test. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Disable core.autocrlf in the temp repo and compare trimmed content, so the content assertion doesn't depend on the Windows runner's autocrlf default. Co-authored-by: Isaac <no-reply@databricks.com>

ben-hansen-db

LGTM, nice work!

vinchenzo-db changed the title ~~[POC] Native tgz artifacts: build code_source from include paths / a …~~ [POC] DABs AIR native fields Aug 28, 2026

vinchenzo-db added 3 commits August 28, 2026 19:01

Merge remote-tracking branch 'origin/main' into air-artifacts-code-so…

eab9d13

…urce

Fix CI: lint (errors.New) + regenerate schema after merging main

f2a8fd9

- tarball.go: use errors.New for the no-args git error (perfsprint lint). - jsonschema.json: regenerated post-merge; picks up the variable-ref regex the newer generator emits (validate-generated). Co-authored-by: Isaac

vinchenzo-db requested review from ben-hansen-db and pietern August 28, 2026 20:11

vinchenzo-db mentioned this pull request Aug 31, 2026

Add tgz artifact type #6438

Closed

ben-hansen-db reviewed Aug 31, 2026

View reviewed changes

vinchenzo-db and others added 5 commits August 31, 2026 21:56

Fix lint: use t.Context() in tarball tests

1f344dc

Co-authored-by: Isaac <no-reply@databricks.com>

Fix Windows CI: make git tarball test EOL-agnostic

53dd3b3

Disable core.autocrlf in the temp repo and compare trimmed content, so the content assertion doesn't depend on the Windows runner's autocrlf default. Co-authored-by: Isaac <no-reply@databricks.com>

ben-hansen-db approved these changes Sep 1, 2026

View reviewed changes

Navigation Menu

Uh oh!

[POC] DABs AIR native fields - #6428

[POC] DABs AIR native fields#6428
vinchenzo-db wants to merge 9 commits into
mainfrom
air-artifacts-code-source

vinchenzo-db commented Aug 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Aug 28, 2026

Uh oh!

ben-hansen-db left a comment

Uh oh!

ben-hansen-db Aug 31, 2026

Uh oh!

vinchenzo-db Aug 31, 2026

Uh oh!

ben-hansen-db Aug 31, 2026

Uh oh!

vinchenzo-db Aug 31, 2026

Uh oh!

ben-hansen-db Aug 31, 2026

Uh oh!

ben-hansen-db Aug 31, 2026

Uh oh!

vinchenzo-db Aug 31, 2026

Uh oh!

Uh oh!

Uh oh!

ben-hansen-db left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

vinchenzo-db commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

github-actions Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approval status: pending

/acceptance/bundle/ - needs approval

/bundle/ - needs approval

Uh oh!

eng-dev-ecosystem-bot commented Aug 28, 2026

Integration test report

Uh oh!

ben-hansen-db left a comment

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

vinchenzo-db Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

vinchenzo-db Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

ben-hansen-db Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

vinchenzo-db Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ben-hansen-db left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vinchenzo-db commented Aug 28, 2026 •

edited

Loading

github-actions Bot commented Aug 28, 2026 •

edited

Loading

`/acceptance/bundle/` - needs approval

`/bundle/` - needs approval