Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
18 changes: 18 additions & 0 deletions acceptance/bin/print_requests.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,15 @@
ADD_PREFIX = "/"
NEGATE_PREFIX = "^/"

# What --nostamp deletes. A job create sends the deployment block at the top level, an
# update nests it under new_settings.
STAMP_FIELDS = [
"deployment.deployment_id",
"deployment.version_id",
"new_settings.deployment.deployment_id",
"new_settings.deployment.version_id",
]


def read_json_many(s):
result = []
Expand Down Expand Up @@ -239,10 +248,19 @@ def main():
"--del-body, which edits the parsed JSON body, this drops a field of the request "
"record itself, e.g. raw_body for a binary upload payload.",
)
parser.add_argument(
"--nostamp",
action="store_true",
help="Drop the deployment stamp (deployment_id, version_id) from job and pipeline "
"bodies, so a test asserts the same requests whether or not deployment history "
"recording is on. Shorthand for the --del-body fields it implies.",
)
parser.add_argument("--fname", default="out.requests.txt")
args = parser.parse_args()

del_body_fields = [field for group in args.del_body for field in group.split(",")]
if args.nostamp:
del_body_fields += STAMP_FIELDS
del_fields = [field for group in args.del_field for field in group.split(",")]

test_tmp_dir = os.environ.get("TEST_TMP_DIR")
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ src/train.py

=== both code_source_paths point into the bundle .air_snapshots, command_paths rewritten, deps on environments spec

>>> print_requests.py --sort --del-field raw_body //.air_snapshots/ //jobs/create
>>> print_requests.py --nostamp --sort --del-field raw_body //.air_snapshots/ //jobs/create
{
"method": "POST",
"path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/ai-runtime-test/default/files/.air_snapshots/[SNAPSHOT].tar.gz",
Expand Down Expand Up @@ -114,7 +114,7 @@ Updated jobs.train
Files: 4 uploaded, 1 deleted
Resources: 0 created, 1 changed, 0 deleted, 0 unchanged

>>> print_requests.py --sort --del-field raw_body //.air_snapshots/
>>> print_requests.py --nostamp --sort --del-field raw_body //.air_snapshots/
{
"method": "POST",
"path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/ai-runtime-test/default/files/.air_snapshots/[SNAPSHOT].tar.gz",
Expand Down
6 changes: 3 additions & 3 deletions acceptance/bundle/ai_runtime_task/local_code_source/script
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,16 @@ trace list_code_snapshot.py
title "both code_source_paths point into the bundle .air_snapshots, command_paths rewritten, deps on environments spec\n"
# --del-field raw_body drops the binary tarball upload payload (kept readable). Filters
# use a leading // so Git Bash on Windows does not path-convert them. --keep is not
# passed, so print_requests.py consumes out.requests.txt.
trace print_requests.py --sort --del-field raw_body '//.air_snapshots/' '//jobs/create'
# passed, so print_requests.py --nostamp consumes out.requests.txt.
trace print_requests.py --nostamp --sort --del-field raw_body '//.air_snapshots/' '//jobs/create'

title "re-planning unchanged code is a no-op (no changes)\n"
trace $CLI bundle plan

title "editing a file changes the snapshot hash (content-addressed name changes)\n"
update_file.py src/train.py 'print("training")' 'print("training v2")'
trace $CLI bundle deploy
trace print_requests.py --sort --del-field raw_body '//.air_snapshots/'
trace print_requests.py --nostamp --sort --del-field raw_body '//.air_snapshots/'

title "destroy removes the deployed bundle (including the synced snapshots)\n"
trace $CLI bundle destroy --auto-approve
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/apps/app_yaml/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/apps/compute_size/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/apps/delete_deleting/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/apps/git_source/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/apps/job_permissions/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/apps/value_from_warning/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/build_and_files/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/glob_exact_whl/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/globs_in_files/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/globs_invalid/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/issue_3109/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/nil_artifacts/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/bash/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/basic/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/cmd/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/default/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/err-bash/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/err-sh/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/invalid/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/shell/sh/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/whl_dbfs/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/whl_dynamic/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions acceptance/bundle/artifacts/whl_dynamic/script
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ cp -r $TESTDIR/../whl_explicit/my_test_code .
mkdir prebuilt
cp -r $TESTDIR/../whl_prebuilt_multiple/dist/lib/other_test_code-0.0.1-py3-none-any.whl prebuilt

trace $CLI bundle validate -o json | jq .artifacts
trace $CLI bundle validate -o json | nostamp | jq .artifacts

$CLI bundle plan -o json > out.plan_create.$DATABRICKS_BUNDLE_ENGINE.json
$CLI bundle plan -o json | nostamp > out.plan_create.$DATABRICKS_BUNDLE_ENGINE.json
trace $CLI bundle deploy

title "There are 2 original wheels and 2 patched ones"
Expand All @@ -25,7 +25,7 @@ rm out.requests.txt

title "Updating the local wheel and deploying again\n"
touch my_test_code/src/new_module.py
$CLI bundle plan -o json > out.plan_update.$DATABRICKS_BUNDLE_ENGINE.json
$CLI bundle plan -o json | nostamp > out.plan_update.$DATABRICKS_BUNDLE_ENGINE.json
trace $CLI bundle deploy

title "Verify contents, it should now have new_module.py"
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/whl_explicit/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/whl_implicit/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ trace $CLI bundle deploy
trace find.py --expect 1 whl

title "Expecting 1 wheel in libraries section in /jobs/create"
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body' out.requests.txt
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body' out.requests.txt | nostamp

title "Expecting 1 wheel to be uploaded"
trace jq .path < out.requests.txt | grep import | grep whl | sort
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/whl_multiple/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/artifacts/whl_no_cleanup/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion acceptance/bundle/artifacts/whl_via_environment_key/script
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ trace $CLI bundle deploy
trace find.py --expect 1 whl

title "Expecting 1 wheel in environments section in /jobs/create"
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body' out.requests.txt
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body' out.requests.txt | nostamp

title "Expecting 1 wheel to be uploaded"
trace jq .path < out.requests.txt | grep import | grep whl | sort
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ trace $CLI bundle deploy
trace find.py --expect 1 whl

title "Expecting the environments dependency to keep its [train] extras suffix"
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body' out.requests.txt
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body' out.requests.txt | nostamp

title "Expecting 1 wheel to be uploaded under its bare filename (no extras)"
trace jq .path < out.requests.txt | grep import | grep whl | sort
Expand Down
1 change: 1 addition & 0 deletions acceptance/bundle/benchmarks/deploy/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/benchmarks/plan/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/benchmarks/validate/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions acceptance/bundle/bundle_tag/id/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions acceptance/bundle/bundle_tag/id/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Created jobs.foo
Files: 5 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged

>>> print_requests.py //jobs
>>> print_requests.py --nostamp //jobs
{
"method": "POST",
"path": "/api/2.2/jobs/create",
Expand Down Expand Up @@ -78,7 +78,7 @@ All files and directories at the following location will be deleted: /Workspace/

Destroy: 1 deleted

>>> print_requests.py //jobs
>>> print_requests.py --nostamp //jobs
{
"method": "POST",
"path": "/api/2.2/jobs/delete",
Expand Down
6 changes: 3 additions & 3 deletions acceptance/bundle/bundle_tag/id/script
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
trace $CLI bundle validate
trace $CLI bundle validate -o json | jq .resources
trace $CLI bundle validate -o json | nostamp | jq .resources
trace $CLI bundle plan
trace $CLI bundle deploy
trace print_requests.py //jobs
trace print_requests.py --nostamp //jobs
trace $CLI bundle summary
trace $CLI bundle destroy --auto-approve
trace print_requests.py //jobs
trace print_requests.py --nostamp //jobs
1 change: 1 addition & 0 deletions acceptance/bundle/bundle_tag/url/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions acceptance/bundle/bundle_tag/url/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Created jobs.foo
Files: 5 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged

>>> print_requests.py //jobs
>>> print_requests.py --nostamp //jobs
{
"method": "POST",
"path": "/api/2.2/jobs/create",
Expand Down Expand Up @@ -78,7 +78,7 @@ All files and directories at the following location will be deleted: /Workspace/

Destroy: 1 deleted

>>> print_requests.py //jobs
>>> print_requests.py --nostamp //jobs
{
"method": "POST",
"path": "/api/2.2/jobs/delete",
Expand Down
Loading
Loading