From f61b5ca106c303d2e246fd8d752854588f82075e Mon Sep 17 00:00:00 2001 From: Grigory Panov Date: Thu, 27 Aug 2026 22:31:32 +0200 Subject: [PATCH 1/3] Document conservative DB Connect / serverless env upgrade rule Follow-up to #6378. Tightens the comment above conservative_db_connect_version_spec into crisp upgrade guidance, and adds an .agents/rules doc capturing the conservative DB Connect upgrade rule and the env-version / Python / DB Connect consistency invariant so the next serverless environment bump is smoother. Co-authored-by: Isaac --- .agents/rules/template-versions.md | 19 ++++++++++++++++ .cursor/rules/template-versions.mdc | 1 + .../templates/default/library/versions.tmpl | 22 ++++++++++--------- 3 files changed, 32 insertions(+), 10 deletions(-) create mode 100644 .agents/rules/template-versions.md create mode 120000 .cursor/rules/template-versions.mdc diff --git a/.agents/rules/template-versions.md b/.agents/rules/template-versions.md new file mode 100644 index 00000000000..041d4c917e7 --- /dev/null +++ b/.agents/rules/template-versions.md @@ -0,0 +1,19 @@ +--- +description: Version pins in bundle template versions.tmpl files (DBR, DB Connect, serverless env, Python) +globs: + - "libs/template/templates/**/library/versions.tmpl" +paths: + - "libs/template/templates/**/library/versions.tmpl" +--- + +# Bundle template version pins + +Several bundle templates pin their runtime versions in `library/versions.tmpl`: `default` pins the most (DBR, DB Connect, serverless environment, Python, and the `databricks-bundles` package), while `dbt-sql` and `default-sql` pin subsets. The `.tmpl` files are the source of truth for the current values — the rules below describe how the pins relate, not what they are. (`default-scala` pins its versions in `library/template_variables.tmpl` instead and is out of scope here.) + +**RULE: Keep the serverless environment version, the Python pins, and the DB Connect pin mutually compatible. This is a hard constraint.** A serverless environment version dictates a runtime Python version, and the DB Connect pin must support that Python. Cross-check the [serverless environment version release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/) and the [DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements) whenever you change any of the three. + +**RULE: Within that constraint, keep `conservative_db_connect_version_spec` as low as compatibility allows.** The DB Connect client is only forward-compatible (it reaches compute of its own version and higher), so the lowest compatible pin maximizes the range of DBR versions a customer can connect to; a higher pin rules out customers on older DBR. Do not raise it merely to match the newest serverless environment — raise it only when compatibility forces it, or when the pinned DBR release falls out of support (see the [DBR release notes](https://docs.databricks.com/aws/en/release-notes/runtime/) for supported versions). Customers can move to a newer version themselves after initializing the template. See PR #3897 and PR #6378 for prior history. + +**RULE: `serverless_environment_version` must stay in sync across every template that defines it — bump them together.** Other macros are defined per-template and may legitimately differ (the SQL templates, for example, currently pin an older `latest_lts_dbr_version` than `default`). Do not sync a value across templates just because a macro name matches: check each template's intended value and change only the ones that should move, for the same reason. + +Changing a version pin changes rendered template output, so regenerate the acceptance goldens afterward (see [auto-generated-files.md](auto-generated-files.md)). diff --git a/.cursor/rules/template-versions.mdc b/.cursor/rules/template-versions.mdc new file mode 120000 index 00000000000..7ad1818ffe4 --- /dev/null +++ b/.cursor/rules/template-versions.mdc @@ -0,0 +1 @@ +../../.agents/rules/template-versions.md \ No newline at end of file diff --git a/libs/template/templates/default/library/versions.tmpl b/libs/template/templates/default/library/versions.tmpl index 2c58247bc6c..7acdb93df43 100644 --- a/libs/template/templates/default/library/versions.tmpl +++ b/libs/template/templates/default/library/versions.tmpl @@ -5,18 +5,20 @@ 16.4.x-scala2.12 {{- end}} -{{/* A conservative version of DB Connect for local development. +{{/* A conservative DB Connect pin for local development. * - * DB Connect is only forward-compatible: a client connects to compute of the - * same version and higher. We keep this conservative (low) so a freshly - * initialized project can reach the widest range of DBR versions; a high - * version would rule out customers on older DBR. We use 16.4 rather than an - * older release because DBR 15 is no longer supported and used Python 3.11, - * whereas 16.4 uses Python 3.12 (matching the serverless environment and the - * template's Python pin). Customers can move to a newer version themselves - * after initializing the template. + * The DB Connect client is only forward-compatible (it reaches compute of its + * own version and higher), so keep this at the LOWEST version that still + * works: bump it only when the pinned DBR release falls out of support, never + * to match the newest serverless environment version. A higher pin rules out + * customers on older DBR; they can upgrade themselves after init. * - * See https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements + * Current floor is 16.4 (DBR 15 is out of support and ran Python 3.11; 16.4 + * runs Python 3.12, matching serverless env 5 and python_version_spec). + * + * See .agents/rules/template-versions.md for the full upgrade rule and the + * env-version / Python / DB Connect consistency invariant, and + * https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements * for DB Connect release notes and version compatibility. */}} {{define "conservative_db_connect_version_spec" -}} From 56ed261d5eb79a4771f66601d88fe84a04404347 Mon Sep 17 00:00:00 2001 From: Grigory Panov Date: Wed, 2 Sep 2026 14:36:50 +0200 Subject: [PATCH 2/3] Address review: correct stale facts and reconcile the rule with the comment - Drop the claims about default-sql's versions.tmpl and the SQL templates pinning an older DBR, both invalidated by #6420 and #6418 - Attribute the 16.4 DB Connect floor to the Python/serverless-env compatibility constraint instead of DBR 15's support status - Remove the comment's absolute "never bump to match the newest serverless environment", which contradicted the hard compatibility constraint - Extend the sync rule and the globs to the hardcoded environment_version and requires-python occurrences outside library/versions.tmpl - Shrink the versions.tmpl comment to the invariant plus a pointer, so the rule has a single home - Reference the rule file by repo-root path and name ./task test-update-templates Co-authored-by: Isaac --- .agents/rules/template-versions.md | 18 ++++++++++++----- .../templates/default/library/versions.tmpl | 20 +++++++++---------- 2 files changed, 22 insertions(+), 16 deletions(-) diff --git a/.agents/rules/template-versions.md b/.agents/rules/template-versions.md index 041d4c917e7..a55973eff5d 100644 --- a/.agents/rules/template-versions.md +++ b/.agents/rules/template-versions.md @@ -1,19 +1,27 @@ --- -description: Version pins in bundle template versions.tmpl files (DBR, DB Connect, serverless env, Python) +description: Version pins in bundle templates (DBR, DB Connect, serverless env, Python) globs: - "libs/template/templates/**/library/versions.tmpl" + - "libs/template/templates/**/library/template_variables.tmpl" + - "libs/template/templates/**/resources/*.job.yml.tmpl" + - "libs/template/templates/**/pyproject.toml.tmpl" paths: - "libs/template/templates/**/library/versions.tmpl" + - "libs/template/templates/**/library/template_variables.tmpl" + - "libs/template/templates/**/resources/*.job.yml.tmpl" + - "libs/template/templates/**/pyproject.toml.tmpl" --- # Bundle template version pins -Several bundle templates pin their runtime versions in `library/versions.tmpl`: `default` pins the most (DBR, DB Connect, serverless environment, Python, and the `databricks-bundles` package), while `dbt-sql` and `default-sql` pin subsets. The `.tmpl` files are the source of truth for the current values — the rules below describe how the pins relate, not what they are. (`default-scala` pins its versions in `library/template_variables.tmpl` instead and is out of scope here.) +Bundle templates pin the runtime versions they render. Most pins are macros in a template's `library/versions.tmpl` — `default` defines the most (DBR, DB Connect, serverless environment, Python, and the `databricks-bundles` package) — while `default-scala` defines its own in `library/template_variables.tmpl`, and some templates hardcode a version in the rendered file instead of going through a macro. The `.tmpl` files are the source of truth for the current values; the rules below describe how the pins relate, not what they are. **RULE: Keep the serverless environment version, the Python pins, and the DB Connect pin mutually compatible. This is a hard constraint.** A serverless environment version dictates a runtime Python version, and the DB Connect pin must support that Python. Cross-check the [serverless environment version release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/) and the [DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements) whenever you change any of the three. -**RULE: Within that constraint, keep `conservative_db_connect_version_spec` as low as compatibility allows.** The DB Connect client is only forward-compatible (it reaches compute of its own version and higher), so the lowest compatible pin maximizes the range of DBR versions a customer can connect to; a higher pin rules out customers on older DBR. Do not raise it merely to match the newest serverless environment — raise it only when compatibility forces it, or when the pinned DBR release falls out of support (see the [DBR release notes](https://docs.databricks.com/aws/en/release-notes/runtime/) for supported versions). Customers can move to a newer version themselves after initializing the template. See PR #3897 and PR #6378 for prior history. +**RULE: Within that constraint, keep `conservative_db_connect_version_spec` as low as compatibility allows.** The DB Connect client is only forward-compatible (it reaches compute of its own version and higher), so the lowest compatible pin maximizes the range of DBR versions a customer can connect to; a higher pin rules out customers on older DBR, and they can move to a newer version themselves after initializing the template. Today the binding constraint is Python: the pin sits at 16.4 because that is the lowest DB Connect release whose Python (3.12) satisfies `python_version_spec` and the pinned serverless environment. Do not raise the pin merely to match a newer serverless environment version — raise it only when the compatibility constraint above forces it, or when the pinned DBR release falls out of support (see the [DBR release notes](https://docs.databricks.com/aws/en/release-notes/runtime/) for supported versions). See PR #3897 and PR #6378 for prior history. -**RULE: `serverless_environment_version` must stay in sync across every template that defines it — bump them together.** Other macros are defined per-template and may legitimately differ (the SQL templates, for example, currently pin an older `latest_lts_dbr_version` than `default`). Do not sync a value across templates just because a macro name matches: check each template's intended value and change only the ones that should move, for the same reason. +**RULE: When bumping the serverless environment version, update every occurrence, not just the macros.** More than one template defines `serverless_environment_version` and those definitions must stay in sync, but several templates hardcode `environment_version` (and `requires-python`) in the files they render instead — grep both under `libs/template/templates/` and update every hit together. -Changing a version pin changes rendered template output, so regenerate the acceptance goldens afterward (see [auto-generated-files.md](auto-generated-files.md)). +**RULE: Do not sync a value across templates just because a macro name matches.** Outside the serverless environment version, a pin is a per-template choice that may legitimately differ, so check each template's intended value and change only the ones that should move. + +Version pins are rendered into the materialized templates checked into `acceptance/`, so regenerate them with `./task test-update-templates` after changing one (see `.agents/rules/auto-generated-files.md`). diff --git a/libs/template/templates/default/library/versions.tmpl b/libs/template/templates/default/library/versions.tmpl index 7acdb93df43..511e17cd38b 100644 --- a/libs/template/templates/default/library/versions.tmpl +++ b/libs/template/templates/default/library/versions.tmpl @@ -5,19 +5,17 @@ 16.4.x-scala2.12 {{- end}} -{{/* A conservative DB Connect pin for local development. +{{/* A conservative version of DB Connect for local development. * - * The DB Connect client is only forward-compatible (it reaches compute of its - * own version and higher), so keep this at the LOWEST version that still - * works: bump it only when the pinned DBR release falls out of support, never - * to match the newest serverless environment version. A higher pin rules out - * customers on older DBR; they can upgrade themselves after init. + * DB Connect is only forward-compatible: a client connects to compute of the + * same version and higher. We keep this conservative (low) so a freshly + * initialized project can reach the widest range of DBR versions; a high + * version would rule out customers on older DBR. Customers can move to a + * newer version themselves after initializing the template. This pin must + * stay compatible with python_version_spec and serverless_environment_version + * below. * - * Current floor is 16.4 (DBR 15 is out of support and ran Python 3.11; 16.4 - * runs Python 3.12, matching serverless env 5 and python_version_spec). - * - * See .agents/rules/template-versions.md for the full upgrade rule and the - * env-version / Python / DB Connect consistency invariant, and + * See .agents/rules/template-versions.md before changing this, and * https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements * for DB Connect release notes and version compatibility. */}} From 0ec3578ac162cae21f9e008a9db5ac11ab5a9267 Mon Sep 17 00:00:00 2001 From: Grigory Panov Date: Wed, 2 Sep 2026 17:14:53 +0200 Subject: [PATCH 3/3] Review round 3: fix the DB Connect floor rationale and scope Rule 2 - The floor is the lowest release that is both still supported (in practice the DBR LTS) and Python-compatible. Attributing it to Python alone was wrong: a whole DBR major line shares one Python version, so 16.0-16.3 also satisfy python_version_spec and are lower than the pin - Drop the 16.4/3.12 literals so the rule stops restating current values - Explain the upper bound: widening it resolves a much newer client than the floor advertises - Scope Rule 2 to default's macro and state default-scala's opposite policy (tracks the newest LTS, derives an open-ended client from build.sbt.tmpl) - Add latest_lts_dbr_version to the compatibility set; it drifted from the DB Connect pin between #3671 and #6378 - Sync rule now covers default_python_version (notebook kernel version) and names the two hardcoding templates instead of saying "several" - Point at ./task test-update: acceptance/pipelines renders the pins too, so test-update-templates leaves those goldens stale - Note the alias templates (template_dir ../default) that share default's pins - Fix the serverless_environment_version comment to link the environment version release notes instead of the DBR runtime notes Co-authored-by: Isaac --- .agents/rules/template-versions.md | 12 +++++++----- .../template/templates/default/library/versions.tmpl | 3 ++- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/.agents/rules/template-versions.md b/.agents/rules/template-versions.md index a55973eff5d..9f0aeaa9471 100644 --- a/.agents/rules/template-versions.md +++ b/.agents/rules/template-versions.md @@ -14,14 +14,16 @@ paths: # Bundle template version pins -Bundle templates pin the runtime versions they render. Most pins are macros in a template's `library/versions.tmpl` — `default` defines the most (DBR, DB Connect, serverless environment, Python, and the `databricks-bundles` package) — while `default-scala` defines its own in `library/template_variables.tmpl`, and some templates hardcode a version in the rendered file instead of going through a macro. The `.tmpl` files are the source of truth for the current values; the rules below describe how the pins relate, not what they are. +Bundle templates pin the runtime versions they render. Most pins are macros in `libs/template/templates/default/library/versions.tmpl` (DBR, DB Connect, serverless environment, Python, and the `databricks-bundles` package), and that single file serves five shipped templates: `default-minimal`, `default-python`, `lakeflow-pipelines` and `pydabs` all alias it through `"template_dir": "../default"` in their `databricks_template_schema.json`. `dbt-sql` defines its own subset, `default-scala` defines its own in `library/template_variables.tmpl`, and two templates hardcode a version in the files they render instead of going through a macro. The `.tmpl` files are the source of truth for the current values; the rules below describe how the pins relate, not what they are. -**RULE: Keep the serverless environment version, the Python pins, and the DB Connect pin mutually compatible. This is a hard constraint.** A serverless environment version dictates a runtime Python version, and the DB Connect pin must support that Python. Cross-check the [serverless environment version release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/) and the [DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements) whenever you change any of the three. +**RULE: Keep the serverless environment version, the Python pins, the DB Connect pin, and `latest_lts_dbr_version` mutually compatible. This is a hard constraint.** A serverless environment version dictates a runtime Python version, and the DB Connect pin must support that Python. The DBR LTS pin belongs in the same check rather than being treated as independent — the two have drifted before: `latest_lts_dbr_version` reached 16.4 in #3671 while the DB Connect pin stayed on 15.4 until #6378. Cross-check the [serverless environment version release notes](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/) and the [DB Connect requirements](https://docs.databricks.com/dev-tools/databricks-connect/python/index.html#requirements) whenever you change any of the four. -**RULE: Within that constraint, keep `conservative_db_connect_version_spec` as low as compatibility allows.** The DB Connect client is only forward-compatible (it reaches compute of its own version and higher), so the lowest compatible pin maximizes the range of DBR versions a customer can connect to; a higher pin rules out customers on older DBR, and they can move to a newer version themselves after initializing the template. Today the binding constraint is Python: the pin sits at 16.4 because that is the lowest DB Connect release whose Python (3.12) satisfies `python_version_spec` and the pinned serverless environment. Do not raise the pin merely to match a newer serverless environment version — raise it only when the compatibility constraint above forces it, or when the pinned DBR release falls out of support (see the [DBR release notes](https://docs.databricks.com/aws/en/release-notes/runtime/) for supported versions). See PR #3897 and PR #6378 for prior history. +**RULE: Within that constraint, keep `default`'s `conservative_db_connect_version_spec` as low as compatibility allows.** The DB Connect client is only forward-compatible (it reaches compute of its own version and higher), so the lowest compatible pin maximizes the range of DBR versions a customer can connect to; a higher pin rules out customers on older DBR, and they can move to a newer version themselves after initializing the template. The floor is the lowest release that is *both* still supported — in practice the current DBR LTS, since the non-LTS minor releases below it fall out of support first — *and* whose Python satisfies `python_version_spec`. Python alone does not determine the floor: a whole DBR major line typically shares one Python version, so several releases below the LTS will also satisfy the Python pin. Keep the upper bound at the next minor release, so the spec stays inside one minor: widening it lets a freshly initialized project resolve a client far newer than the floor advertises, which is the opposite of conservative. Do not raise the pin merely to match a newer serverless environment version — raise it only when the constraint above forces it, or when the pinned release falls out of support (see the [DBR release notes](https://docs.databricks.com/aws/en/release-notes/runtime/) for supported versions). See PR #3897 and PR #6378 for prior history. -**RULE: When bumping the serverless environment version, update every occurrence, not just the macros.** More than one template defines `serverless_environment_version` and those definitions must stay in sync, but several templates hardcode `environment_version` (and `requires-python`) in the files they render instead — grep both under `libs/template/templates/` and update every hit together. +This rule governs `default`'s pin. `default-scala` deliberately does the opposite: it tracks the newest LTS in `dbr_version` and derives its client from that macro in `template/{{.project_name}}/build.sbt.tmpl` as an open-ended `.+` dependency. Do not lower `dbr_version`, or pin that dependency, to satisfy the rule above. + +**RULE: When bumping the serverless environment version or the Python version, update every occurrence, not just the macros.** More than one template defines `serverless_environment_version`, and those definitions must stay in sync. Two templates hardcode the value in the files they render instead — `default-scala` in one job and `lakeflow-integrations` in two — and `lakeflow-integrations` also hardcodes `requires-python`. A Python bump additionally needs `default_python_version`, which supplies the notebook kernel version in `src/sample_notebook.ipynb.tmpl`. Grep `environment_version`, `requires-python` and `default_python_version` under `libs/template/templates/` and move every hit together. **RULE: Do not sync a value across templates just because a macro name matches.** Outside the serverless environment version, a pin is a per-template choice that may legitimately differ, so check each template's intended value and change only the ones that should move. -Version pins are rendered into the materialized templates checked into `acceptance/`, so regenerate them with `./task test-update-templates` after changing one (see `.agents/rules/auto-generated-files.md`). +Version pins are rendered into the materialized templates checked into `acceptance/`, and not all of those live under `acceptance/bundle/templates/` — `acceptance/pipelines/` renders the pins too. Regenerate with `./task test-update`, not `./task test-update-templates`, which covers only the `bundle/templates` subtree. See `.agents/rules/testing.md` and `.agents/rules/auto-generated-files.md`. diff --git a/libs/template/templates/default/library/versions.tmpl b/libs/template/templates/default/library/versions.tmpl index 511e17cd38b..d7f89eb65e3 100644 --- a/libs/template/templates/default/library/versions.tmpl +++ b/libs/template/templates/default/library/versions.tmpl @@ -24,7 +24,8 @@ {{- end}} {{/* The serverless environment version to use. - * See https://docs.databricks.com/release-notes/runtime/index.html for available versions. + * See https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/ + * for available versions. */}} {{define "serverless_environment_version" -}} 5