Support all features implemented in Python 3.12-3.15

.github/workflows/ci.yml

@@ -8,19 +8,20 @@
   push:
 
 jobs:
   build:
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.8", "3.11", "3.12"]  # no particular need for 3.9 or 3.10
+        python-version: ["3.8", "3.11", "3.12", "3.13", "3.14", "3.15"]
         os: [macos-latest, ubuntu-latest, windows-latest]
     steps:
     - uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871  # v4.2.1
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d  # v4.6.0
       with:
         python-version: ${{ matrix.python-version }}
+        allow-prereleases: true
     - name: Upgrade pip
       run: >-
         python -m pip install

README.md

@@ -351,9 +351,21 @@ optional arguments:
   --binary BINARY       location of binary to use for YAPF
 ```
 
-## Python features not yet supported
-* Python 3.12 – [PEP 695 – Type Parameter Syntax](https://peps.python.org/pep-0695/) – [YAPF #1170](https://github.com/google/yapf/issues/1170)
-* Python 3.12 – [PEP 701 – Syntactic formalization of f-strings](https://peps.python.org/pep-0701/) – [YAPF #1136](https://github.com/google/yapf/issues/1136)
+## Python syntax support
+
+* Python 3.12 - [PEP 695 - Type parameter syntax](https://peps.python.org/pep-0695/) - **Supported.** See [implementation notes](docs/pep-0695.md).
+* Python 3.13 - [PEP 696 - Type parameter defaults](https://peps.python.org/pep-0696/) - **Supported.** See [implementation notes](docs/pep-0696.md).
+* Python 3.12 - [PEP 701 - F-string syntax](https://peps.python.org/pep-0701/) - **Supported.** See [implementation notes](docs/pep-0701.md).
+* Python 3.14 - [PEP 750 - Template strings](https://peps.python.org/pep-0750/) - **Supported.** See [implementation notes](docs/pep-0750.md).
+* Python 3.14 - [PEP 758 - Unparenthesized exception lists](https://peps.python.org/pep-0758/) - **Supported.** See [implementation notes](docs/pep-0758.md).
+* Python 3.15 - [PEP 798 - Unpacking comprehensions](https://peps.python.org/pep-0798/) - **Supported.** See [implementation notes](docs/pep-0798.md).
+* Python 3.15 - [PEP 810 - Explicit lazy imports](https://peps.python.org/pep-0810/) - **Supported.** See [implementation notes](docs/pep-0810.md).
+
+Interpolated string literals, including replacement and debug expressions, are
+preserved verbatim; YAPF formats the surrounding code, not their contents.
+
+See [branch dependencies and validation](docs/pep-feature-branches.md) for the
+feature-branch layout and test limitations.
 
 ## Knobs
 

docs/pep-0695.md

@@ -0,0 +1,24 @@
+# PEP 695: Type parameter syntax
+
+Specification: https://peps.python.org/pep-0695/
+
+Language version: Python 3.12. Branch dependency: the shared grammar-cache isolation fix.
+
+## Supported formatting
+
+Type aliases, generic classes and functions, bounds, constraints, `*Ts`, and `**P`.
+
+New syntax is parsed by the vendored grammar/tokenizer, independently of the
+Python version running YAPF. Executing the formatted program still requires a
+compatible interpreter. YAPF is a formatter, not a type checker or a replacement
+for compiler-level semantic validation.
+
+## Regression tests
+
+```bash
+PYTHONPATH=third_party:. python -m unittest yapftests.pep695_test
+```
+
+Tests cover exact output, idempotence, style options, and relevant edge cases.
+Native AST/semantic checks run only on an interpreter supporting this PEP;
+formatter assertions run on older hosts too.

docs/pep-0696.md

@@ -0,0 +1,24 @@
+# PEP 696: Type parameter defaults
+
+Specification: https://peps.python.org/pep-0696/
+
+Language version: Python 3.13. Branch dependency: PEP 695.
+
+## Supported formatting
+
+Defaults for type variables, variadic type variables, and parameter specifications, including assignment spacing, comments, and wrapping.
+
+New syntax is parsed by the vendored grammar/tokenizer, independently of the
+Python version running YAPF. Executing the formatted program still requires a
+compatible interpreter. YAPF is a formatter, not a type checker or a replacement
+for compiler-level semantic validation.
+
+## Regression tests
+
+```bash
+PYTHONPATH=third_party:. python -m unittest yapftests.pep696_test
+```
+
+Tests cover exact output, idempotence, style options, and relevant edge cases.
+Native AST/semantic checks run only on an interpreter supporting this PEP;
+formatter assertions run on older hosts too.

docs/pep-0701.md

@@ -0,0 +1,29 @@
+# PEP 701: F-string grammar
+
+Specification: https://peps.python.org/pep-0701/
+
+Language version: Python 3.12. Branch dependency: the shared grammar-cache isolation fix.
+
+## Supported formatting
+
+Quote reuse, nesting, comments, backslashes, multiline replacement fields, token positions, and line-range formatting.
+
+New syntax is parsed by the vendored grammar/tokenizer, independently of the
+Python version running YAPF. Executing the formatted program still requires a
+compatible interpreter. YAPF is a formatter, not a type checker or a replacement
+for compiler-level semantic validation.
+
+Interpolated literals remain opaque STRING tokens. Literal text, replacement
+expressions, debug-expression whitespace, and template expression metadata are
+preserved verbatim. Only surrounding code is reformatted. The scanner identifies
+lexical boundaries; it does not validate arbitrary replacement expressions.
+
+## Regression tests
+
+```bash
+PYTHONPATH=third_party:. python -m unittest yapftests.pep701_test
+```
+
+Tests cover exact output, idempotence, style options, and relevant edge cases.
+Native AST/semantic checks run only on an interpreter supporting this PEP;
+formatter assertions run on older hosts too.

docs/pep-0750.md

@@ -0,0 +1,29 @@
+# PEP 750: Template strings
+
+Specification: https://peps.python.org/pep-0750/
+
+Language version: Python 3.14. Branch dependency: PEP 701.
+
+## Supported formatting
+
+All `t`/`T` and raw prefixes, every quote form, mixed f-string/t-string nesting, comments, backslashes, and multiline fields.
+
+New syntax is parsed by the vendored grammar/tokenizer, independently of the
+Python version running YAPF. Executing the formatted program still requires a
+compatible interpreter. YAPF is a formatter, not a type checker or a replacement
+for compiler-level semantic validation.
+
+Interpolated literals remain opaque STRING tokens. Literal text, replacement
+expressions, debug-expression whitespace, and template expression metadata are
+preserved verbatim. Only surrounding code is reformatted. The scanner identifies
+lexical boundaries; it does not validate arbitrary replacement expressions.
+
+## Regression tests
+
+```bash
+PYTHONPATH=third_party:. python -m unittest yapftests.pep750_test
+```
+
+Tests cover exact output, idempotence, style options, and relevant edge cases.
+Native AST/semantic checks run only on an interpreter supporting this PEP;
+formatter assertions run on older hosts too.

docs/pep-0758.md

@@ -0,0 +1,28 @@
+# PEP 758: Unparenthesized exception lists
+
+Specification: https://peps.python.org/pep-0758/
+
+Language version: Python 3.14. Branch dependency: the shared grammar-cache isolation fix.
+
+## Supported formatting
+
+Any number of exception types for `except` and `except*`, including a trailing comma; unparenthesized lists cannot bind an `as` target.
+
+New syntax is parsed by the vendored grammar/tokenizer, independently of the
+Python version running YAPF. Executing the formatted program still requires a
+compatible interpreter. YAPF is a formatter, not a type checker or a replacement
+for compiler-level semantic validation.
+
+An unparenthesized list with no explicit continuation remains on one physical
+line, even when over the column limit. No implicit parentheses or unsafe line
+breaks are introduced. Use an explicitly parenthesized list to enable wrapping.
+
+## Regression tests
+
+```bash
+PYTHONPATH=third_party:. python -m unittest yapftests.pep758_test
+```
+
+Tests cover exact output, idempotence, style options, and relevant edge cases.
+Native AST/semantic checks run only on an interpreter supporting this PEP;
+formatter assertions run on older hosts too.

docs/pep-0798.md

@@ -0,0 +1,24 @@
+# PEP 798: Unpacking comprehensions
+
+Specification: https://peps.python.org/pep-0798/
+
+Language version: Python 3.15. Branch dependency: the shared grammar-cache isolation fix.
+
+## Supported formatting
+
+List, set, dictionary, and generator unpacking; a generator used as the sole call argument; async comprehensions and conditional generator/mapping heads.
+
+New syntax is parsed by the vendored grammar/tokenizer, independently of the
+Python version running YAPF. Executing the formatted program still requires a
+compatible interpreter. YAPF is a formatter, not a type checker or a replacement
+for compiler-level semantic validation.
+
+## Regression tests
+
+```bash
+PYTHONPATH=third_party:. python -m unittest yapftests.pep798_test
+```
+
+Tests cover exact output, idempotence, style options, and relevant edge cases.
+Native AST/semantic checks run only on an interpreter supporting this PEP;
+formatter assertions run on older hosts too.

docs/pep-0810.md

@@ -0,0 +1,28 @@
+# PEP 810: Explicit lazy imports
+
+Specification: https://peps.python.org/pep-0810/
+
+Language version: Python 3.15. Branch dependency: the shared grammar-cache isolation fix.
+
+## Supported formatting
+
+Both lazy import forms, relative imports, aliases, parenthesized import lists, soft-keyword handling, and import/variable blank-line policy.
+
+New syntax is parsed by the vendored grammar/tokenizer, independently of the
+Python version running YAPF. Executing the formatted program still requires a
+compatible interpreter. YAPF is a formatter, not a type checker or a replacement
+for compiler-level semantic validation.
+
+Compiler restrictions on lazy imports (for example, function/class/try scopes,
+wildcard imports, and future imports) remain the responsibility of the Python
+compiler. Parsing by YAPF is not evidence that a program passes those checks.
+
+## Regression tests
+
+```bash
+PYTHONPATH=third_party:. python -m unittest yapftests.pep810_test
+```
+
+Tests cover exact output, idempotence, style options, and relevant edge cases.
+Native AST/semantic checks run only on an interpreter supporting this PEP;
+formatter assertions run on older hosts too.

docs/pep-feature-branches.md

@@ -0,0 +1,58 @@
+# PEP feature branches
+
+All changes are based on the uploaded repository at
+`12005095296072751e3e4c1f33a047d41b0ce18d`. `main` is unchanged.
+
+| Branch | Prerequisite |
+| --- | --- |
+| `fix/grammar-cache-isolation-20261009` | Original `main` |
+| `fix/soft-keyword-lookahead-20261009` | Cache isolation |
+| `feature/pep-695-type-parameters-20261009` | Cache isolation + lookahead |
+| `feature/pep-696-type-defaults-20261009` | PEP 695 |
+| `feature/pep-701-fstrings-20261009` | Cache isolation + lookahead |
+| `feature/pep-750-template-strings-20261009` | PEP 701 |
+| `feature/pep-758-exception-lists-20261009` | Cache isolation + lookahead |
+| `feature/pep-798-unpacking-comprehensions-20261009` | Cache isolation + lookahead |
+| `feature/pep-810-lazy-imports-20261009` | Cache isolation + lookahead |
+| `feature/python-syntax-integration-20261009` | Merges all seven PEP branches |
+
+Each PEP branch has its own implementation, regression tests, and notes. The
+integration branch retains merge ancestry, combines the two contextual keyword
+checks in `FormatToken.is_keyword`, adds cross-feature tests, and extends CI/tox
+coverage through Python 3.15. It does not squash or replace the feature branches.
+For isolated reviews, target PEP 696 at PEP 695, and PEP 750 at PEP 701.
+
+## Review follow-up
+
+The original feature commits are retained. Every PEP branch now also includes
+`fix/soft-keyword-lookahead-20261009`, which reimplements the shared token buffer
+to fix nested speculative parsing and quadratic scanning of old release records.
+The new `type` and `lazy` soft keywords exposed those defects: for example,
+`match(type)` parsed on original `main` but failed after the PEP additions.
+Nine regression test methods cover stream invariants and 68 syntax subcases.
+See [the lookahead design note](soft-keyword-lookahead.md).
+
+No PEP grammar or interpolation scanner was replaced merely for stylistic reasons.
+The reviewed integration preserves the original seven implementation branches,
+and the follow-up commits are fast-forward descendants of their prior heads.
+
+The shared cache fix fingerprints the grammar source. Without it, checkouts with
+the same YAPF/interpreter version can reuse a different branch's cached grammar.
+
+## Validation
+
+Local execution uses CPython 3.13.5 on Linux. Formatter tests run on every host;
+native AST comparisons and native semantic tests for newer syntax run only when
+the interpreter supports it. Python 3.12/3.14/3.15 downloads were unavailable in
+the implementation environment. CI configuration is supplied but was not run on
+GitHub; Windows and macOS are not locally validated.
+
+```bash
+PYTHONPATH=third_party:. python -m unittest discover -p '*_test.py' yapftests/
+```
+
+This is syntax-formatting support, not type checking or complete compiler
+validation. F-string and t-string interiors are preserved, not reformatted.
+Long unparenthesized exception lists remain on one physical line. The separate
+Unicode-identifier limitation from the earlier audit is not addressed by these
+PEP branches. See each PEP's implementation note for scope details.

docs/soft-keyword-lookahead.md

@@ -0,0 +1,29 @@
+# Soft-keyword lookahead isolation
+
+The PEP 695 and PEP 810 grammars add the contextual keywords `type` and `lazy`.
+That exposed two problems in the existing speculative token reader:
+
+* Nested lookahead could attach tokens to the wrong release record. Valid code
+  such as `match(type)` and `match (type):\n case _: pass` then failed to parse.
+* Release records survived until the end of the file, and every committed token
+  scanned every earlier record. Ordinary repeated `type(value)` calls therefore
+  made parsing quadratic rather than linear in the length of the token stream.
+
+`TokenProxy` now uses one absolute-position buffer for unread tokens and a stack
+of independent speculative cursors. Nested routes start immediately after the
+parent's currently observed token. Exiting a route restores its parent without
+consuming input. Committed tokens are removed from the buffer immediately.
+This bounds token storage by outstanding lookahead rather than by the number of
+soft keywords already parsed, and makes buffered token access constant-time.
+The parser's existing grammar-selection algorithm is unchanged.
+
+Tests cover replay, competing and nested routes, exceptions, end of input,
+negative offsets, prompt removal of consumed tokens, and AST-preserving
+formatting of ambiguous calls, subscripts, and match subjects. Buffer lifetime
+is asserted structurally rather than with a machine-dependent timing threshold.
+The implementation and tests remain compatible with YAPF's Python 3.7 minimum;
+native AST comparisons for match statements require Python 3.10 or later.
+
+```bash
+PYTHONPATH=third_party:. python -m unittest yapftests.token_proxy_test
+```