Add opt-in Hugging Face .tok tokenizer integration

[JacobSzwejbka]

Summary

Depends on meta-pytorch/tokenizers#212.

This advances the meta-pytorch/tokenizers submodule to the commit from that draft PR and adds the minimal ExecuTorch integration for its optional Hugging Face .tok backend:

• add EXECUTORCH_BUILD_HF_RUST_TOKENIZER, defaulting to OFF
• forward that option and EXECUTORCH_OPTIMIZE_SIZE into the tokenizers subproject
• link tokenizers::hf_rust_tokenizer into the LLM runner only when enabled
• try the .tok loader before the existing JSON tokenizer fallback

The Rust implementation, C++ adapter, installation rules, and dependency ownership remain in meta-pytorch/tokenizers. With the option disabled, ExecuTorch does not invoke Cargo or link the optional archives, so the default tokenizer binary is unchanged.

This is a draft until the dependency lands; the submodule pointer will then be updated to the upstream commit.

Test plan

• git diff --check origin/main...HEAD
• default-off tokenizers CMake configure/build with Cargo absent from PATH
• opt-in GPT-2 .tok smoke test covering load, encode, token lookup, and incremental decode
• installed tokenizers package consumed with find_package(tokenizers CONFIG REQUIRED)
• tokenizer Rust checks in the dependency PR: cargo fmt --check, cargo clippy --locked --tests -- -D warnings, and cargo test --locked

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22548

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures

As of commit f0de270 with merge base c65b23a ():

NEW FAILURES - The following jobs have failed:

• Lint / lintrunner (gh)
  >>> Lint for CMakeLists.txt:
• pull / test-moshi-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 268c9cbbbd9b7c50a58a1ea3e9da01c1da4275881e47f259c8fa07931598cf6c /exec failed with exit code 2
• pull / unittest / linux / linux-job (gh)
  pytorch_tokenizers
• pull / unittest-buck / linux / linux-job (gh)
  pytorch_tokenizers
• pull / unittest-editable / linux / linux-job (gh)
  pytorch_tokenizers

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.