Skip to content

Add standalone OpenVINO EPContext compilation support - #2732

Open
Xiaoyu Z (xiaoyu-work) wants to merge 3 commits into
mainfrom
feature/openvino-ep-context-binary
Open

Xiaoyu Z (xiaoyu-work) wants to merge 3 commits into
mainfrom
feature/openvino-ep-context-binary

Conversation

@xiaoyu-work

@xiaoyu-work Xiaoyu Z (xiaoyu-work) commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Describe your changes

Extract the OpenVINO plugin EP context-compilation support from #2717 into a standalone change to EPContextBinaryGenerator. This PR intentionally excludes OpenVINOModelPackage, package manifests, shared assets, and multi-device package assembly.

  • Dispatch explicitly to _run_qnn and _run_openvino; preserve QNN's existing session-based compilation, composite/weight-sharing behavior, metadata, and optional GenAI config updates.
  • Add OpenVINO plugin registration, accelerator-type/device filtering, virtual cross-compilation targets, AOT via ModelCompiler, and JIT via InferenceSession.
  • Correct the extracted OpenVINO path to honor embed_context, disable_cpu_fallback, and typed logging options. Parse JSON boolean/compiler enum values correctly and avoid redundant AOT session entries already controlled by the compile API.
  • Preserve registration ownership, keep context sidecars in the output resource, and reject missing EPContext nodes or referenced binaries. Validate protobuf attributes directly so embedded binary data is not incorrectly decoded as UTF-8.
  • Document standalone usage, compiler options, and current single-ONNX-model limitations.

Review fixes

  • Discover the installed OpenVINO plugin in the existing EP bootstrap before Windows LocalSystem provider validation, preserving explicitly configured library paths. Include a regression through the full workflow entry point with a virtual NPU target and no physical NPU.
  • Compare hexadecimal device IDs numerically, accepting case differences, an optional 0x prefix, leading zeros, and integer IDs.
  • Clean the failed invocation's context ONNX and matching binaries after compilation or validation failure, allowing retries at the same output path. Keep existing successful outputs and unrelated files protected; no staging/publish framework was added.

Validation

  • & .\.venv\Scripts\python.exe -m pytest test\passes\onnx\test_context_binary.py test\common\test_ort_inference.py test\systems\test_local.py -q — passed: 72 tests; 4 QNN hardware tests skipped because QNNExecutionProvider is unavailable.
  • lintrunner olive\common\ort_inference.py olive\passes\onnx\context_binary.py test\common\test_ort_inference.py test\passes\onnx\test_context_binary.py docs\source\reference\pass.rst — passed, including PYLINT.
  • Additional real integration checks using ONNX Runtime 1.28.0 and OpenVINO EP plugin 1.7.0: ran the complete Olive workflow from a fresh process without manual EP registration, using a generated external-weight MatMul/Relu ONNX model. Virtual NPU platform 5010 and physical CPU both passed AOT/JIT with embedded/external context data (8 combinations). The NPU workflow used device_id: 0x00B03E.
  • Injected a transient failure after real context files were generated, then reran the unchanged workflow successfully at the identical cache output path. All four CPU workflow outputs reloaded and produced correct inference results with the original model/weights unavailable.
  • Full repository suite and on-device NPU/GPU inference were not run; the plugin exposed only a physical CPU on this machine.

Release note: EPContextBinaryGenerator now supports standalone OpenVINO plugin EP compilation through ORT's AOT/JIT APIs, including automatic Windows LocalSystem plugin discovery, virtual target selection, embedded/external context data, and clean retries after failed compilation.

Checklist before requesting a review

  • Add unit tests for this change.
  • Make sure all tests can pass.
  • Update documents if necessary.
  • Lint and apply fixes to your code by running lintrunner -a
  • Is this a user-facing change? If yes, give a description of this change to be included in the release notes.

(Optional) Issue link

Extracted compilation support from #2717; the original packaging PR remains separate.

Copilot AI balanced review requested due to automatic review settings October 15, 2026 23:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Workflow initialization can reject OpenVINO before compilation, and device matching and failed-output cleanup need correction.

3 open findings
What changed in this PR

Adds standalone OpenVINO EPContext compilation to Olive’s existing context-generation pass, without model packaging.

Changes:

  • Adds OpenVINO AOT/JIT compilation, device filtering, and virtual targets.
  • Validates generated context artifacts while preserving QNN behavior.
  • Documents configuration and adds focused tests.
File Description
test/​passes/​onnx/​test_context_binary.py Tests OpenVINO options, registration ownership, and output validation.
olive/​passes/​onnx/​context_binary.py Adds OpenVINO compilation and explicit provider dispatch.
olive/​olive_config.json Declares OpenVINO and CPU/GPU/NPU support.
docs/​source/​reference/​pass.rst Documents standalone compilation and limitations.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread olive/passes/onnx/context_binary.py Outdated
Comment thread olive/passes/onnx/context_binary.py Outdated
Comment thread olive/passes/onnx/context_binary.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants