Skip to content

Recover uploads abandoned by preparation exceptions - #10

Closed
bmehta001 wants to merge 1 commit into
mainfrom
fix/upload-exception-recovery
Closed

bmehta001 wants to merge 1 commit into
mainfrom
fix/upload-exception-recovery

Conversation

@bmehta001

Copy link
Copy Markdown
Owner

Summary

Port all SDK-relevant changes from bmehta001/onnxruntime#11 (source commit 611834d7eafd18ea0d4ca9ab28dfa3b539b3c61b) directly into the SDK, with regression tests adapted to this repository's harness.

  • Share exception recovery between scheduled uploads and immediate events. Release reserved records before retiring an unregistered upload, and still retire it if cleanup throws.
  • Preserve terminal completion ownership once an HTTP callback is registered, including synchronous completion followed by an exception; ignore already-finished uploads.
  • Own pending callbacks with RAII until registry insertion succeeds, and move potentially throwing request-log formatting before registration.
  • Add the source PR's 14 scheduled/immediate component cases and six native-transport fault modes, plus a request-log formatting regression and ownership assertions in existing HTTP tests.

The native tests use a private collector URL on IPv4 loopback and the actual platform HTTP transport. They verify retry of the original reserved event, subsequent uploads, immediate teardown without retry after standard/nonstandard preparation failures, throwing HTTP handoff, and completion-then-throw. Each mode has a 30-second CTest timeout.

ORT-specific dependency-patch plumbing, telemetry wrapper/build integration, enable/disable behavior, and ORT documentation are intentionally not copied. The SDK now contains the fix itself rather than an external patch.

Validation

Fresh Debug builds with MATSDK_WARNINGS_AS_ERRORS=ON, vendored SQLite/zlib, and the repository's pinned GoogleTest submodule:

Check Windows x64, Visual Studio 2026 Ubuntu 24.04 / WSL, GCC 13
Build UnitTests and UploadExceptionTests Passed Passed
TPM, upload-exception, and all HTTP-manager component suites 82 passed 84 passed
All six upload-* native CTests 6 passed (WinHTTP) 6 passed (curl)
Existing full-SDK no-exceptions-smoke build and CTest Passed Passed

Component selector: TransmissionPolicyManagerTests.*:*UploadExceptionTests.*:HttpClientManager*.*. No selected tests were skipped or excluded; the two extra Linux cases exercise response-dispatcher fallback paths compiled only outside Windows.

Formatting checks passed for the new native test and changed C++ implementation/test lines. Existing aligned header declarations were preserved. git diff --cached --check passed before committing. No public package registries were used.

This is targeted SDK/lifecycle validation, not a full SDK test-suite run or a reproduction of the reporter's COLMAP workload. No new sanitizer run was performed for this port. As in the source PR, microsoft/onnxruntime#33923's configuration-map SIGSEGV remains unconfirmed and is not claimed fixed.

Related: microsoft/onnxruntime#33929, microsoft/onnxruntime#33923 (non-closing references).

Port the SDK recovery from bmehta001/onnxruntime#11 so failed upload
preparation cannot leave teardown waiting for a callback that was never
registered. Release reserved records before retiring an upload, contain
cleanup exceptions, and preserve HTTP ownership after registration.

Protect pending callback allocation with RAII and evaluate request logging
before handing ownership to the registry. Adapt all 14 component cases and
six native fault modes to the SDK harness, plus a request-log failure test.

Files changed:
- lib/http/HttpClientManager.cpp
- lib/system/Contexts.hpp
- lib/system/TelemetrySystem.cpp
- lib/tpm/TransmissionPolicyManager.cpp
- lib/tpm/TransmissionPolicyManager.hpp
- tests/functests/CMakeLists.txt
- tests/functests/upload-exception-tests.cpp
- tests/unittests/HttpClientManagerTests.cpp
- tests/unittests/TransmissionPolicyManagerTests.cpp

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6982f030-2ce2-47a3-b52b-9d8a1cdb05d9
@bmehta001

Copy link
Copy Markdown
Owner Author

Superseded by the upstream PR: microsoft#1559

@bmehta001 bmehta001 closed this Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant