Skip to content

[WIP] Fix agent output history and Northbound tracing for v2.8.5 - #4328

Draft
JasonW404 wants to merge 8 commits into
mainfrom
hotfix/v2.8.5
Draft

JasonW404 wants to merge 8 commits into
mainfrom
hotfix/v2.8.5

Conversation

@JasonW404

@JasonW404 JasonW404 commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

CodeAgent rejected ordinary HTML reasoning and replayed model-authored tool-result claims into later context. This patch narrows parsing to protocol markers, accepts ordered case-insensitive actions with intervening prose, keeps Markdown examples as data, and confirms mixed final answers after real execution. Completed steps replay one read-only assistant record from actual calls/runtime fields, preserving masking, deduplication, errors, compaction and authorized recall.

Northbound HTTP tracing was explicitly disabled, its deployment omitted monitoring configuration, and Runtime forwarding omitted trace context. Enable the existing FastAPI/ASGI instrumentation and propagate only W3C traceparent/tracestate with the internal JWT. The Northbound server span covers reception, authentication, preparation, Runtime execution and streaming through completion of the final ASGI send. Runtime/Agent/model/tool spans share the same trace. HTTP disconnect ends its span while the existing persisted-run continuation policy remains unchanged. No global HTTPX instrumentation or new telemetry dependency is introduced.

Deployment notes:

  • Rebuild the shared backend main image and recreate the affected backend containers. Apply the updated Docker Compose or Helm configuration: Northbound now receives the same monitoring.env / nexent-monitoring-env as Config and Runtime.
  • Keep ENABLE_TELEMETRY and OTLP/Phoenix configuration enabled. Existing installations retaining an explicit URL allowlist must add /nb/ and /agent/internal/northbound/. Explicit exclusions and normal sampling behavior remain respected.
  • Completion means the last server ASGI send returned, not client receipt acknowledgement. Native tool-message support is not required by the earlier history fix.

Validation:

  • Earlier output/history patch: 733 local regressions, TPC-D3-001 33 tests, 14 deployed API/real-model scenarios, Phoenix inspection and exact image/source checks passed. Browser/page-refresh checks remain BLOCKED by Work settings; the user chose to complete other validation and record that limitation.
  • Northbound addition: 412 adjacent/deployment UT, 6 propagation/config tests and 8 real ASGI lifecycle cases (isolated from legacy OTel test stubs) passed. Last-send blocking, send failure, cancellation, concurrent trace isolation, disabled/missing telemetry, no forwarded baggage/credentials and configuration parity are covered. Helm render verified the monitoring secret.
  • Native VM NB-OTEL-001: 3 passed, using real Qwen and skill execution plus authenticated Phoenix: complete response ancestry/timing, unauthenticated 401 without downstream calls, headers-only disconnect with one persisted completed run.
  • Final clean-snapshot ARM64 container validation on 7995430: NB-OTEL-001 all 3 tests passed. Four backend images, installed SDK/backend source hashes, full authentication, Northbound telemetry configuration and the new default URL allowlist verified. Authenticated Phoenix MCP retrieved all three actual traces. The initial UT run found stale Helm expectations excluding Northbound from monitoring secret/precedence; those expectations are aligned and all 24 owning tests pass. Runtime product files are unchanged by this test correction. Latest-head automated UT, CodeQL (all languages), and Codecov patch checks passed; the UT workflow head SHA matches 7995430. Codecov reports 94.62366% patch coverage. Main/MCP/Data-process AMD64 and ARM64 image CI passed on the identical product-code commit 36109ae; the final test-only commit did not trigger those path-filtered builds, while local image/container validation uses the exact final commit.

@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.62366% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
sdk/nexent/core/agents/output_protocol.py 94.54% 1 Missing and 2 partials ⚠️
sdk/nexent/core/agents/core_agent.py 75.00% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@JasonW404 JasonW404 changed the title [WIP] Fix output protocol parsing and runtime execution history for v2.8.5 [WIP] Fix agent output history and Northbound tracing for v2.8.5 Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant