Repository navigation
Conversation
Open WebUI's instrumentation sets gen_ai.retrieval.documents to a list of JSON strings. _units only kept dict elements, so rules on retrieved documents saw zero documents on those traces. Decode string elements and skip ones that are not valid JSON.
The evidence packet passed each element of gen_ai.retrieval.documents to _source_values as is, so a document sent as a JSON string became one source ID holding the whole string. Parse elements that decode to an object or a list; other strings stay source names as before.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bundled Open WebUI instrumentation (
deploy/openwebui-subpath/subpath.patch) setsgen_ai.retrieval.documentson retrieval spans to a list of JSON strings, one per document (document_id,file_id,chunk_id,rank,score, andcontentwhen document capture is on). Since #65,_unitsinaudits/agentic/rule_engine.pykeeps only list elements that are already dicts, so on those traces everyretrieval_documentrule saw zero documents: "At least 1 retrieved document" reported FAIL with "0 matching retrieval_document unit(s)" although the span carried documents.This decodes string elements with
json.loads, the same handling the retrieval check had before #65, and skips strings that are not valid JSON or do not decode to an object. Dict elements and the case where the whole attribute is one JSON string go through the same path as before.The new test in
audits/agentic/tests/test_rule_engine.pyuses a document in the Open WebUI shape plus an invalid string: the count andretrieval.textchecks pass, and aretrieval.sourcecheck is INCONCLUSIVE because these documents carry no source; a JSON-string document that does carry a source still matches. The test fails onmainand passes with the change.The same cause reached the semantic judge's evidence packet:
_retrieved_sourcesinaudits/agentic/semantic_judge.pypassed each element of the attribute to_source_valuesas is, so every Open WebUI document became oneretrieval_source_idsentry holding the whole JSON string. It now parses elements that decode to a JSON object or list and passes them on like dict documents, so the packet lists the document'ssourceorfile_id; any other string stays a source name as before. The new test inaudits/agentic/tests/test_semantic_judge_execution.pyfails on the first commit and passes with the second.pytest -n 4 -m "not embedded_hatchet"(no API keys in the environment, the local-Ollama live test deselected) gave 1378 passed, 2 skipped onmain, 1379 passed, 2 skipped after the first commit and 1380 passed, 2 skipped after the second;ruff check .is clean.AI disclosure: implemented with Claude Code.