Repository navigation
Conversation
Migration 0004 keeps rules that need order or grouping as
{"type": "check", "check": "unsupported", "legacy": {...}} and says
the engine reports them as INCONCLUSIVE. validate_rules rejected that
shape, so validate_agentic_metadata raised for every such revision: the
agentic evaluation ended in ERROR and the scenario could not be saved.
Skip validation for exactly that shape.
This was referenced Oct 31, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migration
0004_convert_rules_to_primitiveskeeps rules that need order or grouping as{"type": "check", "check": "unsupported", "legacy": {...}}, and its docstring says the engine reports them as INCONCLUSIVE.validate_rulesinaudits/agentic/rules.pyranvalidate_checkon that shape, which rejects thelegacykey and thecheckvalue, sovalidate_agentic_metadataraised for every revision that holds such a rule. That function runs before the engine ininfra/engine.py, in the post-hoc evaluation ininfra/worker.py(which turns the exception into an ERROR evaluation) and inscenarios/services.pywhen a scenario is saved.This skips validation for exactly the shape the migration writes: the keys
type,checkandlegacy, withcheckset tounsupportedandlegacyan object. Any other rule is validated as before.The new test in
audits/agentic/tests/test_rule_migration.pyconverts aprecedesrule with the migration, validates the metadata and evaluates the rule: it fails onmainwithInvalid metadata.agentic.rulesand passes with the change, where the rule comes out INCONCLUSIVE. A second test checks that anunsupportedrule with other keys is still rejected.pytest -n 4 -m "not embedded_hatchet"(no API keys in the environment, the local-Ollama live test deselected) gave 1378 passed, 2 skipped onmainand 1380 passed, 2 skipped with the change;ruff check .is clean.AI disclosure: implemented with Claude Code.