Skip to content

Structured agentic rules: one custom-check engine - #65

Merged
SushantGautam merged 14 commits into
mainfrom
feat/structured-agentic-rules
Oct 30, 2026
Merged

Structured agentic rules: one custom-check engine#65
SushantGautam merged 14 commits into
mainfrom
feat/structured-agentic-rules

Conversation

@SushantGautam

@SushantGautam SushantGautam commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Structured agentic rules replace the legacy constraint panels.

  • Rule engine: count, exists, none, all, distinct over tool, error, retrieval and answer units. Order, grouping, quote and retry checks were removed because they cannot be expressed per unit.
  • Stored rules are custom checks. Presets are editor starters (grouped picker).
  • Stored revisions migrated; the migration step is now a no-op.
  • Semantic judge packet: retrieved sources from every step; compacted tool results.
  • Result UI: rule sentences as card titles, collapsed verdict cards, readable span references (LLM-1, TOOL-1), response tag on the answer call.
  • Target metadata: off-state flags hidden on display only.

Checks: full local suite 1353 passed, 3 skipped; ruff clean. Browser checks of the editor and result pages were partial.

SushantGautam and others added 14 commits October 30, 2026 14:07
@SushantGautam @claude
feat(agentic): deterministic checks driven by scenario config, not Ac…
58f1ca6 
…me specifics

- Forbidden tools accept name or name(arg=value) entries, saved and shown in the builder.
- Forbidden values checked in tool-call arguments, tool results and the final answer.
- Retrieval steps recognised from the trace (gen_ai.retrieval.documents), not tool names.
- Trace error check takes no per-scenario exemptions.
- Remove order-specific scope synthesis, access-refused and session-customer logic,
  and the isolate-customer flag; the tool reports its own refusals.
- Scenario pack: CUST-001 story, file-name sources, D02/D09 expectations.
- Bootstrap no longer resets the admin password on each start (keeps sessions).
- Cookie Secure flag switchable via SIMPLEAUDIT_SECURE_COOKIES (default unchanged).
- Run-detail: agentic status and judge badge per result, bar filter, tooltips.

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
feat(agentic): structured rule model (phase 1)
a8ccd9e 
- Fixed set of rule types (tool called/not called, sequence with four modes,
  precedes, budget, repeated calls, forbidden values, grounded value, retrieval,
  errors, retries), each with required and optional fields.
- Validation returns per-rule messages; defaults filled by normalize_rules.
- Not wired into the checks or the builder yet (phase 2 and 3).

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
feat(agentic): evaluate structured rules from the trace (phase 2)
23d1e65 
- checks/rules.py: one check per rule (rule.<index>.<type>) for all eleven
  rule types, reading tool calls, arguments, results, retrieval steps,
  error status and the final answer only. Missing evidence is INCONCLUSIVE.
- Schema accepts agentic.rules and reports invalid rules by index.
- Orchestrator runs the rule checks alongside the existing checks.
- Retrieval sources read from both the documents attribute and tool results.

Full non-embedded suite: 1458 passed, 3 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
feat(agentic): rule editor in the scenario builder (phase 3)
fe9c906 
- Rule cards: pick a rule type, fill its fields, see a one-line summary.
- Stored as agentic.rules via a hidden field; "Edit as JSON" is an advanced toggle.
- Server validates rules on save and shows the exact problem; a cleared editor
  removes rules saved earlier.
- Page preview includes the rules.
- Tests: form merge (save, reject, clear, untouched scenarios).

Full non-embedded suite: 1463 passed, 3 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
feat(agentic): migrate legacy controls to rules where the meaning is …
1ed28de 
…the same

- audits/agentic/migration.py converts tools.expected/forbidden, budgets,
  required sequence, retrieval sources/min documents and forbidden data scopes
  to rules. Unmapped settings stay as legacy sections.
- The rule editor shows saved rules plus converted legacy settings; saving
  takes them over. Mapped legacy inputs are removed from the form.
- Fixture scenarios store converted metadata (verdicts unchanged, run 33 vs 35:
  9 of 10 identical; D07 keeps FAIL with one combined forbidden-values check).
- Metadata validator accepts and validates the rules key.

Full suite: 1471 passed, 3 skipped (parallel, incremental).

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
feat(agentic): show rules and rule checks in the result UI
df91480 
- Rule checks read as 'Rule N · <label>' instead of raw ids (rule.0.tool_called).
- The controls summary lists the scenario's rules as sentences.
- describe_rule / check_label shared by the templates and tests.

Full suite (incremental): 1473 passed, 3 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
fix(agentic): keep Edit as JSON hidden unless ?advanced=1
c573e85 
Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
feat(agentic): show scenario rules with the description and expected …
4629194 
…behaviour

- Result page: Rules listed after Expected behaviour, as readable sentences.
- Scenario list: Rules listed after Expected behavior for each scenario.

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
feat(agentic): one rule engine: select trace units, then check (phase 1)
b776895 
- rule_engine.py: custom check rules (select kind and conditions, then check
  count / exists / none / all / distinct, then expect). Field catalog read from
  the trace (tool name, arguments, result, error; retrieval source and text;
  answer text; span kind and error). Missing capture is INCONCLUSIVE.
- Presets tool_called, tool_not_called, budget, errors, forbidden_values and
  retrieval compile into the engine; equivalence tests check each against its
  original verdict.
- Retrieval minimum now counts documents, not retrieval steps (deliberate).
- Forbidden values with no final answer are INCONCLUSIVE, not PASS (deliberate).
- Ordering presets (sequence, precedes, retries, grounded value, repeats) stay
  on their evaluators until the engine gains ordering checks.

Full suite (incremental): 1559 passed, 3 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
feat(agentic): ordering and repeat checks in the rule engine (phase 2)
f140f45 
- Engine gains in-order, before, largest identical group, and longest run checks.
- Sequence, precedes and repeated-call presets compile into the engine (equivalence
  tests against their original verdicts).
- Retries, grounded value and any-order sequences stay on their evaluators.
- Replica regression: run 37 matches run 35 for all ten scenarios.

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
feat(agentic): custom check builder in the rule editor
2451cc8 
- Custom check card: look at a kind of unit, add conditions from the trace field
  catalog, choose the check and the expectation. The summary reads as a sentence.
- Field catalog served to the editor from the engine, so only recorded fields
  can be picked.
- Server accepts the saved rule; browser test built and saved a count check.

Full suite: 1673 passed, 3 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam @claude
feat(agentic): migrate trace-checkable constraints to rules; remove o…
c68d3e7 
…ld panels

- Allowed tools, retrieval maximum, rerank required/min/max, step limit,
  consecutive repeats, forbidden side effects, and handoff required/maximum
  now convert to rules (custom checks or existing rule types).
- Remove the Tools, Retrieval, Policy, Guardrails, Handoffs and Trajectory
  panels. The form keeps Judge and enforcement.
- Settings with no trace evidence (read-only, side-effect permission,
  destinations, data scopes, guardrails, approvals, state, handoff details,
  retries per tool) stay stored but are not shown or evaluated.
- Engine: not-one-of operator.
- Replica regression: run 38 matches run 37 for all ten scenarios.

Full suite: 1673 passed, 3 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>
@SushantGautam
feat(agentic): one rule engine of custom checks; presets become starters
c838ca6 
- Rule engine keeps only count/exists/none/all/distinct over tool, error,
  retrieval and answer units; removed in-order, before, group, quotes,
  retries and all-of checks (not expressible per unit).
- Stored rules are custom checks; presets are editor starters.
- Semantic judge packet: retrieved sources from every step, compacted results.
- Result UI: rule sentences as card titles, collapsed cards, readable span refs
  (LLM-1, TOOL-1), response tag on the answer call.
- Target metadata: off-state flags hidden on display only.
@SushantGautam
feat(agentic): convert stored rules on migration; mark unsupported rules
35f397b 
- Migration 0004 converts stored preset rules to count/exists/none checks
  (self-contained), flattens all-of groups, and keeps rules that need order
  or grouping as 'unsupported' with the original under 'legacy'.
- Unsupported rules show a message in the UI and evaluate as INCONCLUSIVE.
- Non-check rules no longer crash evaluation.
@SushantGautam
SushantGautam merged commit 226cf24 into main Oct 30, 2026
1 of 2 checks passed
@SushantGautam
SushantGautam deleted the feat/structured-agentic-rules branch October 30, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@SushantGautam