Hi,
I read the trace-backed tool-call scorer documentation and #2725, and saw the stored-message scorer proposed in #2854. Keeping incomplete evidence UNDETERMINED is particularly relevant to a small integration question I have. I also saw #2859 / #2860 on reporting coverage, so this is not a request to duplicate that work.
I'm the solo developer of Agentic Security Harness. Its application-owned Gateway keeps a model's proposed operation separate from permission to execute it. I'd like to explore a small external, mock-only example that supplies execution evidence to PyRIT while retaining the separate authorization decision.
What would be the best existing observation/scorable or trace-client interface for this? The cases I want to distinguish are:
- Allowed and invoked, with the expected synthetic result.
- Denied before invocation: the Gateway has a decision, but there must be no fabricated tool-execution span.
- Invoked but failed inside the tool: invocation is still observed, even though completion failed.
- Missing or incomplete evidence: no confident negative conclusion.
The invocation scorer would keep its documented meaning; policy decisions and operation outcomes would be separate fields, with coverage reported explicitly. I am not proposing that a model response, a Gateway receipt or an unpaired request should prove execution.
Here's the existing small Harness example. The PyRIT adapter is not implemented. Is this a useful external example to develop, or is there an existing one I should build on? I'm asking before writing a PR or proposing a new scorer.
Best,
krivonosoff161
Hi,
I read the trace-backed tool-call scorer documentation and #2725, and saw the stored-message scorer proposed in #2854. Keeping incomplete evidence
UNDETERMINEDis particularly relevant to a small integration question I have. I also saw #2859 / #2860 on reporting coverage, so this is not a request to duplicate that work.I'm the solo developer of Agentic Security Harness. Its application-owned Gateway keeps a model's proposed operation separate from permission to execute it. I'd like to explore a small external, mock-only example that supplies execution evidence to PyRIT while retaining the separate authorization decision.
What would be the best existing observation/scorable or trace-client interface for this? The cases I want to distinguish are:
The invocation scorer would keep its documented meaning; policy decisions and operation outcomes would be separate fields, with coverage reported explicitly. I am not proposing that a model response, a Gateway receipt or an unpaired request should prove execution.
Here's the existing small Harness example. The PyRIT adapter is not implemented. Is this a useful external example to develop, or is there an existing one I should build on? I'm asking before writing a PR or proposing a new scorer.
Best,
krivonosoff161