From fb5cceca6033b86137e9c653f85c3647e47c2242 Mon Sep 17 00:00:00 2001 From: Raja Sekhar Rao Dheekonda Date: Tue, 20 Oct 2026 21:19:02 -0700 Subject: [PATCH] fix(ai-red-teaming): correct stale catalog counts (goals, transforms, scorers) Verified against dreadnode-tiger latest main + the capability registries: - Harm goals: 260 -> 279 (goals.csv has 279) across capability.yaml, agent greeting, compliance-mapping skill, tools/attacks.py. - transform-reference skill: dropped the false "all 183 transforms" claim - the capability resolves 429 text (596 incl. multimodal/variants), matching the greeting's "590+"; the skill documents the core set. - scorer-reference skill: dropped "all 84 scorers" - 146 are user-selectable (SCORER_REGISTRY) plus agentic *_detected scorers auto-wired per OWASP-ASI category, matching the greeting's "140+". Attacks (70+) and the 590+/140+ greeting claims were already accurate; attacks auto-discover from the SDK (73, incl. all 15 white-box). --- capabilities/ai-red-teaming/agents/ai-red-teaming-agent.md | 4 ++-- capabilities/ai-red-teaming/capability.yaml | 2 +- .../ai-red-teaming/skills/compliance-mapping/SKILL.md | 2 +- capabilities/ai-red-teaming/skills/scorer-reference/SKILL.md | 4 ++-- .../ai-red-teaming/skills/transform-reference/SKILL.md | 4 ++-- capabilities/ai-red-teaming/tools/attacks.py | 2 +- 6 files changed, 9 insertions(+), 9 deletions(-) diff --git a/capabilities/ai-red-teaming/agents/ai-red-teaming-agent.md b/capabilities/ai-red-teaming/agents/ai-red-teaming-agent.md index 1b3af6f..cb50d7f 100644 --- a/capabilities/ai-red-teaming/agents/ai-red-teaming-agent.md +++ b/capabilities/ai-red-teaming/agents/ai-red-teaming-agent.md @@ -5,7 +5,7 @@ description: > Unified AI Red Teaming agent for probing security and safety vulnerabilities in LLMs, agentic systems, MCP servers, multi-agent architectures, ML classifiers, and custom AI endpoints. Orchestrates 70+ attack strategies covering traditional ML and generative AI, 590+ transforms, - 140+ scorers, and 260 bundled harm goals across OWASP LLM Top 10, + 140+ scorers, and 279 bundled harm goals across OWASP LLM Top 10, OWASP ASI01-ASI10, and MITRE ATLAS frameworks. --- @@ -379,7 +379,7 @@ Three tiers with sub-categories: | agentic_cascading_failure | Agentic Cascading Failure | 10 | ASI08 | | agentic_trust_exploitation | Agentic Trust Exploitation | 10 | ASI09 | -**Total: 25 sub-categories, 260 bundled goals across 3 tiers.** +**Total: 25 sub-categories, 279 bundled goals across 3 tiers.** ## Agentic Red Teaming diff --git a/capabilities/ai-red-teaming/capability.yaml b/capabilities/ai-red-teaming/capability.yaml index fa77e75..6b89a07 100644 --- a/capabilities/ai-red-teaming/capability.yaml +++ b/capabilities/ai-red-teaming/capability.yaml @@ -11,7 +11,7 @@ description: > and NIST AI RMF compliance frameworks. Also probes traditional black-box ML classifiers: model extraction (model stealing), membership inference (training-data leakage), and model evasion (adversarial examples) across tabular, image, and text. - 590+ transforms, 140+ scorers, and 260 bundled harm goals across 25 + 590+ transforms, 140+ scorers, and 279 bundled harm goals across 25 sub-categories in safety, security, and agentic tiers. agents: diff --git a/capabilities/ai-red-teaming/skills/compliance-mapping/SKILL.md b/capabilities/ai-red-teaming/skills/compliance-mapping/SKILL.md index f1780e6..511d27c 100644 --- a/capabilities/ai-red-teaming/skills/compliance-mapping/SKILL.md +++ b/capabilities/ai-red-teaming/skills/compliance-mapping/SKILL.md @@ -96,7 +96,7 @@ Maps AIRT attacks, transforms, and scorers to major AI security compliance frame - MCP security — 12 transforms, 7 scorers - Multi-agent security — 10 transforms, 6 scorers - Exfiltration — 8 transforms, 5 scorers -- Content policy — 260 bundled goals across 25 sub-categories +- Content policy — 279 bundled goals across 25 sub-categories - System prompt extraction — 6 transforms - RAG poisoning — 7 transforms diff --git a/capabilities/ai-red-teaming/skills/scorer-reference/SKILL.md b/capabilities/ai-red-teaming/skills/scorer-reference/SKILL.md index f17e8d7..5cdb88d 100644 --- a/capabilities/ai-red-teaming/skills/scorer-reference/SKILL.md +++ b/capabilities/ai-red-teaming/skills/scorer-reference/SKILL.md @@ -1,12 +1,12 @@ --- name: scorer-reference -description: Complete catalog of all 84 scorers with exact names, categories, and when to use each +description: Catalog of AIRT scorers - 146 user-selectable (+ agentic *_detected scorers auto-wired per OWASP-ASI category); documents the core set with exact names, categories, and when to use each allowed-tools: generate_attack generate_agentic_attack --- # Scorer Reference -All 84 scorers available in the AIRT SDK. Use these EXACT names in the `scorers` parameter. +The AIRT SDK exposes **146 user-selectable scorers** (plus agentic `*_detected` scorers that the agentic suite auto-wires per OWASP-ASI category). The tables below document the core set - use these EXACT names in the `scorers` parameter, or ask `"Show me the available scorers"` for the full list. ## Default Scorer diff --git a/capabilities/ai-red-teaming/skills/transform-reference/SKILL.md b/capabilities/ai-red-teaming/skills/transform-reference/SKILL.md index e931ea1..875ba20 100644 --- a/capabilities/ai-red-teaming/skills/transform-reference/SKILL.md +++ b/capabilities/ai-red-teaming/skills/transform-reference/SKILL.md @@ -1,12 +1,12 @@ --- name: transform-reference -description: Complete catalog of all 183 transforms with exact names, parameters, categories, and LLM requirements +description: Catalog of AIRT transforms - 429 text transforms (596 incl. multimodal/variants); documents the core set with exact names, parameters, categories, and LLM requirements allowed-tools: generate_attack generate_agentic_attack --- # Transform Reference -All 183 transforms available in the AIRT SDK. Use these EXACT names in the `transforms` parameter. +The AIRT SDK resolves **429 text transforms** (596 incl. multimodal + parameterized variants). The tables below document the core set - use these EXACT names in the `transforms` parameter, or ask `"Show me all available transforms"` for the full list. ## Encoding (17 transforms) diff --git a/capabilities/ai-red-teaming/tools/attacks.py b/capabilities/ai-red-teaming/tools/attacks.py index 0308b04..94b4caa 100644 --- a/capabilities/ai-red-teaming/tools/attacks.py +++ b/capabilities/ai-red-teaming/tools/attacks.py @@ -224,7 +224,7 @@ def generate_category_attack( """Generate a multi-goal attack across bundled harm categories. Sweeps attacks across entire goal sub-categories from the bundled - dataset (260 goals across 25 sub-categories in safety/security/agentic + dataset (279 goals across 25 sub-categories in safety/security/agentic tiers). Goal text is embedded in the generated script — never returned to the agent.