feat(ai-red-teaming): generalize attack registry + unified white-box support (v1.20.0)

capabilities/ai-red-teaming/capability.yaml

@@ -1,6 +1,6 @@
 schema: 1
 name: ai-red-teaming
-version: "1.18.8"
+version: "1.20.0"
 description: >
   Probe the security and safety of AI applications, agents, and foundation models.
   Orchestrates adversarial attack workflows to discover vulnerabilities in LLMs,

capabilities/ai-red-teaming/scripts/attack_runner.py

@@ -390,7 +390,7 @@ def _auto_execute_workflow(filename: str, timeout: int = 3600) -> str:
     "databricks llama": "databricks/databricks-meta-llama-3-1-70b-instruct",
 }
 
-_ATTACK_DEFS: dict[str, dict] = {
+_MANUAL_ATTACK_DEFS: dict[str, dict] = {
     "tap_attack": {
         "module": "tap",
         "function": "tap_attack",
@@ -491,6 +491,21 @@ def _auto_execute_workflow(filename: str, timeout: int = 3600) -> str:
             "top_k_strategies": 5,
         },
     },
+    # AutoDAN (Liu et al. ICLR 2024): genetic-algorithm readable jailbreak. Closed-box
+    # (needs only text output + a judge), so it runs against hosted dn/ models like any
+    # other jailbreak. The other white-box attacks (GCG family, embedding, activation)
+    # are SDK-only: they require dreadnode[whitebox] + a LOCAL open-weight surrogate for
+    # gradients, which the hosted inference sandbox does not provide.
+    "autodan_attack": {
+        "module": "autodan",
+        "function": "autodan_attack",
+        "has_attacker": True,
+        "default_iterations": 50,
+        "extra_defaults": {
+            "early_stopping_score": 0.9,
+            "beam_width": 8,
+        },
+    },
     "renellm_attack": {
         "module": "renellm",
         "function": "renellm_attack",
@@ -928,6 +943,88 @@ def _auto_execute_workflow(filename: str, timeout: int = 3600) -> str:
     },
 }
 
+# White-box gradient attacks optimize against a LOCAL open-weight surrogate: the
+# user must pass that model path/id as `attacker_model` (see the
+# whitebox-local-surrogate skill). They are discoverable (unified SDK + TUI
+# support), but flagged `requires_surrogate` so the agent knows to supply a local
+# model and run in local runtime. In the hosted sandbox with a dn/* attacker, the
+# SDK raises a clear error (no gradients over the proxy) rather than crashing.
+_WHITEBOX_REQUIRES_SURROGATE = frozenset(
+    {
+        "gcg_attack",
+        "faster_gcg_attack",
+        "i_gcg_attack",
+        "attngcg_attack",
+        "mask_gcg_attack",
+        "gbda_attack",
+        "pez_attack",
+        "cold_attack",
+        "soft_gcg_attack",
+        "arca_attack",
+        "amplegcg_attack",
+        "refusal_ablation_attack",
+        "soft_prompt_attack",
+        "latentbreak_attack",
+        "adaptive_random_search_attack",
+    }
+)
+
+
+def _discover_attack_defs() -> dict[str, dict]:
+    """Auto-discover text-jailbreak attacks from the installed SDK so new attacks
+    flow into the capability without hand-wiring. Only functions with the standard
+    ``(goal, target, evaluator_model, ...)`` signature are included (skips
+    evasion/extraction/inversion/image attacks, which take a different target).
+    White-box gradient attacks ARE included for unified SDK + TUI support, flagged
+    ``requires_surrogate`` so the agent supplies a local model path as
+    ``attacker_model`` (see the whitebox-local-surrogate skill). Defaults are read
+    from each function's signature; the manual table above overrides these for
+    attacks with specially tuned parameters."""
+    import inspect
+
+    try:
+        import dreadnode.airt as _airt
+    except Exception:
+        return {}
+
+    discovered: dict[str, dict] = {}
+    for _name in dir(_airt):
+        if not _name.endswith("_attack"):
+            continue
+        _fn = getattr(_airt, _name, None)
+        if not callable(_fn):
+            continue
+        try:
+            _params = inspect.signature(_fn).parameters
+        except (TypeError, ValueError):
+            continue
+        if not ({"goal", "target", "evaluator_model"} <= set(_params)):
+            continue
+
+        def _default(key: str, fallback: object, _p: object = _params) -> object:
+            value = _p[key].default if key in _p else inspect.Parameter.empty
+            return fallback if value is inspect.Parameter.empty else value
+
+        extra: dict[str, object] = {}
+        early_stop = _default("early_stopping_score", None)
+        if early_stop is not None:
+            extra["early_stopping_score"] = early_stop
+
+        discovered[_name] = {
+            "module": _fn.__module__.rsplit(".", 1)[-1],
+            "function": _name,
+            "has_attacker": "attacker_model" in _params,
+            "default_iterations": _default("n_iterations", 20),
+            "extra_defaults": extra,
+            "requires_surrogate": _name in _WHITEBOX_REQUIRES_SURROGATE,
+        }
+    return discovered
+
+
+# Auto-discovered base (all standard-signature SDK attacks), with the hand-tuned
+# manual table taking precedence where an attack needs specific defaults.
+_ATTACK_DEFS: dict[str, dict] = {**_discover_attack_defs(), **_MANUAL_ATTACK_DEFS}
+
 # Short aliases -> canonical attack name
 ATTACK_ALIASES: dict[str, str] = {}
 for _canon, _def in _ATTACK_DEFS.items():

capabilities/ai-red-teaming/skills/whitebox-local-surrogate/SKILL.md

@@ -0,0 +1,55 @@
+---
+name: whitebox-local-surrogate
+description: How to run white-box (gradient/embedding/activation) attacks, which need a local open-weight surrogate model the user supplies
+allowed-tools: register_assessment generate_attack
+---
+
+# White-box attacks: local surrogate
+
+White-box attacks (the GCG family, embedding-space, generator, and activation
+attacks) do not just read the target's text - they optimize against model
+internals (gradients, logits, hidden states). That requires a **local open-weight
+surrogate model**, which the user provides. The optimized prompt then transfers to
+the closed-box target (`dn/*` or BYO API) as usual.
+
+Attacks that need a surrogate: `gcg`, `faster_gcg`, `i_gcg`, `attngcg`, `mask_gcg`,
+`arca`, `gbda`, `pez`, `cold`, `soft_gcg`, `amplegcg`, `refusal_ablation`,
+`soft_prompt`, `latentbreak`. (`autodan` is closed-box and needs no surrogate.)
+
+## How to run one
+
+Ask the user for the surrogate (a Hugging Face id or local path) and pass it as
+`attacker_model`; keep the real model as `target_model`:
+
+```
+generate_attack(
+    attack="gcg",
+    goal="...",
+    target_model="dn/claude-opus-4-6",        # closed-box transfer target
+    attacker_model="meta-llama/Llama-2-7b-chat-hf",  # LOCAL open-weight surrogate
+    evaluator_model="dn/claude-opus-4-6",
+)
+```
+
+For `refusal_ablation` / `soft_prompt` / `latentbreak` (local-only: no transferable
+text), the surrogate IS the attacked model and it is judged on its own generation.
+
+## Requirements and guardrails
+
+- **Local runtime only.** These need the surrogate's weights and (for 7B+) a GPU.
+  The managed hosted sandbox has neither - run white-box attacks in a local or
+  self-hosted runtime. If a `dn/*` model is passed as `attacker_model`, the SDK
+  raises a clear error (no gradients over the inference proxy); switch to a local id.
+- **Install the extra:** `pip install "dreadnode[whitebox]"` (torch + transformers).
+- **Trust:** do not enable `trust_remote_code` for untrusted model repos (it runs
+  code on load).
+- **Resources:** 7B-scale surrogates need a GPU with ~80 GB; small models run on CPU
+  for experimentation. `early_stopping_score` keeps cost bounded - easy targets exit
+  fast; only hard ones run the full `n_iterations`.
+- Prefer an **aligned open-weight** surrogate (e.g. Llama-2-7b-chat) for meaningful
+  transfer; a tiny random model produces junk suffixes.
+
+## When NOT to use
+
+If the user only has a hosted model and no local weights/GPU, use closed-box attacks
+instead (`tap`, `pair`, `crescendo`, `autodan`, ...). See the attack-selection-guide.