[yonib05]

Description

Adds a red teaming sample for Strands Evals 1.7.0 under python/06-evaluate/red-teaming/. It attacks a small refund-desk agent with GOAT, scores each conversation with an LLM judge, and then checks the agent's recorded tool calls deterministically, so the judge's score and what the agent actually did can be compared.

This is the sample from TheWitcherish/sample-strands-evals-red-teaming at 44ba621, moved under the org so the launch post (strands-agents/harness-sdk#5543) doesn't depend on a personal repo. Changes from the original:

• run.py saves each run's report with report.to_file() to runs/report-N.json, and saves the suite with the attacker and judge models, so the post's to_file / from_file round-trip snippet runs against files the sample produces.
• README: clone path for this repo, the two new files under "What a run leaves on disk", the blog link, and the summary paragraph moved first so samples.json picks it up as the description.
• samples.json: one entry added for the new sample.

Testing

• uv run pytest: 33 passed (offline, no Bedrock calls).
• uv run src/run.py --runs 1 --only leak_the_code --max-turns 2 against Bedrock in us-east-1: completed, wrote runs/run-1.json, runs/report-1.json, and suite/refund_desk_goat.json (suite includes deepseek.v3.2 and us.anthropic.claude-sonnet-5-5).
• Ran the post's round-trip snippet from the sample directory: RedTeamExperiment.from_file() replayed all 5 cases with agent_factory passed to the run method, and RedTeamReport.from_file() reloaded and displayed the earlier report.

Notes

• License: the original repo has no license file. The author should confirm they're fine contributing it under this repo's license before merge.
• samples.json is stale on main: regenerating it with generate-samples-manifest.mjs also adds fastapi-strands-production-starter, typescript/01-learn/07-express-chatbot, and typescript/02-deploy/02-nextjs-vercel-adapter. I added only this sample's entry to keep the PR focused.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

Latest scan for commit: e7f5191 | Updated: 2026-10-26 22:14:10 UTC

✅ Security Scan Report (PR Files Only)

Scanned Files

• python/06-evaluate/red-teaming/.gitignore
• python/06-evaluate/red-teaming/README.md
• python/06-evaluate/red-teaming/assets/cover.png
• python/06-evaluate/red-teaming/pyproject.toml
• python/06-evaluate/red-teaming/src/cases.py
• python/06-evaluate/red-teaming/src/desk.py
• python/06-evaluate/red-teaming/src/run.py
• python/06-evaluate/red-teaming/src/tool_check.py
• python/06-evaluate/red-teaming/tests/test_tool_check.py
• python/06-evaluate/red-teaming/uv.lock
• ... and 1 more files

---

Security Scan Results

Critical	High	Medium	Low	Info
0	0	0	0	0

Threshold: High

No security issues detected in your changes. Great job!

This scan only covers files changed in this PR.