WorldAuditBench

Interactive 3D World Auditing with Multimodal Agents

Ziyan Jiang^1,* · Jingbo Yang^1,* · Jiabao Ji^1,* · Yujian Liu¹
Qiucheng Wu¹ · Tommi Jaakkola² · Yang Zhang³ · Shiyu Chang¹

¹ UC Santa Barbara   ² MIT CSAIL   ³ MIT-IBM Watson AI Lab
^* Equal contribution

213 tasks · 13 environments · 5 anomaly families · 2 auditing paradigms

WorldAuditBench evaluates whether multimodal agents can explore a 3D world, investigate suspicious observations, and identify anomalies with visual evidence. It covers both Unreal Engine 5 and Three.js environments, from furnished interiors to cities and open landscapes.

Demonstrations · Benchmark · Quick start · Resources · Citation

Demonstrations

Representative cases from the paper, covering static physics, interactive physics, spatial consistency, temporal consistency, and semantic consistency.

Benchmark

Auditing requires more than recognizing an unusual image. An agent may need to approach an object, test a collision, change its viewpoint, or revisit a location to establish what is wrong.

Anomaly family	Examples	Tasks
Static physics	Floating objects, intersections, implausible scale	59
Interactive physics	Missing or unexpected collisions, abnormal trajectories	41
Spatial consistency	Visibility, viewpoint, lighting, and material inconsistencies	51
Temporal consistency	Changes in existence, attributes, or operational state	40
Semantic consistency	Improper configurations and historical incompatibilities	22
Total	126 Unreal Engine 5 + 87 Three.js tasks	213

The paper compares two auditing paradigms:

• VLM agents: reason during exploration and choose their next actions using observations and evidence memory, with a budget of 40 actions.
• VLA + VLM: a VLA explores for 60 simulated seconds, then a VLM analyzes the recorded trajectory.

The strongest evaluated agent reaches 42.3% success, compared with 83.4% for humans.

Quick start

Python 3.11+. Unreal environments require Linux and an NVIDIA GPU. Three.js environments run in Chromium on Linux or macOS.

git clone https://github.com/UCSB-NLP-Chang/WorldAuditBench.git
cd WorldAuditBench
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m playwright install chromium
python agent/vlm/native/setup.py

Install and authenticate the model client you want to use: Codex, Claude Code, Gemini CLI, Qwen Code or OpenCode. See the agent guide.

Run an agent

python scripts/experiments/run.py --agent gemini --tasks S01 --download

Use codex, claude, gemini, qwen or muse for --agent. Model IDs and reasoning settings are in agents.json. --download installs the selected environment from Hugging Face; existing verified downloads are reused.

# Three.js task
python scripts/experiments/run.py --agent codex --tasks JS_AF01 --download

# A batch of tasks
python scripts/experiments/run.py --agent gemini \
  --tasks @data/benchmark/splits/unreal.txt --download --output out/gemini-unreal

# Budget and example settings
python scripts/experiments/run.py --agent claude --tasks S01 \
  --max-actions 20 --no-icl --output out/claude-20-noicl

Each task produces report.json, evidence.json and its recording under the output directory. --dry-run prints the commands without running an environment or calling a model. Additional client options can be passed after --.

Run the judge

python -m judge.judge --task S01 \
  --model-output out/runs/gemini/S01/report.json \
  --evidence out/runs/gemini/S01/evidence.json \
  --output out/runs/gemini/S01/judge.json

The judge loads the task rubric and returns score (0 or 1) and reason. It uses GPT-6 Astra with medium reasoning through an authenticated Codex CLI. See judge options.

View a task

python scripts/view_task.py --download
# Or preselect a task:
python scripts/view_task.py JS_AF01 --download

The viewer lets you filter all 213 tasks, read the scene instructions and explore with keyboard and mouse. It includes reset, stop and full-screen controls; anomaly answers are collapsed until you choose to reveal them. No annotation accounts or review database are needed.

Three.js scenes run directly in your browser. For Unreal, run the viewer on a Linux NVIDIA GPU host and install the UE 5.6 Pixel Streaming player once (Node.js 22+ and npm required):

python scripts/setup_pixel_streaming.py
python scripts/view_task.py S01 --download --no-browser

Unreal runs continuously on the GPU host and streams live video to the browser. For a remote host, forward the viewer's loopback port over SSH; configure STUN/TURN when WebRTC cannot reach the GPU host directly. See the viewer setup guide for connection commands and options.

VLA exploration

Open-P2P records exploration, then a VLM analyzes the recording. See VLA setup and commands.

Resources

Hugging Face hosts the 213-task table, shared in-context examples and compiled environments. Task schema · Download options

Repository structure

agent/vlm/             VLM clients and auditing tools
agent/vla/             VLA exploration and recording analysis
judge/                 GPT-6 judge and prompt
scripts/experiments/    Agent presets and experiment launcher
scripts/view_task.py    Interactive task viewer
scripts/               Downloads and environment launchers
environments/          Environment connections
data/                  Dataset loading and release manifests
tests/                 Automated checks

pip install -r requirements-dev.txt
python scripts/check_release.py
python -m pytest -q

Citation

@misc{jiang2026worldauditbench,
  title  = {WorldAuditBench: Interactive 3D World Auditing
            with Multimodal Agents},
  author = {Ziyan Jiang and Jingbo Yang and Jiabao Ji and
            Yujian Liu and Qiucheng Wu and Tommi Jaakkola and
            Yang Zhang and Shiyu Chang},
  year   = {2026},
  eprint = {2609.40325},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  url    = {https://arxiv.org/abs/2609.40325}
}

For questions or bug reports, please open an issue.