Official implementation of "Guided Zero-Shot Anomaly Explanation for Anonymized Crowd Video Monitoring: Context, Categorization, and Spatial Localization"
The XAVAD offers zero-shot multi-facet anomaly XAI for anonymized (AN) video anomaly detection (VAD).
System highlight: a unified, explainable, and full-body privacy-protecting VAD system.
- Explainable and privacy-protecting VAD will build transparency and trust in safety and security monitoring computer vision systems.
- XAVAD presents unified zero-shot methods for anomaly type categorization, visual localization, and scene context description.
- Proposes open-vocabulary segmentation (GDSN) that mitigates weapon occlusions during full-masking anonymization.
- Proposes LLM-assisted prompting with pool-scoring mechanisms over video frames for improved alert-type identification using CLIP.
- Introduces a novel prompt-guided patching autofocus (PPA) mechanism for Grad-ECLIP to generate heatmaps for visual alert localization, with substantial computation and accuracy gains.
- Proposes curated VAD-AN-XAI-aware prompt engineering for VLMs to generate anomaly descriptions robustly.
- Demonstrates a considerable gain over SOTA approaches.
- Delivers insights into privacy protection trade-offs in XAI for VAD.
XAVAD focuses on XAI for anonymized VAD, and a detailed study on anonymization and VAD can be found in LA3D (repo, paper).
XAVAD System Pipeline Diagram: Initial Setup Stage
XAVAD System Pipeline Diagram: Inference Stage
Table of Contents
We thank and credit the public datasets and source codes listed below.
-
Data sets
-
Models:
-
Clone repository:
git clone https://github.com/muleina/XAVAD.git && cd XAVAD -
Install dependencies:
Ollama, following Windowws, macOs, and Linux. -
Recommend setting up and installing CUDA PyTorch with Anaconda following the PyTorch installation instructions.
-
Anaconda install options using YAML
conda env create -f environment.yaml -
Alternatively, install using
setup.pyin an existing environment:pip install -e .or:
pip install git+https://github.com/muleina/XAVAD.git
We provide notebooks for step-wise demos of the XAVAD pipelines using different models and data sources.
- [Zero-Shot Occlusion Robust Anonymization]
- [Zero-Shot Scalable with Efficiency Anomaly Heatmap Localization ]
- [Zero-Shot VAD XAI: End-to-End (XAVAD) on VAD Data]
- [Zero-Shot VAD XAI: End-to-End (XAVAD) on Synthetic Demo Data]
Cross-dataset empirical and qualitative evaluations demonstrate that XAVAD has significantly outperformed existing SOTA benchmark approaches.
Our LLM-Assisted Prompting on CLIP enhances the performance of anomaly type classification.
Ablation study on Maximum Score Pooling and LLM-Assisted Prompting.
Anomaly heatmap localization using Grad-ECLIP, improved with our Prompt-Guided Patched Autofocus (PPA) approach.
Anomaly localization for weapon occlusion after AN, improved with our Zero-Shot Person Segmentation Model (GDSN).
Anomaly context description using lightweight VLM models, leveraged with our structured prompting mechanism.
GPU time and memory cost on a system with 32 AMD EPYC Milan Processors, 64 GB RAM, and a 40 GB NVIDIA A100D-40C.
- Our PPA achieves
$4 \times$ memory efficiency and 30% faster time while improving ZSAM AP by$4\text{--}6 \times$ compared to Grad-ECLIP. - In contrast, simple patching with Grad-ECLIP increases the cost by
$1.8 \times$ and$6.8 \times$ , respectively. - YOLO is the most memory-efficient, and RF-DETR is the fastest among the AN mask generation models.
- Despite substantial accuracy gains of the GDSN in ZSAM, it has a relatively higher memory cost (due to large SAM models), while the difference in speed remains marginal compared to YOLO-X.
- The InternVL3.5 VLMs maintain fast inference and promising ZSAD accuracy.
If you employ any part of the study or the code, please kindly cite the following paper:
@article{asres20266xavd,
title={Guided Zero-Shot Anomaly Explanation for Anonymized Crowd Video Monitoring: Context, Categorization, and Spatial Localization},
author={Asres, Mulugeta Weldezgina and Jiao, Lei and Omlin, Christian Walter},
journal={arXiv:2610.32073},
year={2026}
}









