Skip to content
muleinaPublic

About

XAVAD: Guided Zero-Shot Anomaly Explanation for Anonymized Crowd Video Monitoring: Context, Categorization, and Spatial Localization.

Topics

Resources

Stars

1 star

Watchers

1 watching

Forks

Repository files navigation

XAVAD CI Anaconda
XAVAD NB Open XAVAD in Colab XAVAD DEMO XAVAD PAPER XAVAD PAPER Supplementary

XAVAD: Zero-Shot Anomaly XAI for Anonymized Video Anomaly Detection

Official implementation of "Guided Zero-Shot Anomaly Explanation for Anonymized Crowd Video Monitoring: Context, Categorization, and Spatial Localization"

The XAVAD offers zero-shot multi-facet anomaly XAI for anonymized (AN) video anomaly detection (VAD).

System highlight: a unified, explainable, and full-body privacy-protecting VAD system.

XAVAD System Design

XAVAD Highlights

  • Explainable and privacy-protecting VAD will build transparency and trust in safety and security monitoring computer vision systems.
  • XAVAD presents unified zero-shot methods for anomaly type categorization, visual localization, and scene context description.
  • Proposes open-vocabulary segmentation (GDSN) that mitigates weapon occlusions during full-masking anonymization.
  • Proposes LLM-assisted prompting with pool-scoring mechanisms over video frames for improved alert-type identification using CLIP.
  • Introduces a novel prompt-guided patching autofocus (PPA) mechanism for Grad-ECLIP to generate heatmaps for visual alert localization, with substantial computation and accuracy gains.
  • Proposes curated VAD-AN-XAI-aware prompt engineering for VLMs to generate anomaly descriptions robustly.
  • Demonstrates a considerable gain over SOTA approaches.
  • Delivers insights into privacy protection trade-offs in XAI for VAD.

XAVAD focuses on XAI for anonymized VAD, and a detailed study on anonymization and VAD can be found in LA3D (repo, paper).

System Design of XAVAD

XAVAD System Pipeline Diagram: Initial Setup Stage

XAVAD System Pipeline Diagram: Initial Stage

XAVAD System Pipeline Diagram: Inference Stage

XAVAD System Pipeline Diagram: Inference Stage

Table of Contents

Resources

We thank and credit the public datasets and source codes listed below.

Installation

  1. Clone repository:

    git clone https://github.com/muleina/XAVAD.git && cd XAVAD
    
  2. Install dependencies:Ollama, following Windowws, macOs, and Linux.

  3. Recommend setting up and installing CUDA PyTorch with Anaconda following the PyTorch installation instructions.

  4. Anaconda install options using YAML

    conda env create -f environment.yaml
    
  5. Alternatively, install using setup.py in an existing environment:

    pip install -e .
    

    or:

    pip install git+https://github.com/muleina/XAVAD.git
    

The setup configurations of the XAVAD are given in the config/config.py file.

Notebooks

We provide notebooks for step-wise demos of the XAVAD pipelines using different models and data sources.

  • [Zero-Shot Occlusion Robust Anonymization] Open XAVAD in Colab
  • [Zero-Shot Scalable with Efficiency Anomaly Heatmap Localization ] Open XAVAD in Colab
  • [Zero-Shot VAD XAI: End-to-End (XAVAD) on VAD Data] XAVAD NB Open XAVAD in Colab
  • [Zero-Shot VAD XAI: End-to-End (XAVAD) on Synthetic Demo Data] XAVAD NB Open XAVAD in Colab

Anomaly XAI Results

Cross-dataset empirical and qualitative evaluations demonstrate that XAVAD has significantly outperformed existing SOTA benchmark approaches.

Zero-Shot Anomaly Categorization

Our LLM-Assisted Prompting on CLIP enhances the performance of anomaly type classification.

Ablation study on Maximum Score Pooling and LLM-Assisted Prompting.

Zero-Shot Anomaly Heatmap Localization

Anomaly heatmap localization using Grad-ECLIP, improved with our Prompt-Guided Patched Autofocus (PPA) approach.

ZSAM: Simuletic Dataset

Anomaly localization for weapon occlusion after AN, improved with our Zero-Shot Person Segmentation Model (GDSN).

AN-ZSAM: Simuletic Dataset

AN-ZSAM: Simuletic Dataset

Zero-Shot Anomaly Context Description

Anomaly context description using lightweight VLM models, leveraged with our structured prompting mechanism.

ZSAD: HIVAU-70k Dataset

Computational Cost Analysis

GPU time and memory cost on a system with 32 AMD EPYC Milan Processors, 64 GB RAM, and a 40 GB NVIDIA A100D-40C.

  • Our PPA achieves $4 \times$ memory efficiency and 30% faster time while improving ZSAM AP by $4\text{--}6 \times$ compared to Grad-ECLIP.
  • In contrast, simple patching with Grad-ECLIP increases the cost by $1.8 \times$ and $6.8 \times$, respectively.
  • YOLO is the most memory-efficient, and RF-DETR is the fastest among the AN mask generation models.
  • Despite substantial accuracy gains of the GDSN in ZSAM, it has a relatively higher memory cost (due to large SAM models), while the difference in speed remains marginal compared to YOLO-X.
  • The InternVL3.5 VLMs maintain fast inference and promising ZSAD accuracy.

ZS XAI and AN Computational Cost

BibTeX Citation

If you employ any part of the study or the code, please kindly cite the following paper:

@article{asres20266xavd,
  title={Guided Zero-Shot Anomaly Explanation for Anonymized Crowd Video Monitoring: Context, Categorization, and Spatial Localization},
  author={Asres, Mulugeta Weldezgina and Jiao, Lei and Omlin, Christian Walter},
  journal={arXiv:2610.32073},
  year={2026}
}

About

XAVAD: Guided Zero-Shot Anomaly Explanation for Anonymized Crowd Video Monitoring: Context, Categorization, and Spatial Localization.

Topics

Resources

Stars

1 star

Watchers

1 watching

Forks

Contributors

Languages