AGILE

Hand-Object Interaction Reconstruction from Video via Agentic Generation

Prepared data · Generated objects · Optimized results · Installation

Jin-Chuan Shi^1*, Binhong Ye^1*, Tao Liu¹, Xiaoyang Liu¹, Yangjinhui Xu¹, Junzhe He¹, Zeju Li¹, Hao Chen¹, Chunhua Shen^1,2

¹State Key Lab of CAD & CG, Zhejiang University   ²Zhejiang University of Technology

^*Equal contribution

ACM SIGGRAPH 2026, Conference Track

News

• [2026.10] AGILE code and data released! Dataset configurations, preprocessed inputs and generated assets are being released progressively, starting with HO3D.
• [2026.03] AGILE is conditionally accepted to ACM SIGGRAPH 2026 (Conference Track)!
• [2026.03] Project page and paper released.

Overview

AGILE turns monocular videos of hand-object interaction into textured 3D objects and reconstructed hand-object motion. It combines agentic object generation with contact-anchored tracking, providing a workflow from video frames to meshes, motion, quantitative evaluation and interactive visualization.

Highlights

• Generate, review and refine. A vision-language model selects complementary reference frames, reviews candidate geometry and appearance, and uses visual feedback to guide revisions. Run the workflow through configurable VLM and image-generation APIs, or use the optional Codex integration. Geometry and texture are reviewed separately, so texture refinement can preserve the approved shape and UV layout.
• Two routes to object geometry. Use multiview-to-3D generation, or let Codex construct a procedural mesh directly from reference images without a separate 3D generative model. Both routes support textured asset delivery; the Codex route uses texture images supplied through the agent client.
• Track from a contact anchor. Initialize object pose and scale at a contact frame, then track forward and backward. Silhouettes and local image features guide object motion, while contact-aware hand-object relative-motion constraints help stabilize the interaction.
• Preserve detail while optimizing efficiently. Use a lightweight mesh proxy for optimization and retain the selected textured mesh for visualization. Recovered poses and scale drive the display asset, keeping its appearance separate from the tracking workload.

Modules

The modules can be used together or independently. Start with the released prepared data to run reconstruction, or use the generation and preprocessing tools to prepare new inputs.

Module	What it does	Outputs
Preprocessing	Takes RGB frames, hand/object masks and a selected object mesh; runs hand and depth estimation, aligns the hand to depth, initializes object poses and checks frame consistency.	Depth, hand estimates, camera parameters and initial object poses.
Object generation	Selects reference views, builds and reviews candidates, prepares UVs, refines textures and exports the selected asset. Includes the paper's VLM/API workflow, Codex-assisted generation, Codex procedural geometry and external-asset workflows.	Textured GLB/OBJ meshes, materials, textures and candidate review records.
Motion optimization	Refines hand translation, object pose and initial object scale with the released configuration. Tracks in both directions and tests prepared-pose candidates when silhouette overlap collapses.	Per-frame hand geometry, object transforms and scale.
Evaluation	Measures hand accuracy, object shape and hand-object interaction with explicit frame coverage and metric protocols.	MPJPE, object Chamfer distance and F-scores, wrist-relative and hand-aligned interaction Chamfer distances.
Export and visualization	Packages reconstruction results and assets; provides interactive 3D playback and camera-aligned renders and overlays.	Portable result bundles with meshes, joints, poses, camera calibration and frame IDs; images and animations.

Code and Data

The AGILE code is available in this repository. Prepared data, generated assets and optimized reconstructions are hosted on Hugging Face.

The current data release includes HO3D configurations, preprocessed inputs and generated objects for 18 sequences. The downloader retrieves the matching components for each sequence and verifies their checksums, so you can start optimization without running preprocessing or generation. Additional dataset configurations and prepared results will be released progressively. See prepared data for sequence combinations and the data manifest for checksums.

Optimized results for all 18 HO3D sequences are also available. These sequence checkpoints contain the reconstructed hands, object poses and scale, meshes, camera data and evaluation metrics, ready for visualization and downstream use. See download and evaluate results.

HO3D Results

Result	MPJPE (mm) ↓	Object CD (cm²) ↓	F@5 (%) ↑	F@10 (%) ↑	Interaction CDₕ (cm²) ↓
Paper, Table 1	3.92	0.27	86.63	97.77	15.81
Released code and data	3.9453	0.2710	84.30	98.52	15.7669

Release scores average all 18 sequences equally over the full available prepared range (3,590 sampled frames), using the evaluation script included in this repository. Interaction CDₕ measures object error relative to each hand's wrist; hand-aligned similarity-transform scores are reported separately. Prepared frame coverage and object-registration details differ from the original paper evaluation, so the paper row is a reference rather than an exact reproduction claim. Per-sequence results and protocol.

Getting Started

Clone the repository and run the following commands from its root. Replace /ABS/... with absolute paths on your machine.

git clone https://github.com/aim-uofa/AGILE.git
cd AGILE

1. Follow the installation guide.

2. Download a prepared sequence:

   python tools/download_ho3d.py --scenes MC1 --output /ABS/data

3. Follow the optimization guide to reconstruct, evaluate and visualize the interaction.

To view completed reconstructions without running optimization, follow download and visualize results. Downloads are public and do not require a Hugging Face account.

To prepare your own inputs, provide RGB frames and hand/object masks, choose a generation route, then follow the HO3D preprocessing guide. Generation accepts user-supplied frames; a complete in-the-wild video workflow with dedicated configurations and examples is planned. Generation uses Conda and provides doctor, generate, status and resume commands for setup and execution. See release notes for updates and third-party notices for dependencies.

TODO

• Release the preprocessing, object generation and optimization code.
• Release evaluation, result packaging and visualization tools.
• Release HO3D configurations, preprocessed inputs and generated object assets.
• Release validated HO3D optimized reconstructions and per-sequence metrics.
• Release DexYCB optimization and evaluation configurations.
• Release DexYCB preprocessed inputs and generated object assets.
• Release the in-the-wild video workflow, including frame and mask preparation, hand/depth alignment, object initialization, optimization configurations and an end-to-end example.
• Release the retargeting module with documentation and examples.

License

AGILE is available for non-commercial research use under the AGILE license. For commercial use, contact Professor Chunhua Shen.

Third-party code, dependencies, model weights and datasets follow their original licenses. See third-party notices.

Citation

If you find this work useful, please consider citing:

@article{shi2026agile,
  title={AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation},
  author={Shi, Jin-Chuan and Ye, Binhong and Liu, Tao and Liu, Xiaoyang and Xu, Yangjinhui and He, Junzhe and Li, Zeju and Chen, Hao and Shen, Chunhua},
  journal={arXiv preprint arXiv:2602.04672},
  year={2026}
}

Acknowledgements

We thank the authors of HOLD, MagicHOI, WiLoR, FoundationPose, MoGe, MegaSAM, and Viser for their excellent work.