LVMem: Image-Augmented Memory for Long-Horizon Question Answering

Reference implementation of LVMem. Runs as a Docker container and talks to an OpenAI-compatible inference backend.

The inference backend can be either:

1. Local SGLang servers (GPU required), or
2. Any external OpenAI-compatible API (e.g. OpenRouter).

Only the lvmem container is always required; the SGLang containers are optional and gated behind a Compose profile.

---

Quick start

Clone this repository.

0. Fetch ReMEmbR sources

LVMem imports utils and tools from ReMEmbR:

git clone --filter=blob:none --sparse https://github.com/NVIDIA-AI-IOT/remembr.git
cd remembr
git sparse-checkout set remembr/utils remembr/tools

mv remembr/utils ./utils
mv remembr/tools ./tools
rmdir remembr
cd ..

1. Prepare the environment file

Enter the docker/ directory.

cp .env.example .env

Edit .env if needed. In particular, set UID / GID to match your host user:

echo "UID=$(id -u)" >> .env
echo "GID=$(id -g)" >> .env

2. Start Milvus (vector database)

LVMem uses Milvus as its vector store, following the setup from ReMEmbR.

curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh \
  -o launch_milvus_container.sh
bash launch_milvus_container.sh start

3. Choose a backend

Note: the SGLang backend is tested end-to-end. The OpenRouter backend is supported by the code but has not been re-tested after refactoring. Use it with that in mind.

Option A — External API (no GPU)

Set in .env:

VLM_ENDPOINT=https://openrouter.ai/api/v1
JUDGE_ENDPOINT=https://openrouter.ai/api/v1
OPENAI_API_KEY=sk-xxx
VLM_MODEL=qwen/qwen3.5-9b
JUDGE_MODEL=qwen/qwen3.5-9b

Run only the method container:

docker compose up lvmem -d --build
docker compose exec lvmem bash

Option B — Local SGLang (GPU)

Leave the default endpoints in .env. Adjust VLM_CUDA_VISIBLE_DEVICES / JUDGE_CUDA_VISIBLE_DEVICES to select the GPUs to use.

Run the full stack:

docker compose --profile local up -d --build
docker compose exec lvmem bash

This starts:

Service	Purpose	Default port
sglang-lvmem-vlm	Vision-language-model backend	8100
sglang-lvmem-judge	Judge backend	8101
lvmem	Method container	—

Model IDs differ between backends: SGLang uses HuggingFace-style paths (Qwen/Qwen3.5-9B), OpenRouter uses lowercase slugs (qwen/qwen3.5-9b).

---

Configuration

All configuration is read from .env (see .env.example).

Variable	Description	Default
UID / GID	Host user UID/GID, passed as Docker build args	1000 / 1000
VLM_ENDPOINT	OpenAI-compatible endpoint for the VLM backend	http://localhost:8100/v1
JUDGE_ENDPOINT	OpenAI-compatible endpoint for the judge backend	http://localhost:8101/v1
OPENAI_API_KEY	API key for external backends	—
VLM_MODEL	Model path or ID for the VLM	Qwen/Qwen3.5-9B
JUDGE_MODEL	Model path or ID for the judge	Qwen/Qwen3.5-0.8B
VLM_CUDA_VISIBLE_DEVICES	GPU index for the VLM container	0
JUDGE_CUDA_VISIBLE_DEVICES	GPU index for the judge container	0
HF_TOKEN	HuggingFace token (for gated models)	—

---

HuggingFace cache

We recommend mounting a shared HuggingFace cache directory so that downloaded models are reused across containers and runs. Create a local docker-compose.override.yaml:

services:
  lvmem:
    volumes:
      - ./hf_cache:/home/user/.cache/huggingface
  sglang-lvmem-vlm:
    volumes:
      - ./hf_cache:/root/.cache/huggingface
  sglang-lvmem-judge:
    volumes:
      - ./hf_cache:/root/.cache/huggingface

Then start the stack as usual — Compose picks up the override automatically.

---

Evaluation

See EVAL.md for dataset download, memory building, question generation, and the full-sequence evaluation pipeline.

---

Third-party code

This project includes code from several third-party sources.

• DAAAM, copyright (c) 2026 Massachusetts Institute of Technology, licensed under the BSD 3-Clause License (see LICENSE-DAAAM).
• ReMEmbR, licensed under the NVIDIA License, non-commercial / research use only (see LICENSE-ReMEmbR).
• PSFR (key-frame-selection), obtained directly from the authors of FocusGraph upon request. Not our code; used as provided by the authors (see LICENSE-PSFR).