Visibility-aware, self-supervised identity learning for long-term fruit tracking and counting in mango orchards.
Pan-tilt-zoom cameras make fruit appear, disappear, and re-appear as they sway in the wind, occlude each other, and move across the view. OrchardVision learns per-fruit identity descriptors from a detection stream and uses them to recover identities after occlusion, so counts stay correct over long video runs.
apps/api FastAPI backend (uv workspace; package `app`)
apps/web React + Vite + Tailwind frontend (npm workspace `web`)
packages/types Shared TypeScript contracts (npm workspace `types`)
ml `orchardvision_ml` python package (uv workspace)
data Local runtime data (uploads, processed, DB) — gitignored
docker Container configuration
cp .env.example .env # adjust as needed
uv run --project apps/api uvicorn app.main:app --port 8001 \
--app-dir apps/api # run from the repository root- Interactive docs:
http://localhost:8001/api/docs - Health check:
http://localhost:8001/api/health
Real (non-mock) model training additionally needs torch. It is an optional extra rather than a hard dependency, so install it only when training:
uv sync --project ml --extra torchMock mode, evaluation, and the API/Celery runtime do not require torch; the torch-gated tests skip cleanly when it is absent.
cd apps/web
npm install
npm run dev # http://localhost:5173uv run --project ml pytest ml/tests -q # ML suite
uv run --project apps/api pytest apps/api/tests -q # API suite
uv run --project ml ruff check ml # ML lint
uv run --project apps/api ruff check apps/api # API lint
cd apps/web && npm run typecheck && npm run lint && npm run buildThe full verified matrix runs on every push/PR via GitHub Actions
(.github/workflows/ci.yml). Locally, use the same commands:
# Backend + ML (repo root)
uv run --project apps/api pytest apps/api/tests -q
uv run --project ml pytest ml/tests -q
# API lint (from apps/api)
uv run --project apps/api ruff check app tests
uv run --project apps/api ruff format --check app tests
# Frontend (from apps/web)
npm run typecheck && npm run lint && npm run test && npm run build && npm run format:checkProcessing runs per video in a background thread:
- Detection — mock detector emits fruit boxes per frame; real detectors plug
in behind the same interface (
orchardvision_ml.detection). - Camera motion compensation — estimates pan/tilt drift and shifts boxes
before tracking (
static/drift). - Tracking — mock identity-primitive tracker (configurable association
distance + recovery buffer), a ByteTrack-style adapter, or the optional
TAPNext++ point tracker
(
orchardvision_ml.tracking; implementation notes indocs/tapnext-tracker.md). - Identity embedding + memory — quality-aware prototypes per fruit with
online prototype updates (
orchardvision_ml.identity.embedding/memory). - Re-identification — after a track loss, queries identity memory to recover
the same fruit (
orchardvision_ml.reid). - Evaluation harness — scores trajectories against scenario ground truth:
IDF1, HOTA, ID switches, fragmentation, count MAE/MAPE, identity recovery
rate (
orchardvision_ml.evaluation).
All per-video artifacts (tracks, identities, identity observations, re-id events, per-frame detections, camera-motion estimate, analytics aggregates) are persisted and re-computed on reprocessing; each pipeline level is mock-first so the whole system runs without heavy model weights.
| Endpoint | Purpose |
|---|---|
GET /api/health |
Service status |
POST /api/videos |
Upload video (MIME/extension/size validated) |
GET /api/videos · GET /api/videos/{id} · DELETE /api/videos/{id} |
Video CRUD |
POST /api/videos/{id}/process |
Start processing (mock/real, tracker, camera motion) |
GET /api/jobs/{job_id} |
Job status, stage, and progress |
GET /api/videos/{id}/tracks |
Track trajectories |
GET /api/videos/{id}/identities(/observations) |
Identity prototypes + observation history |
GET /api/videos/{id}/reid-events |
Re-identification events |
GET /api/videos/{id}/tracks/{track_id} |
Single track trajectory |
GET /api/videos/{id}/frames/{frame_id}?width= |
On-demand JPEG frame (thumbnails) |
GET /api/videos/{id}/file |
Streamed source video (range requests for seeking) |
GET /api/videos/{id}/detections?frame_id= |
Per-frame persisted detections |
GET /api/videos/{id}/motion · GET /api/videos/{id}/analytics |
Motion estimate + aggregates |
GET/POST /api/experiments, GET/DELETE /api/experiments/{id}, GET /api/experiments/{id}/export |
Evaluation experiment runs (JSON/CSV export) |
GET/PUT /api/settings |
Persisted pipeline defaults |
Full schemas are in apps/api/app/schemas/video.py.
See .env.example for the authoritative list. Key variables:
DATABASE_URL— default SQLite atdata/orchardvision.dbUPLOAD_DIR,PROCESSED_DIR— storage locationsMAX_UPLOAD_BYTES,ALLOWED_VIDEO_MIME_TYPES— upload constraintsALLOWED_ORIGINS,CORS_ALLOW_CREDENTIALS— CORSAUTO_CREATE_TABLES— dev-friendlycreate_all; setfalseand usealembic upfor managed schemas, see below
Persisted pipeline defaults (GET/PUT /api/settings) control processing defaults
and video-page visualization:
confidenceThreshold,reidThreshold,memoryUpdateRate,occlusionTimeoutSeccameraMotionMode—staticordriftsimulationtrackerMatchThreshold(px) andtrackerBufferFrames— mock tracker association distance and how long a track survives a missed detectionshowTrackVisualizer,showIdentitiesPanel,showReidPanel,showMotionPanel,showAnalyticsPanel— which panels appear on the video detail page
Per-run POST /api/videos/{id}/process options override these; unset options
fall back to the persisted defaults.
Alembic is wired (apps/api/alembic). Build the schema on a fresh database:
cd apps/api
alembic upgrade headExisting dev databases created with create_all can be aligned with
alembic stamp head or recreated.
In the container, migrations are applied automatically at API startup (the
worker skips this deliberately, since both share one SQLite file). Databases
predating migrations are detected by their missing alembic history and stamped
rather than upgraded, so an existing deployment is adopted instead of failing
with table ... already exists.
-
Start the API with the production entrypoint, which reads
ENVIRONMENT,PORT,WEB_CONCURRENCY, andHOST:ENVIRONMENT=production PORT=8001 WEB_CONCURRENCY=2 uv run --project apps/api python -m app
or run
uvicorn app.main:app --workers Ndirectly. Place behind a reverse proxy (Caddy/nginx) for TLS. -
Job queue: by default videos process in-process via Starlette
BackgroundTasks. SetCELERY_BROKER_URL/CELERY_RESULT_BACKENDto dispatch through a Celery worker instead (identical DB-backed status/progress). The full stack is indocker/docker-compose.yml:docker compose -f docker/docker-compose.yml up --build
-
Mock inference is the default; real models (YOLO, embedding net) swap in via the detection/embedding interfaces.
See TODO.md for the full phase map and known technical debt.
A second short-term tracker backed by Google DeepMind's TAPNext++ running
recurrently under the hood, with the model code vendored and torch kept out of
the default install. How it works, why the query set is fixed per stream, and
how to extend it are documented in
docs/tapnext-tracker.md. To use it:
uv sync --project ml --extra tapnext
python scripts/download_tapnext_weights.py # -> models/tapnextpp_ckpt.pt
python scripts/run_tapnext_tracker.py <video> --model-path models/yolov8n.ptIts tests are in ml/tests/test_tapnext.py (mock-model tests run without
torch; RUN_TORCH_TESTS=1 exercises a tiny real model).
See CONTRIBUTING.md for setup, development workflow, and
commit/PR conventions. Report security issues per SECURITY.md.
Participation is governed by CODE_OF_CONDUCT.md.
Repo-wide editor and lint defaults live in .editorconfig,
.pre-commit-config.yaml, and .github/dependabot.yml; code ownership is
enforced via .github/CODEOWNERS.
Proprietary — all rights reserved. See LICENSE.