Skip to content
DerrickUnleashedPublic

About

Resources

Code of conduct

Contributing

Security policy

Stars

3 stars

Watchers

0 watching

Forks

Latest commit

 

History

806 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Visibility-aware, self-supervised identity learning for long-term fruit tracking and counting in mango orchards.

Pan-tilt-zoom cameras make fruit appear, disappear, and re-appear as they sway in the wind, occlude each other, and move across the view. OrchardVision learns per-fruit identity descriptors from a detection stream and uses them to recover identities after occlusion, so counts stay correct over long video runs.

Repository layout

apps/api      FastAPI backend (uv workspace; package `app`)
apps/web      React + Vite + Tailwind frontend (npm workspace `web`)
packages/types Shared TypeScript contracts (npm workspace `types`)
ml            `orchardvision_ml` python package (uv workspace)
data          Local runtime data (uploads, processed, DB) — gitignored
docker        Container configuration

Quickstart

Backend

cp .env.example .env            # adjust as needed
uv run --project apps/api uvicorn app.main:app --port 8001 \
    --app-dir apps/api         # run from the repository root
  • Interactive docs: http://localhost:8001/api/docs
  • Health check: http://localhost:8001/api/health

Real (non-mock) model training additionally needs torch. It is an optional extra rather than a hard dependency, so install it only when training:

uv sync --project ml --extra torch

Mock mode, evaluation, and the API/Celery runtime do not require torch; the torch-gated tests skip cleanly when it is absent.

Frontend

cd apps/web
npm install
npm run dev                     # http://localhost:5173

Tests & checks

uv run --project ml pytest ml/tests -q                 # ML suite
uv run --project apps/api pytest apps/api/tests -q     # API suite
uv run --project ml ruff check ml                      # ML lint
uv run --project apps/api ruff check apps/api          # API lint
cd apps/web && npm run typecheck && npm run lint && npm run build

Testing

The full verified matrix runs on every push/PR via GitHub Actions (.github/workflows/ci.yml). Locally, use the same commands:

# Backend + ML (repo root)
uv run --project apps/api pytest apps/api/tests -q
uv run --project ml pytest ml/tests -q

# API lint (from apps/api)
uv run --project apps/api ruff check app tests
uv run --project apps/api ruff format --check app tests

# Frontend (from apps/web)
npm run typecheck && npm run lint && npm run test && npm run build && npm run format:check

Architecture

Processing runs per video in a background thread:

  1. Detection — mock detector emits fruit boxes per frame; real detectors plug in behind the same interface (orchardvision_ml.detection).
  2. Camera motion compensation — estimates pan/tilt drift and shifts boxes before tracking (static/drift).
  3. Tracking — mock identity-primitive tracker (configurable association distance + recovery buffer), a ByteTrack-style adapter, or the optional TAPNext++ point tracker (orchardvision_ml.tracking; implementation notes in docs/tapnext-tracker.md).
  4. Identity embedding + memory — quality-aware prototypes per fruit with online prototype updates (orchardvision_ml.identity.embedding/memory).
  5. Re-identification — after a track loss, queries identity memory to recover the same fruit (orchardvision_ml.reid).
  6. Evaluation harness — scores trajectories against scenario ground truth: IDF1, HOTA, ID switches, fragmentation, count MAE/MAPE, identity recovery rate (orchardvision_ml.evaluation).

All per-video artifacts (tracks, identities, identity observations, re-id events, per-frame detections, camera-motion estimate, analytics aggregates) are persisted and re-computed on reprocessing; each pipeline level is mock-first so the whole system runs without heavy model weights.

API summary

Endpoint Purpose
GET /api/health Service status
POST /api/videos Upload video (MIME/extension/size validated)
GET /api/videos · GET /api/videos/{id} · DELETE /api/videos/{id} Video CRUD
POST /api/videos/{id}/process Start processing (mock/real, tracker, camera motion)
GET /api/jobs/{job_id} Job status, stage, and progress
GET /api/videos/{id}/tracks Track trajectories
GET /api/videos/{id}/identities(/observations) Identity prototypes + observation history
GET /api/videos/{id}/reid-events Re-identification events
GET /api/videos/{id}/tracks/{track_id} Single track trajectory
GET /api/videos/{id}/frames/{frame_id}?width= On-demand JPEG frame (thumbnails)
GET /api/videos/{id}/file Streamed source video (range requests for seeking)
GET /api/videos/{id}/detections?frame_id= Per-frame persisted detections
GET /api/videos/{id}/motion · GET /api/videos/{id}/analytics Motion estimate + aggregates
GET/POST /api/experiments, GET/DELETE /api/experiments/{id}, GET /api/experiments/{id}/export Evaluation experiment runs (JSON/CSV export)
GET/PUT /api/settings Persisted pipeline defaults

Full schemas are in apps/api/app/schemas/video.py.

Configuration

See .env.example for the authoritative list. Key variables:

  • DATABASE_URL — default SQLite at data/orchardvision.db
  • UPLOAD_DIR, PROCESSED_DIR — storage locations
  • MAX_UPLOAD_BYTES, ALLOWED_VIDEO_MIME_TYPES — upload constraints
  • ALLOWED_ORIGINS, CORS_ALLOW_CREDENTIALS — CORS
  • AUTO_CREATE_TABLES — dev-friendly create_all; set false and use alembic up for managed schemas, see below

Persisted pipeline defaults (GET/PUT /api/settings) control processing defaults and video-page visualization:

  • confidenceThreshold, reidThreshold, memoryUpdateRate, occlusionTimeoutSec
  • cameraMotionMode — static or drift simulation
  • trackerMatchThreshold (px) and trackerBufferFrames — mock tracker association distance and how long a track survives a missed detection
  • showTrackVisualizer, showIdentitiesPanel, showReidPanel, showMotionPanel, showAnalyticsPanel — which panels appear on the video detail page

Per-run POST /api/videos/{id}/process options override these; unset options fall back to the persisted defaults.

Database migrations

Alembic is wired (apps/api/alembic). Build the schema on a fresh database:

cd apps/api
alembic upgrade head

Existing dev databases created with create_all can be aligned with alembic stamp head or recreated.

In the container, migrations are applied automatically at API startup (the worker skips this deliberately, since both share one SQLite file). Databases predating migrations are detected by their missing alembic history and stamped rather than upgraded, so an existing deployment is adopted instead of failing with table ... already exists.

Production notes

  • Start the API with the production entrypoint, which reads ENVIRONMENT, PORT, WEB_CONCURRENCY, and HOST:

    ENVIRONMENT=production PORT=8001 WEB_CONCURRENCY=2 uv run --project apps/api python -m app

    or run uvicorn app.main:app --workers N directly. Place behind a reverse proxy (Caddy/nginx) for TLS.

  • Job queue: by default videos process in-process via Starlette BackgroundTasks. Set CELERY_BROKER_URL/CELERY_RESULT_BACKEND to dispatch through a Celery worker instead (identical DB-backed status/progress). The full stack is in docker/docker-compose.yml:

    docker compose -f docker/docker-compose.yml up --build
  • Mock inference is the default; real models (YOLO, embedding net) swap in via the detection/embedding interfaces.

See TODO.md for the full phase map and known technical debt.

TAPNext++ point tracker (optional)

A second short-term tracker backed by Google DeepMind's TAPNext++ running recurrently under the hood, with the model code vendored and torch kept out of the default install. How it works, why the query set is fixed per stream, and how to extend it are documented in docs/tapnext-tracker.md. To use it:

uv sync --project ml --extra tapnext
python scripts/download_tapnext_weights.py          # -> models/tapnextpp_ckpt.pt
python scripts/run_tapnext_tracker.py <video> --model-path models/yolov8n.pt

Its tests are in ml/tests/test_tapnext.py (mock-model tests run without torch; RUN_TORCH_TESTS=1 exercises a tiny real model).

Contributing

See CONTRIBUTING.md for setup, development workflow, and commit/PR conventions. Report security issues per SECURITY.md. Participation is governed by CODE_OF_CONDUCT.md. Repo-wide editor and lint defaults live in .editorconfig, .pre-commit-config.yaml, and .github/dependabot.yml; code ownership is enforced via .github/CODEOWNERS.

License

Proprietary — all rights reserved. See LICENSE.

About

Resources

Code of conduct

Contributing

Security policy

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages