visual-recognition

Local, no-cloud object detection and lightweight video analytics for Apple Silicon (Metal / MPS). Two entrypoints:

• main.py — live webcam detection + multi-object tracking → JSON events on stdout, optional webhook, optional MQTT. Suitable for feeding a SIEM/EDR pipeline (e.g. LimaCharlie) or a home broker.
• survey.py — long-running vehicle + person traffic survey. Assigns a CLIP-based fingerprint to each pass, saves photos, rolls up hourly stats, and flags repeat visitors within a rolling time window.

Everything runs on-device. YOLO for detection + ByteTrack for association; CLIP ViT-B/32 for appearance embeddings in survey.py. Weights auto-download from GitHub on first run — no account, no API key, no telemetry back to the vendor.

---

Requirements

• macOS on Apple Silicon (Intel + Linux likely work but untested here)
• Python 3.10–3.12 (3.13 currently has a broken pyexpat on some macOS builds; 3.14 is too new for a couple of pinned transitive deps)
• A camera device visible to OpenCV, or a video file / URL
• ~200 MB free for models on first run (yolov8n.pt ~6 MB, CLIP ViT-B/32 ~350 MB if you run survey.py)

---

Install

git clone https://github.com/tekgrunt/visual-recognition.git
cd visual-recognition

python3.12 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

If your default python3 isn't 3.10–3.12, install one via Homebrew (brew install python@3.12) or miniforge and use that binary to create the venv.

First run needs to grant your terminal camera access under System Settings → Privacy & Security → Camera.

---

main.py — live detection + tracking

# stdout JSON events only
python main.py

# with an annotated preview window
RENDER=1 python main.py

# publish to an MQTT broker (topic: visrec/events)
MQTT_BROKER=localhost python main.py

# POST each event as JSON to a webhook (LimaCharlie, Splunk HEC, custom collector, ...)
WEBHOOK_URL=https://your.endpoint/collect python main.py

# swap in a larger / newer detector (auto-downloaded)
MODEL=yolov8s.pt python main.py     # bigger, more accurate
MODEL=yolo11n.pt python main.py     # newer generation

Output schema

One JSON line per tracked detection per frame:

{
  "ts": "2026-07-14T18:32:11.412+00:00",
  "hostname": "some-mac.local",
  "source": "0",
  "model": "yolov8n.pt",
  "frame_id": 142,
  "track_id": 7,
  "class_id": 0,
  "class": "person",
  "category": "person",
  "confidence": 0.8712,
  "bbox": [412.1, 201.3, 588.9, 712.4],
  "bbox_w": 176.8,
  "bbox_h": 511.1,
  "bbox_area": 90362.5,
  "bbox_cx": 500.5,
  "bbox_cy": 456.9,
  "bbox_aspect": 0.346
}

track_id is stable across frames until the object leaves the scene, so counting unique objects in a time window reduces to COUNT(DISTINCT track_id) WHERE class = 'person' on the event stream.

category is a coarse rollup (person / vehicle / animal / object) for downstream detection rules that don't want to enumerate COCO IDs.

Env vars (main.py)

Var	Default	Purpose
MODEL	yolov8n.pt	Any Ultralytics weights (yolov8s.pt, yolo11n.pt, …)
VIDEO_SOURCE	0	Device index (int) or file path / RTSP URL (str)
CONF	0.35	Minimum detection confidence
IMGSZ	640	Inference size. Try 320 for faster / lower-latency at the cost of small-object recall
RENDER	(off)	1 = show annotated preview window
WEBHOOK_URL	(off)	POSTs each event as JSON
MQTT_BROKER	(off)	Publishes each event to visrec/events
MQTT_PORT	1883	MQTT port

---

Sending events to LimaCharlie

LimaCharlie is a security infrastructure platform with a generic Webhook Adapter that accepts arbitrary JSON events and routes them through its detection & response engine. Because main.py already POSTs one JSON object per event, no code changes are needed.

1. Add the Webhook Adapter extension

In the LimaCharlie web UI:

1. Extensions → Adapter → Subscribe (if not already subscribed to the free adapter extension).
2. Adapters → Add Adapter → Type: webhook.

2. Configure the adapter

Fill in the adapter form:

Field	Value
Sensor Name	e.g. visrec-<hostname> — becomes the sensor these events show up under
Client Options → parse_format	json
Client Options → parse_type	evt_visrec (or any label you want to see in the timeline)
Client Options → secret	(optional but recommended) — a random shared secret required in the URL

Save. LimaCharlie will show you a URL of the form:

https://api.limacharlie.io/v1/ingest/webhook/<oid>/<uuid>?secret=<your-secret>

3. Point main.py at it

export WEBHOOK_URL="https://api.limacharlie.io/v1/ingest/webhook/<oid>/<uuid>?secret=<your-secret>"
python main.py

Events start appearing in the LimaCharlie Timeline under the sensor name you chose, tagged with the parse_type you set (evt_visrec).

4. Write a D&R rule (optional)

Once events are flowing, you can trigger on them from Detection & Response → Rules → New Rule. Example — alert on any vehicle with confidence ≥ 0.7:

detect:
  event: evt_visrec
  op: and
  rules:
    - op: is
      path: event/category
      value: vehicle
    - op: is greater than
      path: event/confidence
      value: 0.7

respond:
  - action: report
    name: visrec-vehicle-detected

Because every event carries hostname, class, track_id, and bbox, you can build zone rules, dwell-time rules, or unique-object-per-hour rules without changing the sensor.

Tip. LimaCharlie's adapter can be pointed at a local URL through the LC Sensor running on your Mac, or accept events from any host on the internet if you keep the secret param non-guessable. Rotate the secret in the adapter config if it leaks.

---

survey.py — vehicle + person traffic survey

Long-running pipeline. Watches a video source, tracks vehicles and people, and:

• Assigns each track a fingerprint — CLIP embedding + dominant color + coarse body type
• Saves best-crop + full-frame photo per pass
• Appends one CSV row per pass (data/vehicles/passes.csv, data/people/passes.csv)
• Rolls up hourly stats to data/{vehicles,people}/stats.jsonl
• Flags repeats — same fingerprint seen ≥ REPEAT_THRESHOLD prior times in a rolling window (2 h vehicles, 4 h people by default)
• Saves a montage image + metadata for each flagged repeat

Everything runs on-device. YOLO for detection + tracking, CLIP ViT-B/32 (openai) for body-type zero-shot classification and appearance embeddings — both on MPS when available.

Run

# default: uses webcam (source=0), writes to ./data/
python survey.py

# with preview window
RENDER=1 python survey.py

# feed a recorded traffic video instead of the webcam
VIDEO_SOURCE=/path/to/traffic.mp4 python survey.py

# tune the repeat trigger (default: >= 2 prior matches, so 3+ total passes fires it)
REPEAT_THRESHOLD=1 python survey.py    # fire on the 2nd pass
REPEAT_THRESHOLD=3 python survey.py    # only fire on the 4th+ pass

Output layout

data/
├── vehicles/
│   ├── passes.csv                  # every vehicle pass
│   ├── stats.jsonl                 # hourly rollup
│   ├── photos/YYYY-MM-DD/          # best crop + full frame per pass
│   └── repeats/YYYY-MM-DD/         # montage.jpg + .json for each flagged repeat
└── people/
    ├── passes.csv
    ├── stats.jsonl
    ├── photos/YYYY-MM-DD/
    └── repeats/YYYY-MM-DD/

Stdout also emits one JSON line per event (PASS, REPEAT, STATS_HOURLY) so you can filter live:

python survey.py | jq -c 'select(.type=="REPEAT")'

Expected accuracy

• Vehicle body type (sedan / SUV / pickup / van / box_truck / bus / motorcycle / bicycle): CLIP zero-shot works reasonably at 70–85 % on clear side-profile crops. Confuses SUV ↔ crossover and van ↔ box-truck at the margins.
• Color: 80–90 % on solid single-color vehicles from side view. Two-tones and metallic paints degrade it.
• Vehicle repeat detection: works when body type + color + CLIP embedding all align. Common combos (white sedan, black SUV) will occasionally false-positive across genuinely different vehicles. If noisy, raise SIM_THRESHOLD_VEHICLE from 0.85 → 0.90 for precision, or drop to 0.80 for recall.
• Person repeat detection: body-only ReID — will catch "same person, same session, same clothes" (jogger doing a loop, delivery driver walking back to the truck). Will not recognize the same person tomorrow in different clothes. That's a hard limit of appearance-based ReID without face recognition.

Env vars (survey.py)

Var	Default	Purpose
MODEL	yolov8n.pt	YOLO weights
VIDEO_SOURCE	0	Device index or file path / URL
DATA_DIR	./data	Output root
CONF	0.35	Minimum detection confidence
IMGSZ	640	Inference size
RENDER	(off)	1 = show preview window
TRACK_GRACE_SECONDS	2.0	Finalize a track after this many seconds of absence
MIN_TRACK_FRAMES	3	Discard tracks shorter than this
VEHICLE_WINDOW_HOURS	2.0	Rolling window for vehicle repeat detection
PERSON_WINDOW_HOURS	4.0	Rolling window for person repeat detection
REPEAT_THRESHOLD	2	Prior matches to trigger a REPEAT event
SIM_THRESHOLD_VEHICLE	0.85	Cosine similarity threshold between vehicle CLIP embeddings
SIM_THRESHOLD_PERSON	0.82	Same, for people
CLIP_MODEL	ViT-B-32	open_clip model name
CLIP_PRETRAINED	openai	open_clip pretrained tag

---

Privacy

Both scripts are 100 % local. There is no cloud inference, no analytics, no phone-home. main.py's webhook + MQTT sinks only fire if you set the env vars. survey.py writes photos and CSVs to disk under ./data/ — treat that directory as sensitive (it may contain images of identifiable people or license plates) and back it up / purge it accordingly. .gitignore already excludes data/ so accidental commits are unlikely.

---

License

AGPL-3.0 — see LICENSE.

This project depends on Ultralytics YOLO, which is AGPL-3.0. If you plan to ship this as closed-source or as a hosted SaaS to third parties, either purchase an Ultralytics Enterprise License or swap the detector for something permissive (RT-DETR from transformers, YOLO-NAS, D-FINE) and re-license accordingly.