slop

Tell whether a piece of text was written by an LLM, locally, from the command line.

$ slop draft.md
0.4206  LIKELY AI   draft.md
        above the cut — 4.5x its 1% cut

$ slop notes.txt
0.0153  HUMAN       notes.txt
        below the cut — 0.2x its 1% cut

It runs Open Pangram (EditLens, RoBERTa-large) on your own machine — no API, no upload — and applies a threshold calibrated against a control set of known-human writing.

Why the threshold matters

The model ships a 4-bucket ordinal head whose labels are badly under-calibrated. Taking its argmax bucket as a verdict makes it look far worse than it is: in a 76-document benchmark, 100%-machine-written text averaged 0.11–0.39, nothing at all landed in bucket 2, and only 3 of 76 documents ever reached bucket 3 ("AI-generated"). Read as a classifier, it looks like it barely works.

It works fine. The ranking is excellent and the calibration is off. Measured as AUROC against a 198-document human control — which needs no threshold at all — it separates AI from human prose near-perfectly at every model generation:

Year of ChatGPT output	n	AUROC
2022	4	0.994
2023	18	0.993
2024	18	0.996
2025	18	0.998
2026	18	0.999

So slop ignores the shipped labels and thresholds the raw score instead, at a cut read off the human control's own distribution:

Cut	Threshold	False positives on the control	Detection, 2025–26 prose
default	> 0.0943	1/198	36/36
-s	> 0.0369	9/198	36/36

On that same control set the model produced zero false positives at its own 0.5 cut, with a maximum score of 0.31 — which is exactly why 0.5 is the wrong place to threshold.

Install

slop is stdlib-only. pangram-score needs PyTorch and Transformers.

git clone https://github.com/ericboehs/slop
cd slop
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

The checkpoint is gated on Hugging Face — request access on the model page, then:

.venv/bin/hf auth login

Put both on your PATH. slop symlinks fine, but pangram-score has to run under the venv, so give it a shim rather than symlinking it bare:

ln -s "$PWD/slop" ~/.local/bin/slop
printf '#!/bin/sh\nexec "%s/.venv/bin/python" "%s/pangram-score" "$@"\n' "$PWD" "$PWD" \
  > ~/.local/bin/pangram-score
chmod +x ~/.local/bin/pangram-score

Weights (~1.4 GB) download on first run. It uses CUDA or Apple Silicon MPS when available, CPU otherwise.

Usage

slop FILE...                 # score files
slop --text "some text"      # score a literal string
cat draft.md | slop          # score stdin
slop --json FILE             # machine-readable
slop -q FILE && echo human   # exit status only
slop -s FILE                 # 5% FPR cut: catches more, accuses more

Exit status is 0 below the cut, 1 above it, 2 on error — so it composes:

find . -name '*.md' -exec slop -q {} \; -print   # list the suspect files

Slack

Pass a message permalink and it pulls the text via slk, stripping mentions, channel refs and link markup first — otherwise you measure the formatting, not the author:

$ slop https://example.slack.com/archives/C0123456789/p1700000000000000
0.0742  HUMAN       https://example.slack.com/…/p1700000000000000  (by Jane Doe)
        below the cut — 0.8x its 1% cut
        ⚠️  148 words — shaky below 200; treat this as a hint, not a result

Read the score, not the verdict

A detector you trust blindly is worse than no detector.

• Length dominates. Below ~75 words nothing here is reliable; below ~200 it's shaky. slop warns you. Most Slack messages fall in that range.
• Structure reads as machine-written. Hand-written checklists, technical write-ups and anything heavily formatted score high regardless of author. In the benchmark, a hand-written truck-maintenance checklist scored 0.1142 — a false positive at 1.2× the default cut. This threshold does not fix that, and no threshold that keeps 2026 detection at 100% can.
• The calibration is one person's corpus — a single writer's prose as the human control, ChatGPT as the AI class. Your register isn't theirs. Treat the cuts as a starting point, and if it matters, re-derive them from a control set of your own writing.
• Above the cut is evidence, not proof. Do not accuse anyone on the strength of a number from this or any other detector.

What's in here

slop	the CLI: thresholds, verdicts, Slack, exit codes. stdlib only
pangram-score	raw scorer: loads the model, prints its number. torch + transformers

pangram-score is usable on its own if you want the model's output with no interpretation layered on it. It handles a couple of things the upstream repo's inference script doesn't: it scores arbitrary text rather than only HuggingFace datasets, chunks past RoBERTa's hard 512-token window with length-weighted aggregation, and replicates upstream's clean_text exactly (it lowercases, and strips </think> tags and boilerplate openers) so input matches the training distribution.

License

MIT, for the code in this repo. The model weights are a separate work with their own terms — see the model card.