LLM text detection (authored by agents unless marked 🧑)
how to tell whether text was written by an AI model: what exists, what fails, and what we could research
the short version
- every detector turns text into one score and compares it to a cutoff
- most rest on one fact: models pick the expected word more often than people do
- paid trained classifiers (Pangram, GPTZero) now beat the free score-based ones (Binoculars) in the independent tests I opened
- three things break detectors: rewriting the text, newer models, and human text lightly polished by AI
- when AI text is rare, a 1% false positive rate makes about half the flags wrong
- ordinary readers are near a coin flip; a few heavy ChatGPT users are almost never wrong
- nobody has measured what detector browser extensions send to their servers
- my top research picks: find out when pooling a site’s pages protects the verdict against humanizers and new models, and audit what those extensions send
reading order
- how detection works: the plain picture, and every term defined once
- zero-shot detectors: scores that need no labeled training, like Binoculars
- trained detectors: classifiers and rewrite-and-compare methods
- commercial detectors: how Pangram, GPTZero, Turnitin and others work
- attacks and paraphrase: how detectors get fooled, and defenses
- benchmarks: how detectors are tested, and what the tests hide
- short, mixed, and code text
- human detection: when people can tell AI text, and when not
- browser extensions: academic work and real extensions
- on web pages: running detectors on crawled pages, and what DeGenTWeb already does
- research proposals: what DeGenTWeb already has, five ideas ranked, three more from ChatGPT, and what I would not do
- ChatGPT consultation: its Extra High opinion on the proposals, and what I changed
limits
- about 170 linked sources, most opened as abstracts only; quotes are from the abstract unless the entry says otherwise
- no experiment was run
- watermarks and counts of AI content on the web are in the sibling folder llm_provenance
Last edited: