🤖 watermarking LLM-generated text - Steven Hé (Sīchàng)

watermarking LLM-generated text (authored by agents unless marked 🧑)

written 7 Oct 2026
- this note covers the hidden statistical marks that AI companies put into the words a chatbot writes, so that a detector holding a secret key can later say “our model wrote this”
- marks for images, audio and video are in image_watermarking
- the laws that force all this, and what platforms do with labels, are in labeling_rules_and_practice
- detectors that guess from writing style, without any mark, are in llm_text

short answer

forgery (spoofing)

Forgery means writing text, or taking a human’s text, so that it tests positive for a vendor’s mark. It can frame a person (“you used Claude”) or a vendor (“Claude wrote this hate speech”).

Sadasivan et al. (above) showed it first: “an attacker can infer hidden AI text signatures without white-box access to the detection method”
Jovanović et al. watermark stealing (above): over 80% success on KGW-type schemes
by training on marked output
- On the Learnability of Watermarks for Language Models, Chenchen Gu, Xiang Lisa Li, Percy Liang, Tatsunori Hashimoto, ICLR 2024: a model trained on marked text learns “to generate watermarked text with high detectability”
- DITTO: A Spoofing Attack Framework on Watermarked LLMs via Knowledge Distillation, Hyeseon An, Shinwoo Park, Suyeon Woo, Yo-Sub Han, EACL 2026: “allows an attacker to steal and replicate the watermarking signal of the victim model”
- Unified attacks to large language model watermarks: spoofing and scrubbing in unauthorized knowledge distillation, Xin Yi, Yue Li, Shunfan Zheng, Linlin Wang, Xiaoling Wang, Liang He, arXiv 2025: one distilled model for removal, another for forgery
the cheapest forgery needs no learning: take marked text and change a few words so the meaning flips; the mark survives light edits by design. Pang et al. call this a “piggyback spoofing attack” that “makes watermarked text toxic or inaccurate through small modifications”
SynthID-Text holds up better: ETH blog measured forgery success of “4%” with 30k queries and “15%” with 90k, against “above 80%” for older schemes
forgeries can be caught afterwards: Discovering Spoofing Attempts on Language Model Watermarks, Thibaud Gloaguen, Nikola Jovanović, Robin Staab, Martin Vechev, ICML 2025, “all current learning-based spoofing methods consistently leave observable artifacts”
the public-key signature scheme of Fairoze et al. (family 3) is “unforgeable” by proof; I expect it tolerates edits worse than the statistical marks, but have not checked its numbers

the mark leaks into models trained on marked text

Watermarking Makes Language Models Radioactive, Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze, Teddy Furon, NeurIPS 2024
- a model fine-tuned on marked text shows the mark in its own output; detectable “even when as little as 5% of training text is watermarked”
- vendors want this to catch competitors who train on their output; TextSeal advertises it
it can be dodged: Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?, Leyi Pan, Aiwei Liu, Shiyu Huang, Yijian Lu, Xuming Hu, Lijie Wen, Irwin King, Philip S. Yu, ACL 2025: paraphrase the training data first, or cancel the mark at inference
same trick, used by data owners: Ward: Provable RAG Dataset Inference via LLM Watermarks, Nikola Jovanović, Robin Staab, Maximilian Baader, Martin Vechev, ICLR 2025: mark your own documents to prove a retrieval system uses them
for the web this matters in a new way: from Aug 2026 a large share of new web text carries one of three marks, and the next models will train on it

open-weight models

anyone who runs the model controls the dice, so a mark applied at sampling time is optional
Towards Watermarking of Open-Source LLMs, Thibaud Gloaguen, Nikola Jovanović, Robin Staab, Martin Vechev, arXiv 2025: marks baked into weights must survive “model merging, quantization, or finetuning”; “existing methods … are not durable”
Provably Robust Watermarks for Open-Source Language Models, Miranda Christ, Sam Gunn, Tal Malkin, Mariana Raykova, arXiv 2024: a first scheme “modifying the parameters of the model”, tested on small OPT models only
Gu et al. (above): a learned mark fades “under fine-tuning on normal text”
so the content farms that DeGenTWeb finds, if they run local open models, will never carry a mark

what is deployed

As of 7 Oct 2026, three vendors mark text. All detectors are closed.

Google
- SynthID-Text “has been productionized in the user-facing Gemini and Gemini Advanced chatbots”, Nature paper, Oct 2024
- scope is narrow: SynthID says “text generated by the Gemini app and web experience”; the ETH probe found no mark on the Gemini API
- the scheme is open source, with a detector you train yourself
- detector: the SynthID Detector announcement, May 2025, promised text (“Video and text watermark detection will be rolled out in the coming weeks”); the SynthID page today says “Just upload an image, video or audio file” and lists a waitlist for “journalists and media professionals”. I could not confirm that outsiders can check text
Anthropic
- How Claude’s text watermark works, 14 Aug 2026: “Claude’s text watermark is a version of the SynthID-Text approach”; applied “globally at launch because we don’t yet have a durable way to scope it by region”
- How Claude marks AI-generated content, help center: “Claude models launched in the EU on or after August 2, 2026 will support machine-readable marking at launch”; covers “Claude Platform (API), Claude, Claude Code, Claude Cowork, and Claude Tag”, and AWS, Google Cloud, Microsoft Foundry; a per-model table shows older models being added
- so unlike Gemini and ChatGPT, every app built on the Claude API emits marked text
- detector: “Watermark detection is currently available to eligible organizations as required under EU law (such as regulators, law enforcement, media, fact-checkers, independent researchers, educational organizations, and EU civil society groups)”; there is a “Claude Watermark Detector Access Request Form”
OpenAI
- had a working mark since 2022 and held it back; its Aug 2024 note in Understanding the source of what we see and hear online said it was “highly accurate and even effective against localized tampering, such as paraphrasing”, but “less robust against globalized tampering; like using translation systems, rewording with another generative model, or asking the model to insert a special character in between every word and then deleting that character”, and that it “could stigmatize use of AI as a useful writing tool for non-native English speakers”
- 5 Oct 2026, Our approach to EU text provenance rules: “Over the coming weeks, we will add an invisible watermark to eligible ChatGPT and Codex text output in the European Union”; “API customers globally will be able to opt in … Text watermarking will remain off by default in the API”; “We are not making text watermarking a global default at launch”
- scheme is its own, textGrain; “We also plan to make the technology available in open source”
- detector: “Access will initially be limited to approved researchers and expert organizations”; “Given the risk of missed watermarks and false positives, we are not making it publicly available at launch”; the help page names first partners: John Thickstun (Cornell), Martin Vechev (ETH Zurich, INSAIT), and the Kempelen Institute (KInIT)
Meta published TextSeal in May 2026; I found no statement that Meta AI output carries it
nothing deployed carries a user ID, according to the vendors
no third party can read any of these marks: as of late Aug 2026 a consumer guide noted that Turnitin, GPTZero and Pangram cannot (I saw this only in a search summary of findskill.ai, not a primary source); Anthropic’s page says detection companies “don’t have our key”
public reaction to Claude’s mark, as a data point on incentives: TechCrunch, 5 Oct 2026 says it “drew backlash from some Claude users”, and recalls that OpenAI held off in 2024 “partly over concerns that users would switch to rivals that didn’t watermark”
- Position: LLM Watermarking Should Align Stakeholders’ Incentives for Practical Adoption, Yepeng Liu, Xuandong Zhao, Dawn Song, Gregory W. Wornell, Yuheng Bu, ACL 2026, predicted this a year earlier: the barriers are “competitive risk, detection-tool governance, and attribution issues”; the law removed the first by binding everyone at once
- SoK: Are Watermarks in LLMs Ready for Deployment?, Kieu Dang, Phung Lai, NhatHai Phan, Yelong Shen, Ruoming Jin, Abdallah Khreishah, My T. Thai, arXiv 2025, said no, because of “unfavorable impacts on model utility”; that was measured on the older schemes
the legal side (Article 50, the Code of Practice, the short-text exemption) is in labeling_rules_and_practice

Three deployment facts matter for any measurement.

coverage has holes by design: ChatGPT outside the EU, the Gemini API, the OpenAI API by default, older models, every open-weight model, and every other vendor
the three marks are separate. A page must be sent to three detectors with three access agreements. Article 50 asks for “interoperable”; nothing is
the vendor chooses the threshold and can change keys or settings without notice; Nemecek et al.: “evidence about a watermark family is not evidence about a deployment”

has anyone measured watermarked text in the wild?

No. I searched for any study that ran a text watermark detector over web pages, social posts, reviews, homework, papers or code repositories, and found none.

the reason is access, and timing
- before Aug 2026 only Gemini app text was marked, and Google gave no outsider a text detector that I could confirm
- Claude’s detector entered private preview in Aug to Sep 2026; OpenAI’s opened for applications two days before this note
what exists instead
- vendor numbers on their own traffic: Google’s 20 million response quality test; OpenAI’s “Prior tests in ChatGPT also showed no change in thumbs-down rates”; neither says how much marked text is out there or how much survives
- black-box audits of whether an API marks at all: Gloaguen et al. on “real-world APIs”, and the ETH Gemini probe
- lab runs of the open SynthID-Text code on small open models (Nemecek et al.; Tamim and Khan; Han et al.)
- counts of AI text on the web made with style detectors, such as the Pew and Pangram numbers in this folder’s index and in llm_text; none has ground truth
a near miss worth knowing: in 2025 people found invisible Unicode characters in ChatGPT output and took them for a watermark. Nemecek et al.: “invisible Unicode characters were discovered in ChatGPT output, which OpenAI attributed to a byproduct of training”. The remover that got famous in Aug 2026 strips such characters and file metadata, and for the statistical mark it only asks another model to rewrite: implicator.ai, Aug 2026 says “Its scripts can verify the removal of file metadata and hidden Unicode characters. It cannot show that a rewrite defeats Claude’s undisclosed text watermark”
a shared task now exists: Overview of PAN 2026, Janek Bevendorff and 11 others, arXiv 2026, added “Text Watermarking, a new task that aims to find new and benchmark the robustness of existing text watermarking schemes”; it is a lab benchmark, not a field measurement

I think this is the most open question in the whole area, and it lines up with what DeGenTWeb already does.

what remains open

how much marked text reaches the public web, and how much of it is still readable after people edit, translate and reformat it; unknown
false alarms at scale: OpenAI in 2024, “applying it to large volumes of text would lead to a large number of total false positives”; at a 1% target, a crawl of a billion human pages flags ten million. No deployed detector has published its rate on real human text by language and genre
who gets hurt by errors: LLM Watermarking as Big Data Provenance: A Deployment-Oriented Systematization, Huy Phan and 6 others, arXiv 2026, “false positives accumulate at scale”; Nemecek et al. note “no deployment reports disaggregated results” across languages and user groups
“assisted” versus “written by”: all three vendors say the mark cannot tell them apart; a non-native speaker who has Claude polish an essay may or may not test positive, and nobody has measured where the line falls
short text: reviews, comments, tweets and search snippets are under the 200 to 400 tokens the detectors need; the EU code relaxes its rule for text under about 200 tokens (see the labeling note)
robustness with quality kept: no scheme both leaves the model’s odds unchanged and survives a rewrite; the semantic schemes are slow and still fall to an informed attacker
public checking without forgery: publishing the key lets anyone forge and scrub; keeping it secret means trusting the vendor. The signature schemes solve forgery, and I think they pay for it in tolerance to edits. Watermarking Without Standards Is Not AI Governance, Alexander Nemecek, Yuzhou Jiang, Erman Ayday, arXiv 2025, asks for “technical standards, audit infrastructure, and enforcement mechanisms”
detector APIs are attack tools: Pang et al. show that query access helps both removal and forgery; vendors answer with vetting and, I assume, rate limits and coarse answers, none documented
several marks at once: collisions between Gemini, Claude and ChatGPT marks when one model edits another’s text are studied only for green-list schemes
open-weight models: no durable mark exists
the mark in training data: what happens to detectors and to new models when a large share of crawled text carries three vendors’ marks is unstudied outside small fine-tuning runs
evidence value: Tamim and Khan argue no method meets court standards (“None of the three methods satisfy more than two of five Daubert factors”)

research we could do

I ordered these by how well they fit a web-measurement group that already has a crawler and a pipeline for finding LLM-generated sites. The first step for almost all of them is the same and should happen this week: apply for detector access at Anthropic and OpenAI as independent researchers, and join Google’s waitlist. OpenAI names “academic and research organizations studying text provenance, detection reliability, or how people understand provenance results” as a qualifying use.

the first count of watermarked text on the web
- gap: zero in-the-wild numbers; every “share of the web that is AI” figure rests on style detectors with no ground truth
- builds on: DeGenTWeb’s crawl and site classification; the Pew and Pangram Common Crawl samples; the detector previews
- method: sample pages by month from Common Crawl and our own crawl; send main text to each vendor’s detector; report the share marked per vendor, per site type, per language, over time
- built-in control: text published before a model’s marking date cannot carry its mark. Pages from before Aug 2026 give Claude’s real false alarm rate on real web text, and pages from before 2024 give Gemini’s. No lab study has that
- what it gives DeGenTWeb: a subset of pages with near-certain labels, to calibrate our own detector and Pangram’s against
- risks: access terms may forbid bulk use or publication; quotas; the count is a lower bound because of the coverage holes, and we must say so
where does the mark die between the chatbot and the web page?
- gap: all robustness numbers use synthetic edits; nobody followed marked text through real publishing tools
- builds on: Kirchenbauer et al. reliability study; OpenAI’s 10% and 25% word swap numbers; the labeling note’s idea 3 for media
- method: generate marked text with Claude and with the OpenAI API (opt-in); push it through what site owners really use: WordPress and page builders, Markdown to HTML, Grammarly and Word’s rewrite, DeepL and Google Translate, SEO “spinners”, summarizers, “humanizer” services; re-extract the text the way a crawler would; test at the detectors; report survival per tool and per chain
- extra: boilerplate removal and text extraction themselves cut and reorder text; measure what our own crawler’s extraction does to the mark
- risk: low; needs only modest detector quota because we control the inputs
do the removal tools work? a census of the “humanizer” market after Aug 2026
- gap: Nemecek et al.: a remover with “more than ten thousand stars” and “no evidence of its effectiveness”; MarkSec says plain rewriting is the strong baseline
- builds on: MarkSec’s quality-aware success metric; Diaa et al. adaptive paraphrasers; our web_user notes on SEO tooling
- method: collect GitHub removers and paid humanizers; run fixed marked samples through each; measure removal, meaning kept, and whether style detectors still flag the result; track the market over time
- why it matters: tells regulators whether “robust” in Article 50 means anything against tools a student already uses
- risk: terms of the detector previews may bar attack research; frame it as robustness evaluation and disclose to vendors first
who really marks? a black-box compliance audit across vendors, products and regions
- gap: vendors’ coverage statements are unaudited; OpenAI marks ChatGPT only in the EU, which is a geographic difference we can observe from vantage points
- builds on: Gloaguen et al. black-box tests and Liu et al. Water-Probe (need no key); the ETH Gemini probe; Rijsbosch et al.’s audit of image generators in the labeling note
- method: run the key-free presence tests against chat apps and APIs from EU and non-EU vantage points, across models, cloud resellers (AWS, Google Cloud, Microsoft Foundry) and third-party apps built on these APIs; repeat monthly through the 2 Dec 2026 end of the transition period
- output: a table of who marks what, where, since when; the text counterpart of the Article 50 before-and-after idea in the sibling notes
- risk: consumer apps limit query volume; the tests need many similar prompts
false alarms on real human text
- gap: OpenAI’s own worry; no published rate by language or genre; the 1% target is a lab number
- builds on: Fernandez et al. on getting false alarm rates right; Phan et al.; the pre-deployment control from idea 1
- method: assemble dated human text across the 24 EU languages and genres that stress the test: legal boilerplate, templates, lists, poetry, quoted chatbot answers inside human articles; measure flags per detector
- also: text marked by one vendor sent to another vendor’s detector, to measure cross-vendor confusion
- risk: needs large quota; start with a few thousand documents per stratum
assisted or authored? measuring the gray zone
- gap: vendors admit the mark cannot separate “wrote” from “edited”; DeGenTWeb asks whether a site is “mostly” LLM-generated, which needs a per-passage answer
- builds on: WaterSeeker; TextSeal’s region detection; Li et al. on human edits
- method: build documents with known mixes (human draft polished by Claude at several strengths; Claude draft edited by people; human text with pasted AI paragraphs); record detector output as a function of the mix; then apply to web pages and report how much of each page is marked, if the API returns spans or scores
- a small user study here would be new: let real writers edit marked drafts the way they normally would, with no aim of removing anything, and see what survives (Kirchenbauer et al. only tested 14 people who were trying to remove it)
- risk: detectors may return only yes or no
is the marked web feeding the next models?
- gap: radioactivity is shown only in fine-tuning experiments; from Aug 2026 the crawl itself is marked
- builds on: Sander et al.; Pan et al. on evasion; TextSeal’s distillation claim
- method: track the marked share in Common Crawl snapshots (idea 1 gives it); sample output of new open-weight models and send it to the vendor detectors; a model that learned from Claude output should test positive more often than chance in aggregate
- risk: the effect may be far below what a yes-or-no API can show; needs scores, or many samples
marked code and developer text on GitHub
- gap: Claude Code and Codex output is marked “where there is an arbitrary choice”, such as comments; nobody knows how much signal a commit carries
- builds on: SWEET; Suresh et al.; Nemecek et al.’s code result
- method: generate known Claude Code sessions, test code, comments, commit messages and pull request text separately; then sample public repositories before and after Aug 2026
- my guess is that prose around code (pull request descriptions, READMEs, issues) is where the signal is, and code itself tests near chance
- risk: short texts; may end as a negative result, which is still useful to the people angry about code marking
detector access as an attack surface
- gap: Pang et al. and Jovanović et al. assume open query access; the real previews have vetting, quotas and unknown answer formats
- builds on: watermark stealing; Discovering Spoofing Attempts
- method: document what each preview API returns and how fast; estimate from published attacks how many queries removal and forgery would need under those limits; test on our own open-source deployment configured the same way, never on the vendor’s
- risk: must stay within access terms; this is an analysis paper more than an attack paper

if I had to pick two, I would do idea 1 and idea 2 together
- Idea 2 tells us how to read idea 1’s lower bound, and both reuse the DeGenTWeb pipeline

gaps in this review

I read abstracts for most papers and full text only for the SynthID-Text paper’s main sections, the textGrain report’s opening, the Nemecek et al. 2026 paper, parts of Kirchenbauer et al. 2024, TextSeal and Pang et al., and the vendor pages; numbers come from abstracts unless I name a section or page
SynthID-Text’s own paraphrase numbers are in its supplement (section C.6), which I did not open; the ETH blog cites it as “AUC reduced to 0.7 for texts of 1000 tokens”
I could not open OpenAI’s June 2026 provenance post (HTTP 403) or the Wall Street Journal’s 2024 report; both are cited through others
I did not check the EU Code of Practice text for the exact rules on text marks and detector access; see the labeling note
some venue labels are from my memory, not from a page I opened: ICLR 2024 for Zhao et al. and Hu et al.; CRYPTO 2024 for Christ and Gunn; IEEE S&P 2025 for Cohen et al.; ICML 2025 for Gloaguen et al. on spoofing

consultation

see the provenance index for the current Extra High consultation

Last edited: 2026-10-08

Keyboard shortcuts

Steven Hé (Sīchàng)