How Does an AI Detector Work? Signals, Scores, Limits

An AI detector works by measuring how predictable a text is and how evenly it is built, then comparing those measurements against what a language model would produce. It never "knows" who wrote the text. It estimates, and it reports that estimate as a probability. Understanding how an AI detector works is the difference between panicking at a 62% score and reading it correctly: as a signal about the writing, not a verdict about the writer.

This piece walks through the actual mechanics: perplexity and burstiness, the stylometric features that most modern tools add on top, the two main architectures (trained classifiers and LLM judges), why the output is a probability, and the specific kinds of human prose that get flagged by mistake.

How does an AI detector work at the most basic level?

Language models write by choosing, one token at a time, the word most likely to come next. At default settings they pick high-probability words nearly every time. That leaves a fingerprint. A detector reads the text back through a model of its own and asks one question at each word: how surprised am I by this?

Low surprise across a whole document means the text follows the statistical grooves a model would follow. High surprise, in places, means a person made choices a model would rate as unlikely. Everything else in detection is a refinement of that one idea.

Perplexity: the surprise score

Perplexity is the formal name for that surprise. A sentence like "The meeting has been moved to Thursday afternoon" has low perplexity; every word follows naturally from the last. A sentence like "The meeting limped into Thursday, wounded" has higher perplexity, because "limped" and "wounded" are not what a model expects after "meeting."

Machine text sits at low perplexity almost everywhere. Human text has a higher average and, more tellingly, a wider spread. A single number per document is still crude, though, which is why the second measure exists.

Burstiness: variance in the surprise

Burstiness measures how much perplexity swings from sentence to sentence. People write in bursts. A long, winding explanation gets followed by a four-word sentence. Then an aside. Then another long one. Models, left alone, produce sentences of similar length and similar predictability, paragraph after paragraph. Flat burstiness is one of the strongest single signals a detector has, and it is also the one that careful human editing changes most.

A detector is not reading meaning. It is reading rhythm and predictability, and it flags text whose rhythm is too even and whose word choices are too safe.

What does an AI detector look for beyond perplexity?

Perplexity alone was the whole story in 2023. Newer models write with more varied vocabulary, and a paraphraser can push perplexity up without making text read any more human. So detectors now add stylometric features, the kind of thing a sharp editor notices without naming it.

Feature	What it measures	Machine-like pattern
Sentence-length variance	Standard deviation of words per sentence	Most sentences land between 15 and 25 words
Repeated openers	How often sentences start the same way	"This", "It is", "These" opening sentence after sentence
Stock vocabulary	Hits against a list of overused model words	"delve", "seamless", "robust", "tapestry", "in today's world"
Contraction rate	Share of "do not" vs "don't" style pairs	Near-zero contractions in otherwise casual prose
Paragraph uniformity	Similarity of paragraph length and shape	Every paragraph is 3 to 4 sentences with a topic-sentence-then-support shape
Transition density	Connectives per 100 words	"Furthermore", "Additionally", "Moreover" at the top of most paragraphs

None of these features is decisive on its own. A legal brief has low contraction rate because that is the genre. A children's story has short, uniform sentences on purpose. Detectors that weigh a bundle of signals together do better than any single one, because a real human document rarely trips all of them at once.

Classifier vs LLM judge: the two ways detectors are built

Under the hood there are two families, and many commercial tools now combine them.

The trained classifier

A classifier is a model trained on a large labelled corpus: this pile is human, this pile is machine. It learns whatever separates the piles, including features nobody designed. That makes it fast, cheap to run, and very accurate on text that resembles its training data. It also makes it brittle. A classifier trained mostly on 2023 ChatGPT output can be thrown by a 2026 model, by a new domain, or by text that has been through a paraphraser. The RAID benchmark, published in 2024, exists partly to expose that brittleness: it tests detectors across 11 generators, 8 domains, and 11 adversarial edits, and detector rankings shuffle noticeably between conditions.

The LLM judge

An LLM judge is a large language model prompted to read the passage and reason about whether it reads machine-written, often with specific instructions about the features above. It generalises better to new styles and can explain itself in words, which matters for editorial workflows. The cost is that it is slower, pricier per call, and its answer can drift with prompt wording. On its own it is also easier to talk out of a verdict.

Humanizeo runs both. Stylometric heuristics score the burstiness, paragraph uniformity, contraction rate, stock-vocabulary hits, and repeated openers; an LLM judge reads the passages on top. When the two disagree, the score reflects that uncertainty rather than hiding it.

Why AI detector scores are probabilities, not verdicts

A score of 78% AI does not mean 78% of the words were generated. It means the detector judged the text to look more like its machine examples than its human examples, with that much confidence. Two consequences follow.

First, scores from different tools are not comparable. Each tool has its own training data, its own feature set, and its own threshold for calling something "likely AI." The same paragraph scoring 12% on one tool and 71% on another is normal, not a malfunction. Second, a single document score hides where the signal comes from. Three flat, list-like paragraphs in the middle of an otherwise lively piece can drag the whole thing over a threshold.

Vendors who publish a headline accuracy number are usually reporting one benchmark under one set of conditions. Grammarly, for example, cites a 99% figure from the RAID leaderboard while stating on the same page that no detector is 100% accurate and results should never be relied on alone. Both statements are true. They just describe different things: performance on a fixed test set, and performance on whatever you paste in next.

Where false positives come from

The uncomfortable part of how AI detectors work is that the features they reward, high perplexity and high burstiness, are features of a particular kind of English. Not all human writing has them.

Non-native English

Writers working in a second language often use the safest, most common word in each slot, because that is the word they are sure of. That is exactly the low-perplexity pattern a detector is built to catch. A 2023 Stanford study found that several detectors flagged more than half of a set of TOEFL essays written by real people, while flagging almost none of a comparable set by native speakers. The effect has narrowed in newer tools but has not disappeared.

Terse technical prose

Documentation, API references, compliance text, and lab methods are written to be predictable on purpose. Short declarative sentences, repeated structures, no contractions, no flourishes. Detectors read that discipline as machine uniformity. Independent tests of consumer detectors have reported false-positive rates on methodology sections and technical writing well above their rates on narrative or opinion pieces.

Heavily templated formats

Press releases, product listings, and FAQ pages share a house shape across thousands of human-written examples. If a classifier saw a lot of machine-written versions of those formats during training, it learns the format, not the author.

A false positive is not proof the detector is broken. It is proof that the text shares surface statistics with machine output, which is a different and more fixable problem.

What per-passage highlighting adds

A document-level percentage tells you that something is off. It does not tell you what. Per-passage highlighting changes the job from "argue with a number" to "edit these four sentences."

Take a 900-word article that scores 64% overall. With highlighting you might see the intro and closing are fine, and the flags cluster in a middle section where five consecutive sentences open with "This" and every paragraph is exactly three sentences. That is a twenty-minute edit. Without highlighting, teams rewrite the entire piece, which wastes time and often makes the prose worse.

Highlighting also exposes the false-positive cases above. When a flagged passage turns out to be a deliberately terse spec table or a quoted regulation, you can decide to leave it alone with a clear reason, rather than mangling accurate text to please a meter.

How to read a detector score sensibly

A few working rules that hold up regardless of which tool you use.

• Treat anything between roughly 30% and 70% as "mixed signal, look closer," not as a call either way.
• Read the highlighted passages before the number. The number follows from them.
• Run a second tool when a decision depends on it. Agreement between two differently built detectors means more than a high score from one.
• Know your text type. Technical, legal, and non-native prose will score higher on machine-likeness for reasons that have nothing to do with how it was produced.
• Never use a score alone to accuse a writer. Every major vendor says this in its own disclaimer, and they mean it.

Humanizeo's own detector lives at AI detector, and it feeds the rewrite loop in the AI humanizer: flagged passages are rewritten, re-scored, and rewritten again until the text reads clearly human, with facts, numbers, and links kept verbatim. If you want the fuller picture on which tools hold up across text types, the companion piece on the most accurate AI detector goes through the published benchmarks.