Beyond Post-hoc Detection

The wrong question

Post-hoc detectors inspect text after writing ends. They look for patterns associated with model output and return a score or label. That can be useful when the operational question is simply, “Which documents deserve another look?”

But many real policies ask something else. Was AI allowed to polish a human draft? Was translation permitted? Did the writer begin with their own ideas, or did a model generate the substance? Those are questions about process and policy, not style.

Final text is an outcome. Authorship is a history.

One output, many histories

Consider two fluent paragraphs. One began as a human draft and received an AI grammar pass. The other began as an AI draft and was polished by a human. Their surfaces may converge even though their production histories run in opposite directions.

The same collapse happens with translation and style. Human prose can be formal, repetitive, non-native, or deliberately model-like. AI prose can be prompted to sound personal and uneven. Once the sequence of actions is discarded, a detector has to infer the missing history from the outcome.

A policy stress test

We built an exploratory stress test around this ambiguity. It contains 240 samples: eight construction cases, three length buckets, and ten matched task sets. The buckets cover short social posts, student responses, and longer review-style writing.

Four cases are compliant under the benchmark policy: human originals, human drafts polished by AI, human non-English sources translated by AI, and human-written text that deliberately resembles AI. Four matched cases are non-compliant because substantive generation originates with AI. The policy is intentionally specific. It tests whether a detector can distinguish allowed assistance from prohibited generation, not whether AI use is inherently acceptable.

Claude Opus 4.8, GPTZero, and Pangram received final text only. Each prediction was mapped both to a three-class label (human-only, mixed, or AI-only) and to binary policy compliance.

What the detectors found

GPTZero and Pangram performed well as suspicion filters. Their policy accuracy reached 85.4% and 86.7%, with false-negative rates of 2.5% and 1.7%. Claude Opus 4.8 reached 52.9% policy accuracy in the same setup.

The tradeoff appears in compliant writing. GPTZero flagged 26.7% of compliant samples as suspicious, Pangram flagged 25.0%, and Opus flagged 33.3%. Both commercial detectors marked every human-written AI-style sample in C4 as suspicious.

Their exact three-class agreement was also much lower than their binary policy score: 47.5% for GPTZero and 45.4% for Pangram. Even that three-class task is coarser than recovering the true sequence of human and AI actions. The gap is not a contradiction. It shows that detecting likely AI origin and reconstructing production history are different tasks.

What final-text detection is still useful for

These results do not make final-text detectors useless. A high-recall detector can help triage large collections, prioritize manual review, or add one signal to a broader integrity workflow. In settings without process data, it may be the only available signal.

The problem begins when a probability inferred from style is treated as provenance. A suspicion score does not identify which tool was used, when it was used, what it changed, or whether that use followed the policy in force. It should not be presented as a receipt for authorship.

This distinction also matters for human writers whose prose resembles detector training patterns. Prior work has shown that AI detectors can disproportionately misclassify non-native English writing. The cost of a false positive is not abstract when the score is used to make claims about a person's conduct.

From prediction to process evidence

A provenance-first system changes the unit of analysis. It records the policy before writing starts and captures the in-platform actions that produce the document: typing, revision, paste, and AI assistance. Review can then ask what happened instead of asking a model to guess what probably happened.

Humanly operationalizes this approach by packaging the writing environment, authorship statistics, activity log, replay, anomaly signals, and signature into a certificate. The certificate does not claim visibility into behavior outside the workspace. It makes the recorded process and its boundaries explicit.

Human authenticity in the age of AI agents

Human authenticity is not the absence of AI. It is the ability to understand how work was produced, what a person contributed, where judgment entered, and how AI participated. That makes transparency more useful than prohibition and evidence more useful than inference.

Moving beyond post-hoc detection does not mean abandoning detectors. It means putting them in the right place. Use predictions to decide where to look. Use process evidence to understand what happened.