Canonicality: the essay Signal & Noise stands behind is the published Issue 23, AI Can Hallucinate a Jury. This redline is part of its published audit. Tags: floor = a binding accuracy row; adopted = an audit proposal the author took; author = a sentence the author wrote unprompted by any critique leg.
AI Can Hallucinate a Jury
AI has collapsed the cost of staging criticism, not the cost of earning confidence.
The most consequential AI hallucination may not be a fake citation. It may be a fake institution.
Give one model a dozen roles: logician, historian, fact-checker, empirical skeptic, hostile referee, defender, judge. Tell each to attack an essay from a different angle. Revise after every round. Add a scorecard and a final ruling: survived.
The result can reproduce the entire theater of a functioning republic of ideas—objections raised, concessions won, verdict rendered. The transcript is long. The voices sound distinct. The finished argument bears the visible marks of scrutiny.
The criticisms may be excellent. The corrections may be real. But names in separate boxes do not establish independent scrutiny. AI has not made criticism more fallible. It has made the appearance of independent criticism almost free.
That distinction matters because there is nothing uniquely artificial about the limits of criticism. Karl Popper and David Deutsch built an epistemology around the proposition that knowledge advances through conjecture, criticism and error correction—not through authority or proof of certaintyfloor. Deutsch has argued that ideas should be judged by their explanatory content and the criticisms against them, not by their source. A counterexample does not become weaker because a machine found it. A bad argument does not improve because a Nobel laureate made it.
From that perspective, complaining that AI criticism cannot prove the surviving argument true misses the point. Neither can human criticism. Criticism is not supposed to issue certificates of truth. It is supposed to expose errors and help replace worse explanations with better ones.
AI can be remarkably useful at that work. It can catch a contradiction, recalculate a number, compare a quotation with its source, retrieve a damaging precedent, identify an equivocation or propose a rival causal explanation. It can force a sweeping thesis to specify its mechanism, boundaries and possible disconfirmers. A writer who rejects such help merely because the critic is synthetic is confusing provenance with validity.
But one valid criticism and ten unsuccessful critics warrant very different inferences.
The first question is: Is this objection sound? Once an objection has been checked on its merits, its source is irrelevant to its validity. One confirmed counterexample can destroy a universal claim even if the same model that drafted the claim later discovers it.
The second question is: What should we infer from the fact that no objection was found? Now the structure of the search matters enormously. Ten critics with overlapping training, shared defaults and correlated blind spots do not provide ten independent tests merely because they were assigned ten personas.
Variation in outputs does not establish independence of errors.
The claim does not require the critics to be identical—only that their blind spots overlap enough that ten silences carry much less than ten tests’ worth of information. Show a persona panel whose misses are as uncorrelated as separate reviewers’, and this worry dissolves.
This is the institutional problem AI creates. It can mass-produce criticism’s outward forms—the panel, the debate, the dissent, the recursive review, the blind referee report—without necessarily reproducing the different evidence, incentives, methods and intellectual histories that give plural scrutiny its value. A committee of mirrors may generate useful angles. It is still not a jury.
Human review is hardly pure. Experts share fashions, institutions and blind spots. Peer review has never guaranteed truth. But at its best, human review can carry information a synthetic transcript alone does not: other people have spent scarce time looking for error, sometimes from outside the author’s control and with distinct experiences and stakes.
For most writing, meanwhile, the realistic alternative was never a jury at all. It was nothing—and against nothing, even correlated criticism is a gain.
AI turns criticism into a self-service institution. The writer can select the model, assign every role, write the instructions, determine the stopping rule, rerun the process and publish only the most impressive transcript. None of that invalidates a criticism the system actually finds. It does invalidate the assumption that the visible ceremony represents independent scrutiny.
The two failures compound: selection poisons the survival inference even when the critics are independent, and correlation keeps the poisoning invisible.
The danger is greatest in verdict-sparse domains: arguments about policy, culture, institutions, technology and the future, where decisive feedback is delayed, selectively observed or permanently contestable. Code, formal proofs and experiments are not infallible, but they provide channels through which something other than persuasive language can push back. Many essays receive no such prompt resistance.An essay about policy, culture or the future may meet no such resistance at all.floor Readers therefore lean more heavily onare left to lean onadopted process signals—fact-checking, peer review, red-teaming, disclosed disagreement—to decide what deserves trust.
AI can reproduce those signals without reproducing the processes they are taken to imply. Once the ceremony becomes nearly free, agent count and procedural display cease to be reliable evidence of independent scrutiny.
Recursion can deepen the illusion. A new round is useful when a revision creates a new vulnerability, a fresh source changes the evidence or a critic brings a genuinely different framework. Without such novelty, repeated review can become critic-overfitting. Each round teaches the defense what this family of critics knows how to attack. The thesis adapts to the test suite.
It can narrow claims defensibly—or hide exposed assumptions behind technical language and distant falsifiers. The result may become harder for this panel to attack without becoming more tightly constrained by the problem it purports to explain. That is armor, not a better explanation.
To be sure, different model runs can surface objections a single pass misses. Different models, tools, corpora and human reviewers can create genuine diversity. Human panels can be deeply conformist too. The relevant distinction is not authentic humans versus fake artificial critics. It is demonstrated opportunities to fail in different ways versus a displayed multiplicity of voices.
This is not an outsider’s complaint. The staged panel in the opening paragraph describes how essays like this one get made—including at this publication, which has praised what survived such review. What follows is a correction of a habit, not a discovery.adopted
ThatThis correctionauthor requires a different standard. Do not report that an essay “passed adversarial review.” Report which objections were raised, which were checked against sources, logic or external evidence, what changed because of them and what remains unresolved. Count new information, methods and failure modes—not agents, personas or rounds. A report like that can be faked too—but faking specifics means publishing claims a reader can check. The standard does not make deception impossible. It makes it expensive again.adopted
AI can make every argument look as though it faced a trial. Rigorous writing asks what, exactly, gave the argument a genuine chance to lose.
Use the criticism. Burn the certificateverdictadopted.
Also considered: the audit proposed further changes the author declined or left open — among them, grounding the correlation premise in measured results (declined, to keep a conceptual essay from needing a literature review) and several precision and register touches (deferred unless a cold read still flags them). The audit page carries every ruling and its reason.