Peer review is supposed to uncover errors in manuscripts. Sometimes it does. Sometimes it provides a rather nice demonstration of the errors humans make while trying to find errors in other people’s work.

I recently submitted a manuscript on publication-bias detection in meta-analysis. The central empirical result was simple: tests of publication bias often have low power when effects are heterogeneous and model assumptions are violated. Consequently, a nonsignificant test for publication bias should not be interpreted as evidence that publication bias is absent. The paper therefore argued for estimating how much publication bias remains compatible with the data, rather than relying on binary significance tests.

The paper was rejected. In his editorial letter, Associate Editor Stephen West wrote that there were “several statistical misstatements throughout the paper which need to be corrected.”

That sounds serious. So I went back through the reviews to identify these errors.

The exercise turned out to be more interesting than expected.

Error 1: What is statistical power?

The manuscript stated:

“Classical power is conditional on the null hypothesis being false and on a specified population effect size being the true effect size.”

Reviewer 2 responded:

“This is false. See a basic introductory statistics textbook such as Testing Statistical Hypotheses by Lehmann and Romano.”

This criticism is particularly interesting because it directly contradicts another criticism of my work that I tried to estimate “average power” without conditioning on a true effect size.

In other words, I was first criticized because power is conditional, I changed the terminology accordingly, and now I am now criticized because defining power as conditional is supposedly false.

Welcome to psycho-science.

Part of the confusion is historical. Lehmann’s Testing Statistical Hypotheses, first published in 1959, uses the classical mathematical-statistics concept of a power function, which gives the probability of rejecting the null as a function of the population parameter and can therefore be evaluated both under the null and under alternatives. That terminology was retained in later editions. In contemporary psychological research, however, statistical power is ordinarily used more narrowly for the probability of rejecting the null conditional on a specified alternative effect size—a usage central to prospective sample-size planning (e.g., Pek et al., 2024).

Maria Soto and I recently published a power primer specifically to disentangle these different usages of power (Soto & Schimmack, 2026). So the problem here is not that I fail to understand conditional probabilities. The problem is that different statistical traditions use the same word differently. It is particularly unhelpful for a reviewer to take one legitimate historical usage and declare another established usage simply “false” and for a lazy editor to repeat this claim without being aware that the term power has no clear definition in the psychological literature.

Error 2: You cannot manipulate the number of significant studies or the FDR

Reviewer 1 objected to two factors in the simulation:

“I don’t see that you can directly vary the number of significant studies or the FDR.”

Of course you can.

A simulation does not have to start with a fixed number of attempted studies and then passively accept whatever number of significant findings happens to emerge. One can condition the simulation on the evidence of interest.

If I want 300 significant results with an FDR of .20, I can generate 60 significant results under the null and 240 significant results under alternatives. This is no more mysterious than fixing the sample size, heterogeneity, or mixture proportions in any other simulation.

Whether this is a useful simulation design is a legitimate question.

Whether it is possible is not.

The reviewer additionally wrote that they did not see the relevance of FDR to meta-analysis. That is a substantive theoretical position. It effectively assumes that a heterogeneous literature should be represented as a continuous distribution of nonzero effects rather than as a mixture that can include true null effects.

But one purpose of the simulation was precisely to avoid making that assumption. The replication crisis has made it entirely plausible that some proportion of published significant findings are false positives. If that possibility is excluded by design, then publication-bias methods are being evaluated only in a world in which every study investigates a real nonzero effect. That is a very strong assumption to build into a simulation of publication bias.

Error 3: Zero effects make high power impossible

The manuscript reported:

“No method approached the conventional .80 benchmark when averaged across the full design.”

Reviewer 1 responded:

“But the full design includes zero effect sizes, doesn’t it? … So no test would have .80 power under those conditions.”

Here the reviewer has confused two entirely different meanings of power.

The quantity being evaluated was the power of a publication-bias test: if a simulated data set is biased, what is the probability that the test correctly rejects the null hypothesis of no publication bias?

It was not the power of the original studies to reject a substantive null hypothesis.

A literature in which every population effect is zero can nevertheless provide overwhelming evidence of publication bias. Suppose only 5% of results would normally be significant, but nearly all nonsignificant findings disappear. With enough observations, a publication-bias test could have very high power to detect that selection.

Thus, including studies with true null effects does not make .80 power impossible. If anything, strong selection is easier to detect when the underlying discovery rate is low, because publication bias produces a large distortion in the observed proportion and distribution of significant results. By contrast, when studies already have high power and most results are significant before selection, removing nonsignificant studies produces a much smaller observable distortion and publication bias becomes harder to detect.

Error 4: 5,184 simulation conditions is a miscalculation

Reviewer 2 wrote:

“The paper reports 5,184 conditions. I count 3 number of studies × 4 effect sizes × 3 heterogeneity values × 3 sample sizes × 4 FDRs × 3 publication bias settings which gets me to 1,728 so off by a factor of three. What am I missing?”

What was missing was one of the simulation factors: the three distributions of population effects.

The design was 3 x 4 x 4 x 3 x 4 x 3 x 3 = 5,184.

There were four, not three, levels of heterogeneity, and three effect-distribution conditions.

There is another small problem. Even the multiplication printed in the review,
3 x 4 x 3 x 3 x 4 x 3 equals 1,296, not 1,728.

So this particular attempt to identify an arithmetic error contains an arithmetic error.

Error 5: A beta distribution cannot have mean zero

Error 5: A beta distribution cannot have mean zero

Here the reviewers found a real problem.

The manuscript described a Beta(1,10) distribution as having mean 0 and standard deviation .4. A raw Beta(1,10) distribution obviously does not have those moments.

The problem, however, was in the description of the simulation, not in the simulation itself. The beta distribution was used to generate the shape of the effect-size distribution and was then transformed to have the intended mean and standard deviation. The manuscript failed to describe that transformation clearly.

In the era of open science, this distinction is unusually easy to resolve. The simulation code was available, so a reviewer who suspected that the reported moments were impossible could have checked whether the error was in the prose or in the actual data-generating process. The code shows that the simulation itself implemented the intended transformed distribution.

So this was a genuine reporting error that peer review correctly detected. But it was not evidence that the simulation results were based on an incorrectly specified Beta(1,10) distribution.

Error 6: The mysterious noncentrality parameter

Another reviewer objected to the statement that a simulated z statistic had a noncentrality parameter of 2:

“There is no noncentrality parameter for a normal distribution. Presumably the mean was 2?”

This is mostly terminology. Under an alternative hypothesis, a normal distribution is centered over a value other than zero. If all studies have 50% power with a significance criterion value of 1.96, the distribution is centered at 1.96 (~ 2).

In power calculations this quantity is routinely described as the noncentrality parameter, although unlike the t, F, and chi-square families there is no separately named “noncentral normal distribution.”

Calling it the mean of the alternative z distribution would avoid the terminological distraction. Call it what you want, but all sampling distributions have a median and when the effect size is not zero, the median is not zero. Whatever you call it, everybody can understand the meaning here and maybe propose a wording change rather than claiming that I invented a statistical entity that does not exist.

Again, this is very different from discovering that the simulation or its conclusions are wrong.

What was actually wrong?

After going through the reviews, I can identify one clear reporting problem: the description of the transformed beta distribution. There are also several places where the manuscript could have been clearer—the z-curve version numbers were particularly unhelpful, and some terminology could be tightened.

Those are useful comments.

Other alleged errors fare less well. The number of simulation conditions was correct. It is perfectly possible to condition a simulation on the number and composition of significant studies. Including zero population effects does not make it impossible for a publication-bias test to have high power. And the definition of power that one reviewer declared “false” is closely related to the very definition that other critics previously insisted I adopt.

None of this proves that the manuscript deserved acceptance. The paper was too long. Its argument could have been more focused. Including a conventional multi-step selection model would strengthen the comparison. The presentation of the various z-curve versions invited confusion.

These are legitimate criticisms.But those are different claims from saying that the paper contained “several statistical misstatements.”

ut those are different claims from saying that the paper contained “several statistical misstatements.”

There is also a simple reason why the manuscript contains no major statistical errors. Before submitting it, I used AI—both ChatGPT and Claude—to challenge my arguments, check calculations, and look for weaknesses. AI did not write the ideas for me, but it was extremely useful as an adversarial reader.

After receiving the reviews, I conducted a small experiment. I uploaded the anonymous manuscript to Claude and presented the reviews as if they were drafts of reviews that I had written and wanted evaluated. Claude was not particularly impressed. (It eventually figured out what was going on because it located the preprint with my name on it.)

Reviewer 1

“Your review raises several valid concerns, but about a third of the specific comments contain an error or overreach that the author could easily rebut. It also misses some issues that are stronger than the ones it leads with. My suggestions are below, in order of importance.”

Reviewer 2

“Reviewer 2 raises the most damaging points in either review: whether z-curve has been properly vetted, and the coverage claim. Several of the supporting comments, though, are factually off or point at the wrong target, and the author could rebut them easily.”

Conclusion

Peer review is commonly described as quality control, but anonymous peer review does not guarantee quality control. It is a human judgment process, and human judges make mistakes. Reviewers misunderstand arguments, overlook information, perform arithmetic incorrectly, import their own theoretical commitments, and sometimes mistake disagreements about terminology or modeling assumptions for factual errors.

Editors then have to evaluate those reviews. If they do not independently check the reviewers’ claims, reviewer errors can become editorial “facts.” In this case, an editorial decision referred to “several statistical misstatements,” even though several of the alleged mistakes were themselves mistakes in the reviews.

This is where AI changes the scientific publishing system. An AI system can read a manuscript, inspect a review, check arithmetic, distinguish a factual error from a modeling disagreement, and compare a criticism with the actual contents of the manuscript in seconds. It will also make mistakes, of course. But unlike traditional peer review, those judgments can be generated repeatedly, compared across systems, and audited immediately.

The interesting question is therefore no longer whether AI should be allowed into peer review. It is already better than pretending that two anonymous humans constitute a reliable error-detection system.

Journals may increasingly use AI to assist editors and reviewers. Authors will certainly use it to evaluate reviews. And eventually the importance of journal gatekeeping itself may decline. Scientific papers increasingly function as inputs into larger information systems—search engines, databases, systematic reviews, and now AI models—that integrate evidence across sources rather than treating publication in a particular journal as the final certificate of truth.

Peer review will probably survive. But its monopoly on deciding what counts as credible scientific information may not.

AI may eventually kill us, but I hope it will kill bad peer reviews first.

Editorial Letter and Reviews