Science is self-correcting, but it often takes too long.
A spreadsheet to compute the R-Index and a manual that shows how to use the spreadsheet is now available on the www.r-index.org website. Researchers from all fields of science that use statistics are welcome to use the R-Index to examine the statistical integrity of published research findings. A high R-Index suggests that a set of studies reported results that are likely to replicate in an EXACT replication study with high statistical power. A low R-Index suggests that published results may be biased and that published results may not replicate. Researchers can share the results of their R-Index analyses by submitting the completed spreadsheets to www.r-index.org and the results will be posted anonymously. Results and spreadsheets will be openly accessible.
Question: When you say “likely to replicate,” do you mean that the high-powered study will find a similar effect size, or that the high-powered study will have a confidence interval for the effect size that excludes zero?
Thank you for your question. There are different definitions of replicate. Sometimes the term is used to indicate that a replication study shows an effect with the same sign (e.g. exercise INCREASES strength in Study 1 and in Study 2). Another meaning is that a study produced a statistically significant result (e.g., p < .05) and a replication study shows again a statistically significant effect of the same sign (p < .05, again). Power analysis and the R-Index refer to the second meaning of the term. How often does a set of studies produce statistically significant results? Two statistically significant results still can have rather different effect sizes. Therefore, a high R-Index does not necessarily imply that a set of studies produces very similar effect sizes. I hope this answers your question.