On February 18, 2022, I gave a webinar (1 hour, 30 minutes of questions) about the use of the z-curve package in R-Studio, including data preparation and management. Here I post the webinar for interested users of z-curve analysis.
The r-script to load data and run z-curve analyses is provided here (r-script).
The video recording of the webinar is here (video).
Many datasets with z-values that can be used for demonstrations or to explore z-curve can be found here (dropbox-link).
Feel free to ask additional questions in the comment section.
Hello,
I am looking forward to the webinar.
I have a question that might be of interest to some people: when you do z-curves on a particular researcher, do you use all the papers in which he has participated ? Or only the papers published as first author? Or do you use the CRediT author statement to determine his involvement in the different papers?
Thank you for your work and see you soon.
I will address this in the webinar, but I can also give a quick answer here.
When I use automatically extracted test statistics, I just started using article lists in Web Of Science. I am using all articles, independent of amount of contribution or author ship position that are covered by the 120 journals I am tracking so far.
https://replicationindex.com/2022/01/29/personalized-z-curve-project/
2. For some eminent researchers, I have also conducted hand-coding of articles. Here I have focused on the most highly cited articles that contribute to their H-Index.
https://replicationindex.com/2019/01/11/replicability-audit/
In the Z-curve graphs what is the leftmost dashed vertical red line? Those didn’t appear in the webinar. Thanks for making the re ording available.
it highlights z = 1.65, which corresponds to p = .10, which is sometimes used to argue that p-values between .05 and .10 are still evidence against the null-hypothesis, “marginal significance” Only values below this value are most likely to be published failures to reject the null-hypothesis.
In the webinar you say that the z-values have to be positive. It makes sense as it makes it a lot easier to interpret the graph if all values are positive. However since z-values also can be negative – I would like to know if you just “remove” the minus in front of the negative z-values in order to make them positive? Or what do I do to obtain only positive z-values that can be plotted into the z-curve?
Take the absolute. The sign only makes sense if you want to interpret the direction of an effect (e.g., men are taller than women or women are taller than men). Z-curve only cares about the strength of the evidence (i.e., the effect size over sampling error ratio).
Thank you so much for your answer! I have another question: We are studying sociology (and we examine publication bias using a z-curve). We cannot in the same way just ‘harvest’ all data, since the reporting is far more messy than papers within the field of psychology. Therefore, we are collecting specific PEs and SEs (detecting which to collect based on the hypothesis) in order to calculate the z-score. But oftentimes each hypothesis uses several variables to examine an association. That means, we have gone through 69 articles but we have collected more than 700 z-scores. Our plan was to randomly select ONE z-score per article (N=69) to make it as unbiased as possible, but can we actually make a z-curve with all the z-scores (N=>700) even though they are connected in some way and often shine light on the same association just using different variables. Will it bias the result? We reconsidered ‘just’ to base our z-curve on the small N since the difference between 69 and >700 is substantial and you used all z-scores in your curve, but are there any problems associated with that approach? Thank you in advance!
Yes, you can. We are also working on an extension of z-curve that samples from the 700 values in a way that preserves their independence. For now, it is ok to use all.
Thank you for your answer once again! We hope it is okay if we ask you another question 🙂 In our analysis we would like to start of by showing the plot of all the z-scores (we believe it is a histogram?). Is there a R command that gives the plot without the z-curve? We have already tried the histogram plot but the distribution does not correspond with the plot we get when we use the z-curve command. Thank you for taking your time to help us!
To be a bit more specific: the first picture is the output we get when we use the histogram command and the second picture is the output we get with the z-curve command (see the pictures here: https://docs.google.com/document/d/1wgNc6gM1P8-MPpA5KiJWKl0GGWnPqp9MdRqfIOLpfsA/edit?usp=sharing). As you can see the difference between the two plots is in the y-axis (the density axis). How do we get an histogram equal to the one we get when we use the z-curve command (without the actual z-curve)?
Hi Emily, happy to help but you can also email me. ulrich.schimmack@utoronto.ca
I looked at the file you shared and the two plots look the same to me, regarding the histogram. Not sure what else you want to add to a histogram or why the z-curve plot is not what you are looking for.
Hi Ulrich, I have sent you an email 🙂