Fig. 6: Meta-analytic estimated effect sizes combining the original and the replication estimated effect sizes (a secondary replication indicator).
From: Examining the replicability of online experiments selected by a decision market
The figure plots the point estimates and 95% and 99.5% CIs of the fixed-effects weighted meta-analytic effect sizes, combining the original and the replication studies (standardized to Cohen’s d units). Studies within the three panels (top-12, random, bottom-12) are sorted based on the decision market prices as in Fig. 1. As preregistered, we report the significance of the estimated meta-analytic effect sizes for both the 0.05 significance threshold and the 0.005 significance threshold (based on a two-sided z-test). Sixteen out of 26 (61.5%; 95% CI (40.6%, 79.8%)) studies replicated according to the statistical significance indicator using the 0.05 significance threshold, and 14 out of 26 (53.8%; 95% CI (33.4%, 73.4%)) studies replicated using the 0.005 significance threshold. The error bars denote the 95% CIs of the estimated meta-analytic effect sizes. The number of observations used to estimate the 95% CIs are the sums of the original and replication sample sizes noted next to the study identifier on the y-axis as nO+R.
