The challenge of the Psychonomic Society Guidelines on statistical issues (2012)

In 2012 the Psychonomic Society adopted new guidelines on statistical issues. We examined all of the empirical papers published in the Society’s journals in 2013 to discover how much needs to change to meet the guidelines. The Guidelines call for authors to consider statistical power and sample size, but only a tiny proportion of papers did so. The Guidelines emphasize the importance of reporting appropriate measures of variability “e.g., confidence intervals around means and/or around standardized effect sizes”, but the reporting of variability was patchy. Nearly half the papers failed to report effect sizes and almost none discussed them. Effect size estimates were mostly limited to partial eta squared and Cohen’s d, rather than population estimates. Confidence intervals were very rarely reported and almost never used or discussed. Psychonomics authors and editors face a challenge to update the standard of statistical reporting if the 2012 Guidelines are to be met.