Meeting the challenge of the Psychonomic Society’s 2012 Guidelines on Statistical Issues: some success and some room for improvement

Peter E Morris, Catherine O Fritz

Research output: Contribution to journalArticleResearchpeer-review

Abstract

The Psychonomic Society (PS) adopted New Statistical Guidelines for Journals of the Psychonomic Society in November 2012. To evaluate changes in statistical reporting within and outside PS journals, we examined all empirical papers published in PS journals and in the Experimental Psychology Society journal, The Quarterly Journal of Experimental Psychology (QJEP), in 2013 and 2015, to describe these populations before and after effects of the Guidelines. Comparisons of the 2013 and 2015 PS papers reveal differences associated with the Guidelines, and QJEP provides a baseline of papers to reflect changes in reporting that are not directly influenced by the Guidelines. A priori power analyses increased from 5% to 11% in PS papers, but not in QJEP papers (2%). The reporting of effect sizes in PS papers increased from 61% to 70%, similar to the increase for QJEP from 58% to 71%. Only 18% of papers reported confidence intervals (CIs) for means; only two PS papers in 2015 reported CIs for effect sizes. Although variability statistics are important to understanding data, and to further analysis, they were only reported as numbers in just over half of the PS journal papers. Almost all PS and QJEP papers relied exclusively on null hypothesis significance testing to guide interpretation of the data. Changes associated with the Guidelines are in the desired direction with respect to reporting effect sizes and power analyses, but are not yet reflected in researchers’ practices in describing their data, addressing data assumptions, and thinking beyond the p value when interpreting their data.
Original languageEnglish
JournalPsychonomic Bulletin and Review
Early online date17 Mar 2017
DOIs
Publication statusE-pub ahead of print - 17 Mar 2017

Fingerprint

Experimental Psychology
Guidelines
Confidence Intervals
Research Personnel
Population

Keywords

  • Statistical inference
  • statistics
  • confidence intervals
  • effect size
  • quantitative analysis

Cite this

@article{c37f06ee92c3497bb5a2caa67d04405f,
title = "Meeting the challenge of the Psychonomic Society’s 2012 Guidelines on Statistical Issues: some success and some room for improvement",
abstract = "The Psychonomic Society (PS) adopted New Statistical Guidelines for Journals of the Psychonomic Society in November 2012. To evaluate changes in statistical reporting within and outside PS journals, we examined all empirical papers published in PS journals and in the Experimental Psychology Society journal, The Quarterly Journal of Experimental Psychology (QJEP), in 2013 and 2015, to describe these populations before and after effects of the Guidelines. Comparisons of the 2013 and 2015 PS papers reveal differences associated with the Guidelines, and QJEP provides a baseline of papers to reflect changes in reporting that are not directly influenced by the Guidelines. A priori power analyses increased from 5{\%} to 11{\%} in PS papers, but not in QJEP papers (2{\%}). The reporting of effect sizes in PS papers increased from 61{\%} to 70{\%}, similar to the increase for QJEP from 58{\%} to 71{\%}. Only 18{\%} of papers reported confidence intervals (CIs) for means; only two PS papers in 2015 reported CIs for effect sizes. Although variability statistics are important to understanding data, and to further analysis, they were only reported as numbers in just over half of the PS journal papers. Almost all PS and QJEP papers relied exclusively on null hypothesis significance testing to guide interpretation of the data. Changes associated with the Guidelines are in the desired direction with respect to reporting effect sizes and power analyses, but are not yet reflected in researchers’ practices in describing their data, addressing data assumptions, and thinking beyond the p value when interpreting their data.",
keywords = "Statistical inference, statistics, confidence intervals, effect size, quantitative analysis",
author = "Morris, {Peter E} and Fritz, {Catherine O}",
year = "2017",
month = "3",
day = "17",
doi = "10.3758/s13423-017-1267-y",
language = "English",
journal = "Psychonomic Bulletin and Review",
issn = "1069-9384",
publisher = "Springer Verlag",

}

Meeting the challenge of the Psychonomic Society’s 2012 Guidelines on Statistical Issues: some success and some room for improvement. / Morris, Peter E; Fritz, Catherine O.

In: Psychonomic Bulletin and Review, 17.03.2017.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Meeting the challenge of the Psychonomic Society’s 2012 Guidelines on Statistical Issues: some success and some room for improvement

AU - Morris, Peter E

AU - Fritz, Catherine O

PY - 2017/3/17

Y1 - 2017/3/17

N2 - The Psychonomic Society (PS) adopted New Statistical Guidelines for Journals of the Psychonomic Society in November 2012. To evaluate changes in statistical reporting within and outside PS journals, we examined all empirical papers published in PS journals and in the Experimental Psychology Society journal, The Quarterly Journal of Experimental Psychology (QJEP), in 2013 and 2015, to describe these populations before and after effects of the Guidelines. Comparisons of the 2013 and 2015 PS papers reveal differences associated with the Guidelines, and QJEP provides a baseline of papers to reflect changes in reporting that are not directly influenced by the Guidelines. A priori power analyses increased from 5% to 11% in PS papers, but not in QJEP papers (2%). The reporting of effect sizes in PS papers increased from 61% to 70%, similar to the increase for QJEP from 58% to 71%. Only 18% of papers reported confidence intervals (CIs) for means; only two PS papers in 2015 reported CIs for effect sizes. Although variability statistics are important to understanding data, and to further analysis, they were only reported as numbers in just over half of the PS journal papers. Almost all PS and QJEP papers relied exclusively on null hypothesis significance testing to guide interpretation of the data. Changes associated with the Guidelines are in the desired direction with respect to reporting effect sizes and power analyses, but are not yet reflected in researchers’ practices in describing their data, addressing data assumptions, and thinking beyond the p value when interpreting their data.

AB - The Psychonomic Society (PS) adopted New Statistical Guidelines for Journals of the Psychonomic Society in November 2012. To evaluate changes in statistical reporting within and outside PS journals, we examined all empirical papers published in PS journals and in the Experimental Psychology Society journal, The Quarterly Journal of Experimental Psychology (QJEP), in 2013 and 2015, to describe these populations before and after effects of the Guidelines. Comparisons of the 2013 and 2015 PS papers reveal differences associated with the Guidelines, and QJEP provides a baseline of papers to reflect changes in reporting that are not directly influenced by the Guidelines. A priori power analyses increased from 5% to 11% in PS papers, but not in QJEP papers (2%). The reporting of effect sizes in PS papers increased from 61% to 70%, similar to the increase for QJEP from 58% to 71%. Only 18% of papers reported confidence intervals (CIs) for means; only two PS papers in 2015 reported CIs for effect sizes. Although variability statistics are important to understanding data, and to further analysis, they were only reported as numbers in just over half of the PS journal papers. Almost all PS and QJEP papers relied exclusively on null hypothesis significance testing to guide interpretation of the data. Changes associated with the Guidelines are in the desired direction with respect to reporting effect sizes and power analyses, but are not yet reflected in researchers’ practices in describing their data, addressing data assumptions, and thinking beyond the p value when interpreting their data.

KW - Statistical inference

KW - statistics

KW - confidence intervals

KW - effect size

KW - quantitative analysis

U2 - 10.3758/s13423-017-1267-y

DO - 10.3758/s13423-017-1267-y

M3 - Article

JO - Psychonomic Bulletin and Review

JF - Psychonomic Bulletin and Review

SN - 1069-9384

ER -