Comment on ‘‘Increasing chemistry students’ knowledge, confidence, and conceptual understanding of pH using a collaborative computer pH simulation’’ by S. W. Watson, A. V. Dubrovskiy and M. L. Peters, Chem. Educ. Res. Pract., 2020, 21, 528

Pract., 2020, 21, This journal is © The Royal Society of Chemistry 2020 Cite this: Chem. Educ. Res. Pract., 2020, 21, 1218 Comment on ‘‘Increasing chemistry students’ knowledge, confidence, and conceptual understanding of pH using a collaborative computer pH simulation’’ by S. W. Watson, A. V. Dubrovskiy and M. L. Peters, Chem. Educ. Res. Pract., 2020, 21, 528 Keith S. Taber This comment discusses some issues about the use and reporting of experimental studies in education, illustrated by a recently published study that claimed (i) that an educational innovation was effective despite outcomes not reaching statistical significance, and (ii) that this refuted the findings of an earlier study. The two key issues raised concern how the research community should understand the concept of refutation when comparing across studies, and whether the adoption of inferential statistics in a study should bind researchers to accept the inferences such tests suggest. Introduction A recent paper in CERP (Watson et al., 2020) reported an investigation of the value of collaborative use of a computer simulation (a ‘PhET sim’) in undergraduate learning. This was a quasi-experimental study, and because it was not possible to meet the conditions for a true experiment (such as randomisation of participants to conditions) the researchers made pre-test comparisons between those learners in the innovation and comparison conditions.† This is a well-motivated and interesting study, and it is reported in sufficient detail for a reader to evaluate, and acknowledges that the ‘‘study is best interpreted with a consideration for its limitations’’ (p. 534). This comment raises some issues about the interpretation and presentation of results in this study – issues that actually arise quite widely in experimental research in education (Taber, 2019). The authors conclude from their study that ‘‘the findings of this research study indicated that the collaborative PhET sim positively impacted students’ pH conceptual understandings, which both supports and refutes the literature’’ (p. 534), but it can be questioned whether this should be concluded given the reported results. More specifically, comments are offered regarding two statements found in the Discussion section of the paper (p. 533): (i) ‘‘This research study found that collaborative computer sim group members experienced higher mean scores regarding pH knowledge and conceptual understanding, and indicated higher levels of pH-related confidence from the beginning to the end of the semester when compared to the traditional group members’’; (ii) ‘‘our findings refute those of Hawkins and Phelps (2013) who found no statistical difference in learning gains between a group of students using a computerized sim on electrochemistry (treatment) versus a group of students who were taught electrochemistry via a traditional hands-on experiment (control)’’. Implications of adopting statistical tests It is common practice when reporting the results of experimental studies, such as those into the effectiveness of a teaching innovation, to not only report descriptive statistics (e.g., mean outcomes on the measures made), but to also use statistical tests to compare between conditions to see if differences in outcomes reach statistical significance. In their recent study, Watson and colleagues report both mean values for various measurements taken, and the outcomes of statistical tests. They report a number of comparisons between the pre-learning and post-learning Faculty of Education, University of Cambridge, UK. E-mail: Received 28th April 2020, Accepted 12th June 2020 DOI: 10.1039/d0rp00131g † Watson and colleagues refer to the experimental ‘‘computer sim group/intervention’’ condition, and the ‘‘traditional/control group’’. As well as simplifying language, a reason for preferring different terms here is that arguably both conditions were interventions (i.e., activities intended to bring about learning), and – as the authors acknowledge (p. 534) – there was no genuine control condition: for example, the two groups may have spent different amount of time working with the materials. The term comparison condition is widely used when there is no strict control condition. C