On the Limitations of Manipulation Checks: An Obstacle Toward Cumulative Science

Authors

  • Marie-Pierre Fayant Laboratoire de Psychologie Sociale: Menaces et Société (EA 4471), Université Paris Descartes Array
  • Harold Sigall Department of Psychology, University of Maryland, College Park Array
  • Aurore Lemonnier Laboratoire de Psychologie Sociale: Menaces et Société (EA 4471), Université Paris Descartes Array
  • Emilie Retsin Laboratoire de Psychologie Sociale: Menaces et Société (EA 4471), Université Paris Descartes Array
  • Theodore Alexopoulos Laboratoire de Psychologie Sociale: Menaces et Société (EA 4471), Université Paris Descartes Array

DOI:

https://doi.org/10.5334/irsp.102

Keywords:

best research practices, construct validity, experimentation, manipulation check

Abstract

Manipulation checks do not allow ruling out or accepting alternative explanations of causal effects (Sigall & Mills, 1998). In order to gauge the influence of this argument on current research practices, we surveyed the views of researchers on manipulation checks. Results confirmed that a manipulation check still stands as a totem of experimental rigor. Except in rare circumstances, such as when pilot testing, manipulation checks do not provide information relevant to construct validity. While it seems cost free to include seemingly informative manipulation checks, we claim it is actually costly because it wrongly enhances subjective confidence in the validity of research findings. We conclude that manipulation checks may hinder efforts to adopt a cumulative culture and practice of hypothesis testing.

Downloads

Published

2017-06-20

Issue

Vol. 30 No. 1 (2017)

Section

Short Research Note

Categories

License

Copyright (c) 2017 The Author(s)

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors who publish with this journal agree to the following terms (if a submission is rejected or withdrawn prior to publication, all rights return to the author(s)):

  1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
  2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).

Submitting to the journal implicitly confirms that all named authors and rights holders have agreed to the above terms of publication. It is the submitting author's responsibility to ensure all authors and relevant institutional bodies have given their agreement at the point of submission.

Note: some institutions require authors to seek written approval in relation to the terms of publication. Should this be required, authors can request a separate licence agreement document from the editorial team (e.g. authors who are Crown employees).