A call for cautious interpretation of meta-analytic reviews

Authors

Frank BoersFollow
Lara Bryfonski
Farahnaz Faez
Todd McKay

Document Type

Article

Publication Date

Fall 10-27-2020

Journal

Studies in Second Language Acquisition

Volume

1

Issue

43

First Page

2

Last Page

24

URL with Digital Object Identifier

https://doi.org/10.1017/S0272263120000327

Abstract

Meta-analytic reviews collect available empirical studies on a specified domain and calculate the average effect of a factor. Educators as well as researchers exploring a new domain of inquiry may rely on the conclusions from meta-analytic reviews rather than reading multiple primary studies. This article calls for caution in this regard, because the outcome of a meta-analysis is determined by how effect sizes are calculated, how factors are defined, and how studies are selected for inclusion. Three recently published meta-analyses are re-examined to illustrate these issues. One illustrates the risk of conflating effect sizes from studies with different design features, another illustrates problems with delineating the variable of interest, with implications for cause-effect relations, and the third illustrates the challenge of determining the eligibility of candidate studies. Replication attempts yield outcomes that differ from the three original meta-analyses, suggesting that also conclusions drawn from meta-analyses need to be interpreted cautiously.

Creative Commons License

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4.0 License.

Citation of this paper:

Boers, F., Bryfonski, L., Faez, F., & McKay, T. (2021). A call for cautious interpretation of meta-analytic reviews. Studies in Second Language Acquisition, 43(1), 2–24. https://doi.org/10.1017/S0272263120000327