Shapefuture provides a better environment for O Level, IGCSE, AS and A Level Training.

'Replication crisis' spurs reforms in how science studies are done

But some researchers say the focus on reproducibility ignores a larger problem

PONDERING SCIENCE A new investigation indicates that reproducibility of social science studies (including one investigating how contemplation of Rodin's The Thinker statue affects religious beliefs), while not great, is improving. But intensifying replication efforts may address only surface problems.

What started out a few years ago as a crisis of confidence in scientific results has evolved into an opportunity for improvement. Researchers and journal editors are exposing how studies get done and encouraging independent redos of published reports. And there's nothing like the string of failed replications to spur improved scientific practice. That's the conclusion of a research team, led by Caltech economist Colin Camerer, that examined 21 social science papers published in two major scientific journals, Nature and Science, from 2010 to 2015. Five replication teams directed by coauthors of the new study successfully reproduced effects reported for 13 of those investigations, the researchers report online August 27 in Nature Human Behavior. Results reported in eight papers could not be replicated.

The new study is an improvement over a previous attempt to replicate psychology findings (SN: 4/2/16, p. 8). But the latest results underscore the need to view any single study with caution, a lesson that many researchers and journal gatekeepers have taken to heart over the past few years, Camerer's team says. An opportunity now exists to create a scientific culture of replication that provides a check on what ends up getting published and publicized, the researchers contend. Still, the new study reveals a troubling aspect of successful experimental redos. Camerer's team found that for repeat studies that panned out, which included four to five times as many participants as originally studied, the statistical strength to detect actual effects was weaker than reported for the initial investigations. In other words, the best replications - which exceeded initial studies in their ability to detect actual effects - were only partially successful.

One reason for that trend is that scientific journals have tended not to publish studies that disconfirm previous findings, leaving initial findings unchallenged until now, says study coauthor and psychologist Brian Nosek of the University of Virginia in Charlottesville. Even the most prestigious journals have often published results that garner lots of scientific and media attention but that could easily have occurred randomly, he says. On the plus side, the new report appears as such practices are changing. "The social and behavioral sciences are in the midst of a reformation in scientific practices," Nosek says. In the last five years, for example, 19 of 33 journals in social and personality psychology have established policies requiring investigators to submit their research designs for peer review before submitting research papers for review. In this way, peer reviewers can check whether experimenters altered their procedures to tease out positive effects. The same journals also collect experimental data from researchers so that replications can be conducted. Intriguingly, when Camerer's group asked a group of nearly 400 researchers, mostly psychologists and economists, to examine data from the 21 experiments and predict whether each could be reproduced, the scientists' forecasts were usually correct. Peer predictions may be one way to bolster peer reviews and help weed out weak studies, Nosek says.

News Source