[ad_1]
At this point, it is hardly surprising to learn that even the best scientific journals publish a lot of mediocre quality work – not just serious experiments that lead, unfortunately, to conclusions that do not stand up to repetition, studies poorly designed that have no real chance of succeeding before ever being conducted.
Studies that had died on arrival. We have seen many examples.
In 1996, a psychological study claimed that non-intrusive priming – the insertion of certain harmless words into a quiz – could lead to a consistent change in behavior.
This document has been cited by other scientists several thousand times – before missed replications many years later clearly show that this discovery, and much of later literature, was nothing more than more than researchers looking for noise models.
As a political scientist, my favorite was the investigation that, in 2012, women were 20 times more likely to support Barack Obama in the presidency for some days of their monthly cycle.
In retrospect, this statement is meaningless and unsupported by data. Even prospectively, the experiment had no chance of succeeding: the way the study was conducted, the noise caused by the estimation of any effect – in this case, any average difference in political attitudes during different parts of the cycle – was much larger than any realistic signal. (actual result)
[[[[Like the Science Times page on Facebook. | Sign up for the Science Times Bulletin.]
We see it all the time. Do you remember claims that subliminal smiles on a computer screen can cause big changes in attitude towards immigration? That elections are decided by college football games and shark attacks? These studies have been published in serious journals or promoted in serious media.
Scientists know that it is a problem. In a recent article published in the journal Nature Human Behavior, a team of renowned economists and psychologists published the results of 21 repetitions of high-profile experiments.
Replication is important for scientists because it means that the discovery may be real. In this study, many results could not be reproduced. On average, the results accounted for only about half of the claims originally published.
This is where it gets really weird. A group of experts using a "prediction market" had predicted in advance the lack of replication, allowing experts to bet on the most likely experiences of (real, be).
Similar prediction markets have been used for many years for the elections, mimicking the movement of the sports betting line. Essentially, the results in this case indicate that informed scientists made it clear from the start that what they read would not be sustainable.
So yes, it's a problem. There has been reluctance to solve it, some of whom have come from prominent researchers at prominent universities. But many, if not most, scientists are aware of the seriousness of the replication crisis and fear its corrosive effects on public confidence in science.
The challenge is what to do next. One of the possible solutions is pre-registration, in which researchers who start a study publish their analysis plan before collecting their data.
Pre-registration can be seen as a sort of replication inverted in time, a firewall against "data dredging", the tendency to look for results when your first idea does not materialize.
But that will not solve the problem alone.
The crisis of replication in science is often presented as a matter of scientific procedure or integrity. But the whole careful procedure and all the honesty of the world will not help you if your signal (the motive you are looking for) is small and if the variation (all the confounding factors, the other things that might explain this motive) is strong.
From this point of view, the scientific crisis is more fundamental and involves going beyond the existing model of systematic discovery.
Suppose you want to study the effects of a drug or educational innovation on a small number of people. Unless the treatment is very clearly targeting a result of interest (for example, a mathematics program focused on a particular standardized test), your study may be too noisy – there will be too many variables – to identify real effects.
If something random happens and reaches statistical significance, it is likely to be a massive overestimation of any real effect. When attempting to replicate, we will probably see something much closer to zero.
Replication failures are not a surprise to many scientists, including myself, who have a lot of experience of false starts and dead ends in our own research.
The big problem of science is not cheaters nor opportunists, but sincere researchers who have unfortunately been trained to think that each statistically "significant" result is remarkable.
When you read articles about research in the news media (and that, as a taxpayer, you are also a funder of the research), you should ask what exactly is measured and why.
Source link
