Recently, a few famous psychology findings were called into question by what some have called the reproducibility crisis. The reproducibility idea seems like common sense – take a study and do it again. If you get the same result, that’s evidence that the findings are true, and if the result doesn’t turn up again, they’re false.
Yet in practice, it’s nowhere near this simple. 3 reasons for this are –
1. Regression to the mean. Initial studies likely had a known bias toward overestimating the magnitude of an effect. So, it is natural that future studies will show a much lower effect.
2. Different design and analysis approaches. While researcher incentives can lead to certain questionable research practices, there is wide variance on design and analysis approaches even among the best researchers.
3. Hard to recreate the exact conditions. Finally, and this is especially true for “social sciences,” the human behavior and motives of the lab subjects is nearly impossible to replicate
So, how should we think of reproducibility? As with the soccer card study in part I, treat an individual study as a data point, encourage more studies and, taken together, we’re likely to get closer to the truth.
Science isn’t broken. It’s just a hell of a lot harder than we give it credit for. – Christie Aschwanden, Lead Sciene Writer @ the excellent FiveThirtyEight
Source and thanks to: The FiveThirtyEight Blog – Failure is Moving Science Forward