“This is one of medicine’s dirty secrets: Most results, including those that appear in top-flight peer-reviewed journals, can’t be reproduced.”
The Wall Street Journal reports,
Two years ago, a group of Boston researchers published a study describing how they had destroyed cancer tumors by targeting a protein called STK33. Scientists at biotechnology firm Amgen Inc. quickly pounced on the idea and assigned two dozen researchers to try to repeat the experiment with a goal of turning the findings into a drug.
It proved to be a waste of time and money. After six months of intensive lab work, Amgen found it couldn’t replicate the results and scrapped the project.
“I was disappointed but not surprised,” says Glenn Begley, vice president of research at Amgen of Thousand Oaks, Calif. “More often than not, we are unable to reproduce findings” published by researchers in journals.
This is one of medicine’s dirty secrets: Most results, including those that appear in top-flight peer-reviewed journals, can’t be reproduced…
There is also a more insidious and pervasive problem: a preference for positive results.
Unlike pharmaceutical companies, academic researchers rarely conduct experiments in a “blinded” manner. This makes it easier to cherry-pick statistical findings that support a positive result. In the quest for jobs and funding, especially in an era of economic malaise, the growing army of scientists need more successful experiments to their name, not failed ones. An explosion of scientific and academic journals has added to the pressure.
When it comes to results that can’t be replicated, Dr. Alberts says the increasing intricacy of experiments may be largely to blame. “It has to do with the complexity of biology and the fact that methods [used in labs] are getting more sophisticated,” he says…
Right on Bill. An experiment produces results. “Positive” and “Negative” are opinions about the meaning of the data.
The infamous 95% rule is a large part of the problem. 95% confidence means there is 1 chance in 20 that they are wrong. That is a pretty good chance of being wrong. It takes about 10 replications to get the odds up to where they are really well supported.
This ties right into the NYT article a year or so ago by the researcher who spent 10 years trying to reproduce the results of his PhD thesis. Every time he tried, the results showed weaker and weaker results, until he had to conclude that his thesis was wrong.
…and return any funds based on their privious “success.”
Yeah, nice idea. It’s about as likely as a doctor refunding money charged for treating a patient who dies….
If pharmaceutical companies cannot reproduce the results of academic researchers then the academic researchers should be charged with reproducing their own results. If they cannot they should withdraw their paper and return any funds based on their privious “success.”
True science must be observable and repeatable, otherwise it’s garbage.
The dumbest thing ever. Negative results are just as important as positive ones in science.