December 2, 2011

Scientists' Elusive Goal: Reproducing Study Results

Two years ago, a group of Boston researchers published a study
describing how they had destroyed cancer tumors by targeting a protein
called STK33. Scientists at biotechnology firm Amgen Inc. AMGN?+1.21%
quickly pounced on the idea and assigned two dozen researchers to try
to repeat the experiment with a goal of turning the findings into a
drug.

WSJ's Gautam Naik has details of challenges scientists face in
reproducing claims made by medical journals. Photo: Sandy Huffaker/The
New York Times

It proved to be a waste of time and money. After six months of
intensive lab work, Amgen found it couldn't replicate the results and
scrapped the project.
vg
"I was disappointed but not surprised," says Glenn Begley, vice
president of research at Amgen of Thousand Oaks, Calif. "More often
than not, we are unable to reproduce findings" published by
researchers in journals.

This is one of medicine's dirty secrets: Most results, including those
that appear in top-flight peer-reviewed journals, can't be reproduced.

Enlarge Image

Bayer
Researchers at Bayer's labs often find their experiments fail to match
claims made in the scientific literature.

"It's a very serious and disturbing issue because it obviously
misleads people" who implicitly trust findings published in a
respected peer-reviewed journal, says Bruce Alberts, editor of
Science. On Friday, the U.S. journal is devoting a large chunk of its
Dec. 2 issue to the problem of scientific replication.

Reproducibility is the foundation of all modern research, the standard
by which scientific claims are evaluated. In the U.S. alone,
biomedical research is a $100-billion-year enterprise. So when
published medical findings can't be validated by others, there are
major consequences.

Drug manufacturers rely heavily on early-stage academic research and
can waste millions of dollars on products if the original results are
later shown to be unreliable. Patients may enroll in clinical trials
based on conflicting data, and sometimes see no benefits or suffer
harmful side effects.

There is also a more insidious and pervasive problem: a preference for
positive results.

Unlike pharmaceutical companies, academic researchers rarely conduct
experiments in a "blinded" manner. This makes it easier to cherry-pick
statistical findings that support a positive result. In the quest for
jobs and funding, especially in an era of economic malaise, the
growing army of scientists need more successful experiments to their
name, not failed ones. An explosion of scientific and academic
journals has added to the pressure.

When it comes to results that can't be replicated, Dr. Alberts says
the increasing intricacy of experiments may be largely to blame. "It
has to do with the complexity of biology and the fact that methods
[used in labs] are getting more sophisticated," he says.

It is hard to assess whether the reproducibility problem has been
getting worse over the years; there are some signs suggesting it could
be. For example, the success rate of Phase 2 human trialswhere a
drug's efficacy is measuredfell to 18% in 2008-2010 from 28% in
2006-2007, according to a global analysis published in the journal
Nature Reviews in May.

Read More:
http://www.***.com/