jottings from tertius

views of the world from my worldview window

"If there was no God, there would be no atheists." G.K. Chesterton


Tektonics Apologetics Ministry
The Adarwinist reader
Bede's Library: the Alliance of Faith and Reason
A Christian Thinktank
Doxa:Christian theology and apologetics
He Lives
Mike Gene Teleologic
Errant Skeptics Research Institute
Stephen Jones' CreationEvolutionDesign
Touchstone: a journal of mere Christianity: mere comments
The Secularist Critique: Deconstructing secularism I Wasn't Born Again Yesterday
imago veritatis by Alan Myatt
Solid Rock Ministries
The Internet Monk: a webjournal by Michael Spencer
The Sydney Line: the website of Keith Windschuttle
Miranda Devine's writings in the Sydney Morning Herald
David Horowitz frontpage magazine
Thoughts of a 21st century Christian Philosopher
Steven Lovell's philosophical themes from C.S.Lewis
Peter S. Williams Christian philosophy and apologetics
Shandon L. Guthrie
Clayton Cramer's Blog
Andrew Bolt columns
Ann Coulter columns


This page is powered by Blogger. Isn't yours?

Blogarama - The Blog Directory

Blogroll Me!

"These are the days when the Christian is expected to praise every creed except his own." G.K.Chesterton

"You cannot grow a beard in a moment of passion." G.K.Chesterton

"As you perhaps know, I haven't always been a Christian. I didn't go to religion to make me happy. I always knew a bottle of Port would do that."C. S. Lewis

"I blog, therefore I am." Anon

Saturday, February 15, 2003

Peering into peer review

One reads a lot of quasi-mystical veneration of the process of peer review on sites dedicating to attacking the notion of Intelligent Design so I think a more balanced perspective should be brought to bear on the matter.

There are lots of definitions of peer review around but I submit that the following are more than acceptable:

"the process by which scientific papers are submitted to journals for publication are reviewed by peers in the field; peer reviewers typically evaluate the importance and usefulness of the research and assess whether the research was carried out in a methodologically appropriate manner and supports the conclusions drawn."

"Peer review is based on the concept that, because much of academic inquiry is relatively specialized, peers with similar expertise are in the best position to judge one another's work. This mechanism is largely designed to evaluate the relative quality of research. However, with appropriate feedback, it can also be a valuable tool to improve a manuscript, a grant application, or the focus of an academic career."

All pretty good stuff but as indicated above peer review goes beyond merely getting published.

"For much of the last century, peer review has been the principal mechanism
by which the quality of research is judged. In general, the most respected
research findings are those that are known to have been reviewed by a
researcher's peers. Most funding decisions in science are based on peer
review. Academic advancement is generally based on success in achieving
peer-reviewed publication and funding, and typically include peer review of
the candidate's academic career. Clearly, research depends greatly on peer

Despite this there remains some confusion over what exactly constitutes
"peer review".

'In a recent systematic review of 1,500 articles and studies on peer review published prior to 1998, I concluded that... Peer review's "outstanding weakness is that errors of judgment, either unintentional or intentional, are sometimes made. ...[There is a] lack of guidelines and standards. Editors have great flexibility in their implementation of the process in their journals ...[and] there is not one solid, accepted definition of what constitutes peer review," other than the basic definition of evaluation of a work by one's peers.

...Similar to the findings in the systematic review of peer review by this
writer and previous congresses before this one, studies providing
unquestionable evidence of the value of peer review remain elusive.

... Given the human dilemma when making judgments of colleagues' works,
editorial peer review is a process that will never be totally codified,
verified, or universally accepted, nor is it likely to receive one final, definitive
stamp of approval (or disapproval)...'

Peer Review: Do Studies Prove Its Effectiveness? by Ann C. Weller

The process of peer review has come to be seen as an essential part to the
scientific enterprise and in general it does seem to work quite well. I am
certainly not anti peer-review but someone needs to puncture the illusion
that it represents some sort of "holy of holies" in science. The theory
itself sounds wonderful but in practice it is not without problems and
failings, and certainly not without criticism. There exists in fact quite
an extensive literature about the problems of the peer review process; so
extensive in fact that only Blind Freddie could pretend to be unaware of the
concerns about fraud and the failings of peer review that have been
published. These problems should be well known to anyone who has even a
cursory interest in the question of honesty, integrity and fairness in
scientific research and who is not blinded by a messianic faith in scientism.

Anyone willing to do a little investigating by searching the Net using "peer
review" "fraud" "problems" "failure" "misconduct in science" etc. etc. will
soon discover the darker underbelly of science. To claim that the
imprimatur of favourable peer review confers infallibility upon some
research paradigm is an overly simplistic and naive assumption. Peer review
may be a pretty good at filtering out dross, but it's not fool-proof:

"Peer review is like democracy which is, to use Churchill's phrase, the
worst form of government except all those other forms that have been tried
from time to time."

There is always potential for fraud:

"A peer reviewer must preserve scholarly integrity by rising above the three
deadly sins of intellectual life: envy, favoritism, and the temptation to
from "Ethics in Research. What's the verdict on peer review?" by Tom Abate [Abate gives a good overview of peer review and its history including pros and cons.]

No one would deny that there are positives in peer review, what some are
failing to appreciate is that "the process of peer review is hampered by
both perceived and real limitations." Peer Review: Do Studies Prove Its Effectiveness?

I find people, particularly those with an anti-creationist or anti-intelligent design agenda, regularly extol the virtues of peer review without any acknowledgment of the many criticisms that are increasingly being levelled at it. Having recently read the book "Impure Science: Fraud, Compromise and Political Influence in Scientific Research" by Robert Bell, (Wiley and Sons 1992) the reality of peer review abuse and fraud is fresh in my mind. I would recommend those who haven't read the book should try to get a copy and to peruse particularly the case studies in the chapter about the abuse of peer review. However there are literally hundreds of articles online both from respected journals as well as in more popular sources chronicling case studies and highlighting the failings and problems of the process. There are so many links to articles about the problems of peer review on the Net that it could legitimately be described not as a trickle but as a flood. The criticism have been aired in respectable forums and journals for some time now and are not the rantings of psuedoscientists or crackpots or nutcases.

Let me suggest a good place to begin is the bibliography of 88 references to
articles dealing with problems of peer review (including brief abstracts) on
this site from the DePaul College of Law in Chicago:

Some excerpts:

British Medical Journal:
"And now, Evidence Based Editing," Vol. 311, No. 7009, pg. 826, Sept. 30,
1995.Peer review can also be corruptible which was best illustrated in
previous communistic countries. Although peer review is imperfect,
it's better than nothing.

"Peer Review: Reform or Revolution?" Vol. 315, No. 7111, pg. 759, Sept.
27, 1997. Peer review is expensive, slow, biased, abused, anti-innovative,
and can't detect fraud.

Journal of the American Medical Association (JAMA):
"The Philosophical Basis of Peer Review and the Suppression of
Innovation," Vol. 263, No. 10, pg. 1438, Mar. 9, 1990. Modern peer review
blocks a lot of innovative work from being published. Peer reviewers may
just make comments as to their own opinion on a subject, rather than an
objective critique of the paper. To publish innovative ideas an editor needs
to choose the reviewer carefully.

"More Peering Into Editorial Peer Review," Vol. 270, No. 23, pg. 2856,
Dec. 15, 1993. Opponents feel peer review is a dictatorship, reviewers are
self-serving, its too secretive, there are no clear standards, it suppresses
innovation, its expensive, ideas are stolen, publishing is delayed, and it
still doesn't guarantee a good article. Even so, it is the only thing out
there and blind review makes it even better.

"Peer Review," "Evaluating Peer Reviews," Vol. 272, No. 2, pg. 96 and pg.
98, July 13, 1994. Deficiencies of how peer review is designed: 1)
inadequately controlled; 2) biased, and 3) inadequately defined

The New England Journal of Medicine:
"Fraud in Biomedical Research," Vol. 318, No. 22, pg. 1462, June 2, 1988.
Peer review is supposed to self correct fraud in research papers, but it is
much too heavily relied upon

"How Good is Peer Review?" Vol. 321, No. 212, pg. 827, Sept. 21, 1989.
Peer review cannot be relied upon to guarantee the validity or honesty of
scientific research.

Technology Review
"Peer Review: Treacherous Servant, Disastrous Master," Oct. 1991, pg. 29.
In review that is not blinded, cronyism is evident. Additionally, referees
have a lot of power that can be misused. Peer review can keep out good
articles as well as bad ones. One suggestion is that specialist journals
shouldn't reject articles and allow the readers to weed out the junk. The
generalized journals should make the editors make the decision with little
or no peer review

Then could I suggest a visit to the site of The Fourth International Congress on Peer Review in Biomedical Publications, organized by JAMA - the Journal of the American Medical Association and the British Medical Journal Publishing Group, that met in Barcelona Sept.14-16, 2001 featuring presentations of original research on peer review...

Here is a summary of the findings of one report presented at that

'Does Peer Review Help? Peer review is a pillar of science. Letting scientists
anonymously judge each other's work is widely considered the "least bad way"
to weed out weak manuscripts or research proposals and improve promising
ones. But this common wisdom was questioned at a meeting here from 14 to 16
September in a study that found little evidence that peer review actually
improves the quality of research papers.

Mention "peer review" and almost every scientist will spout stories about
referees submitting nasty comments, sitting on a manuscript forever, or
rejecting a paper only to repeat the study and steal the glory. Despite its
flaws, most respected journals rely on peer review to choose which studies
to publish. At the Fourth International Congress on Peer Review in
Biomedical Publication, hundreds of editors of medical journals and
academics met to examine the process.

In a meta-analysis that surprised many--and that some doubt--a team led by
Tom Jefferson of the Cochrane Centre in Oxford, United Kingdom, scrounged
the literature for studies that had studied peer review rigorously. They
found 19 papers that fit their criteria, but none clinched the case for peer
review. For instance, nine studies looked at the effects of blinding the
reviewers to the authors or vice versa, but none found much difference in
quality. Two other studies found scant evidence that a standardized
checklist led to better reviews, while two more revealed that training
reviewers was practically useless. Only two papers compared the quality of
papers submitted as a manuscript with the version that later appeared in
print, and their results were difficult to generalize.

Jefferson's study--which, like all contributions at the meeting, had been
peer-reviewed--was "pretty depressing," concedes British Medical Journal
editor Richard Smith. Still, Smith and other editors remain convinced that
the review process helps, even if studies can't objectively show it.

I particularly love that comment from British Medical Journal editor Richard
Smith so I'll replay it:

"Jefferson's study--which, like all contributions at the meeting, had been
peer-reviewed--was "pretty depressing," concedes British Medical Journal
editor Richard Smith. Still, Smith and other editors remain convinced that
the review process helps, even if studies can't objectively show it."

Will all philosophic naturalists and materialists please ponder how it
"helps even if studies can't objectively show it"; sounds like a religious
experience or a faith position to me. Surely not from hard-headed

Then there is this report:

'A recent editorial in Nature sums up the position in which practitioners of
peer review find themselves. Most researchers agree that peer review is the
least imperfect way of upholding the quality of scientific publishing," noted the editorialist. "But those who administer it also have to cope with, and attempt to solve, the problems that peer review gives rise to."

Biomedical Journals Ponder the Failures and Remedies of Peer Review by Joan Stephenson, PhD

"least imperfect"...!?

And this:
Shortcomings of peer review in biomedical journals by Elizabeth Wager and Tom Jefferson; Learned Publishing Volume: 14 Number: 4 Page: 257 -- 263

"Peer review is well established across most academic disciplines,
and publishers, editors, and researchers devote considerable resources to
it. This paper uses examples from biomedical journals to examine its
shortcomings. Although mainly anecdotal, the evidence suggests that peer
review is sometimes ineffective at identifying important research and even
less effective at detecting fraud. Most reviewers identify only the minority
of a paper's defects and they may be biased. Peer review plus other
editorial processes are associated with improvements in papers between
submission and publication, but published papers remain hard to read and a
significant proportion contain errors or omissions. While it is hard to
quantify the costs, peer review does not seem an efficient use of resources.
Research into the outcomes of peer review, the establishment of sound
methods for measuring the quality of the process and its outcomes, and
comparisons with alternative methods are needed."

Or this link: Publish or Perish - Fraud and Ethics by Alfred E. Hartemink
(First published in the IUSS Bulletin 2000 vol. 97, 36-45)

And the very interesting work of Dr Brian Martin associate professor in
Science, Technology and Society at the University of Wollongong:
"Peer review as scholarly conformity" which constitutes Chapter 5 of his book "Suppression Stories" Wollongong: Fund for Intellectual Dissent, 1997, pages 69-83.

"Although the rationale for peer review is quality control, it's obvious
that the process can be used to suppress dissent. It's a powerful method:
peer review can be used to block publications, appointments, promotions and
grants. Most importantly, it is very difficult to demonstrate that bias is
involved. Usually referees are anonymous: only their reports are made
available. Members of selection committees carry out their deliberations in
secret: only a decision and perhaps a brief justification is needed. It is
very difficult to collect systematic information about the role of peer
review in squashing dissent. But in the course of looking into suppression
I've come across a few dramatic cases..."
[case studies then presented]

I would also offer the following extended quote from David Hershey:

"The overrated peer review system for the research literature is very uneven
and has several inherent weaknesses. First of all it is strictly voluntary,
and manuscript reviewers receive little or no professional credit for their
efforts. In the publish or perish world of research, reviewing manuscripts
is not a good use of a researcher's time so doing a poor or cursory
reviewing job may actually be beneficial because poor reviewers may receive
fewer manuscripts to review in the future. The peer review system is
backwards in the sense that once a manuscript is published, it is very
difficult to criticize it in print. Journals often refuse to publish
letters-to-the-editor pointing out flaws in their research articles. If they
do publish such letters, the original authors get the final word in their
rebuttal and often make personal attacks upon the letter writer rather than
discussing the scientific issues raised. A decent peer review system would
allow, indeed welcome, objective evaluations after publication when
thousands more scientists have had an opportunity to read the article.

Several times as a peer reviewer, I have pointed out major flaws in
manuscripts which the editors and authors simply ignored and published the
manuscript anyway. In other cases, the editor rejected the paper because of
the flaws but the authors simply published the uncorrected manuscript in
another journal. Unethical reviewers can easily misuse the peer review
system for their own benefit resulting in valid research being rejected for
no good reason. I had a paper rejected and after reading my manuscript, the
reviewer who rejected it immediately ran similar experiments and tried to
publish the results in the same journal. Another manuscript was rejected for
no valid reasons by an Associate Editor. I appealed to the Editor, and the
manuscript was published unchanged. Later I received a complimentary letter
about the article from a former President of the scientific society that
published the journal.

I am not aware that peer review itself has been subjected to much scientific
scrutiny. I don't remember all the details, but I know there was one study
on peer review where someone took a published article, retyped it in
manuscript form and sent it to about ten peer reviewers. Quite a few, maybe
even a majority, rejected it which illustrated how haphazard peer review can
be. I also remember that the author of the peer review study got in hot
water because he punctured the myth of peer review.

There are many spectacular failures of the peer review system. In the book,
The Double Helix, Watson and Crick mention one published article on a
proposed structure for DNA that violated basic rules of chemistry but was
published anyway because of the author's reputation. Several years ago,
Plant Physiology published a paper claiming that ethylene increased membrane
permeability of red beet root tissue. In a subsequent issue, others showed
that the results were an artifact caused by the low pH of the Ethephon
solution used to supply ethylene. Anyone familiar with Ethephon should have
known that when they read the first manuscript and rejected it. The journal
Science published a 1977 paper claiming that digestion by dodo birds was
required to germinate seeds of the tambalacoque tree. The paper never should
have been published because there was no control treatment, and previous
literature showed that tambalacoque seeds germinated without dodo birds
involved. However, despite later reports to the contrary, the
dodo-tambalacoque story is presented as an undisputed example of obligate
plant-animal mutualism in many textbooks.

You need to maintain some scientific skepticism even for peer-reviewed
literature. As Henry Fonda's character said in the movie Spencer's Mountain,
"Just cause it's wrote down don't make it so."

Hershey's comment below is also relevant in the light of the debate
generated by Jon Wells "Icons of Evolution" about fraud and
misrepresentation in biology tests. This quote is not made with reference to
Wells' book but is an independent assessment predating Wells from a
botanist who recognises failings in these texts.

"Textbooks, particularly introductory biology textbooks for high school and
college, often contain many botanical inaccuracies despite being peer
reviewed. There seem to be multiple reasons for this including that many
biology textbook authors have little or no training in botany. Based on
personal experience, I know that the editors of biology textbooks often
reword sections written by the author and inadvertently change the meaning.
I have also seen editors deliberately change meanings without consulting the
author, illogically relying on their incomplete understanding of botany
rather than on the author's expertise. There are dozens of botanical errors,
or what the politically correct call misconceptions, that are widespread in
introductory biology texts and even some higher level texts."

David R. Hershey, The truth behind some great plant stories. American Biology Teacher 2000, 62:408-413.

Some more criticisms of peer review
[an incomplete list drawn from various sources]

Reviewers may take advantage of ideas in manuscripts that are not yet
published and grant proposals that are not yet funded

Reviewers may be biased in favour of well-known researchers or institutions

They may review the work of a competitor unfairly

They may be insufficiently qualified to provide an authoritative review

Studies have found that reviewers may be less likely to criticize work that
is consistent with their own perceptions (Ernst and Resch, 1994); or to award a fellowship to a woman rather than a man (Wennerds and Wold, 1997).

The secretiveness of the process, making the possibility of abuses more
likely, such as undisclosed conflicts of interest among reviewers.

Research-related problems, such as major flaws in conception or design, are
beyond the power of editorial peer review to remedy.

Reviewers may not spend enough time critically reviewing the methods and
procedures of the research.

Peer review lacks explicit standards for review

It lacks objectivity and fairness in the selection of manuscripts for review

It favours conservative, traditional research and excludes innovative work
by independent researchers

Editors may be biased in their selection of reviewers, sending manuscripts
only to those individuals who share the opinions of the editorial board


the Plight of "Mavericks"

Tunnel Vision


Conflict of Interest

Lobbying and Political Pressure

Because the process of peer review is highly subjective, it is possible that
some people will abuse the process or act based on intentional or unintentional biases.

Then there is the well known "Catch 22" of the peer review process which is
of particular relevance to the intelligent design paradigm: reviewers often
criticize proposals or papers on the ground that the research lacks "a
sufficient predicate" in previously published research.

"Because the panels are representative of the opposite school of [thought],
and it's very threatening for a scientist to see a new idea. It's automatically shot down."

See also the The Potential for Bias in Peer Review by Thomas O. McGarity

Tomorrow's Cures Today? How to Reform the Health Research System
Donald R. Forsdyke. New York: Harwood Academic Publishers, 2000;
A review by Neena L. Chappell
, Professor of Sociology and director of the
Centre on Aging at the University of Victoria.

'Forsdyke's criticisms of peer review are numerous: excellent researchers
maynot always be excellent communicators and therefore may not be able to write for science peers on committees, there is a skill required in grant writing; we need as much excellence in the design of our peer review system as we need excellence in science.

He also notes: the existing system is error-prone and our recognition both
of scientists and of scientific discoveries is also error-prone; good
scientists often go unrecognized and similarly good ideas go unrecognized.
Often, the more creative the thinking, the less communicable it becomes; and
"The dynamics of peer-resistance of novel ideas in the Victorian era may be
similar to those of the present era." (p. 54)

The current system encourages safe, popular areas likely to produce
publishable results in a short time, and forces them away from interfaces
between the disciplines. It tends towards destructive comments rather than
constructive comments; it will not fund contemplative or theoretical work,
and, whether a particular proposal is funded or not depends on who reviews

One may also fruitfully peruse this extensive Bibliography on Misconduct in
prepared by C. K. Gunsalus

Or examine cases of Misconduct in Science here:

Or read and view excerpts including case studies from the NOVA program" Do
Scientists Cheat?"

"NOVA examines ethics in science with an emphasis on biomedical
research. Factors contributing to "bad science" include sloppy research,
personal bias, lack of objectivity, "cooking and trimming", "publish or
perish" pressure, and outright fraud. The limits of peer review and other
quality control systems are discussed."

Or one scientist's personal crusade:

Then there is this from the editor of "The Scientist":
Religion, Rebel Scientists, And Peer Review: Three Hot Topics by Eugene Garfield, THE SCIENTIST 2(24):10,26December 1988

"Science's intolerance-if it is that-for minority views also seems to be
evident in the third most written-about subject: peer review. As I have had
occasion to observe in the past, there is great dissatisfaction with peer
review throughout the scientific community, and especially with blind
reviewing and the lack of recourse a scientist has if his or her work is
reviewed unfairly.What all three of these topics share is a concern with the
standards, professional and ethical, that the scientific community takes
an interest in-matters distinct from the process of scientific investigation
itself. I believe that this interest refutes the common perception that the
scientific endeavor is a value-free enterprise. In fact like all other human
endeavors, science is replete with expressions of value-moral and ethical.
That is not a revelation,but it is important to remind ourselves of this
now and again."

Scientific Misconduct and Peer Review: Is There a Case for an International Medical Scientific Press Council? Douglas G. Altman; Iain Chalmers, MSc; Andrew Herxheimer, FRCP

"Serious abuse of editorial power is rarely publicized, but evidence that it
occurs is accumulating. Authors who believe that they have been dealt with
unfairly have little possibility of a hearing of their complaint, and cases
cannot easily be publicized because of fears of legal action. We describe
briefly three cases in which the alleged misdeeds indicate that there were
legitimate questions that needed answers. In the first case, an editor
republished a previously published article without the authors' permission
(but stated the opposite), attacked it in an accompanying editorial, and
then denied the authors the right of reply. The other cases concerned a
commissioned review article that was plagiarized and an editor with an
undisclosed vested interest...
(JAMA. 1994;272:166-167)

etc., etc.

I give the final word to the judiciary:
"[W]hile peer review improves the overall quality of research, as well as
the likely integrity of individual publications, it does not mechanically
guarantee reliability. Courts therefore should neither be wholly dismissive
of peer review nor accord it more respect than it deserves."

Sheila Jasanoff (1997). Judging Science: Issues, Assumptions, Models.

10:17:00 am