Two major study retractions in one month have left researchers wondering if the peer review process is broken.
One study promised that popular blood-pressure drugs were safe for people infected with the coronavirus. Another paper warned that anti-malaria drugs endorsed by President Trump actually were dangerous to these patients.
The studies, published in the New England Journal of Medicine and The Lancet, were retracted shortly after publication, following an outcry from researchers who saw obvious flaws.
The hasty retractions, on the same day this month, have alarmed scientists worldwide who fear that the rush for research on the coronavirus has overwhelmed the peer review process and opened the door to fraud, threatening the credibility of respected medical journals just when they are needed most.
Peer review is supposed to safeguard the quality of scientific research. When a journal receives a manuscript, the editors ask three or more experts in the field for comments. The reviewers’ written assessments may force revisions in a paper or prompt the journal to reject the work altogether. The system, widely adopted by medical journals in the middle of the 20th century, undergirds scientific discourse around the world.
“The problem with trust is that it’s too easy to lose and too hard to get back,” said Dr. Jerome Kassirer, a former editor in chief of the New England Journal of Medicine, which published one of the retracted papers in early May. “These are big blunders.”
If outside scientists detected problems that weren’t identified by the peer reviewers, then the journals failed, he said. Like hundreds of other researchers, Dr. Kassirer called on the editors to publish full explanations of what happened.
Image
In interviews with The New York Times, Dr. Richard Horton, the editor in chief of The Lancet, and Dr. Eric Rubin, editor in chief of the N.E.J.M., said that the studies should never have appeared in their journals but insisted that the review process was still working.
“We shouldn’t have published this,” Dr. Rubin said of the study appearing in the N.E.J.M. “We should have had reviewers who would recognize the problem.”
Dr. Horton called the paper retracted by his journal a “fabrication” and “a monumental fraud.” But peer review was never intended to detect outright deceit, he said, and anyone who thinks otherwise has “a fundamental misunderstanding of what peer review is.”
“If you have an author who deliberately tries to mislead, it’s surprisingly easy for them to do so,” he said.
In addition, the editors said, there is an urgent need to rapidly publish new findings to improve treatments for desperately ill coronavirus patients. Since the pandemic began, The Lancet is receiving three times the usual number of papers for consideration, Dr. Horton said. And the N.E.J.M. has fielded as many as 200 submissions in a day, including essays, according to Dr. Rubin.
“I’m an infectious disease doctor, I treat Covid-19 patients,” Dr. Rubin said. “I’ve been in the hospital recently treating patients, and we have no idea what to do. I’m the primary driver at the journal of saying, ‘We have to get data out there that people can use.’”
“We are very careful,” he added. “At our editorial meetings, this comes up almost every day. ‘If we publish this, will it hurt people?’ That’s our biggest concern.”
The N.E.J.M. and The Lancet are among the oldest, most respected and most widely read medical journals in the world. They were established in 1821 and 1823 and are ranked often first and second among general-interest medical journals by their “impact factor,” the frequency with which their studies are cited in other research.
A report in one of these journals can have immediate repercussions both for patients and for research. After The Lancet’s initial publication of the study concluding that the antimalarial drugs chloroquine and hydroxychloroquine endangered the lives of coronavirus patients, the World Health Organization and other groups halted clinical trials of the drugs while safety reviews were conducted.
Image
The reputation of these journals rests in large part on vigorous peer review. But the process is opaque and fallible: Journals generally do not disclose who reviewed a study, what they found, how long it took or even when a manuscript was submitted. Dr. Horton and Dr. Rubin declined to provide those details regarding the retracted studies, as well.
Critics have long worried that the safeguards are cracking, and have called on medical journals to operate with greater transparency.
“We are in the midst of a pandemic, and science is moving really fast, so there are extenuating circumstances here,” said Dr. Ivan Oransky, co-founder of Retraction Watch, which tracks discredited research.
“But peer review fails more often than anyone admits,” he said. “We should be surprised it catches anything at all, the way it’s set up.”
Journals used to take many months, or even a full year, to scrutinize and edit a complicated study, a process that included several weeks for outside experts to peer review the research.
Now peer review may be condensed to as little as 48 hours; some studies deemed of vital importance to patients may be published online within 20 days of submission.
“There is always a tension between getting it fast and getting it right,” said Dr. Marcia Angell, another former editor in chief of the N.E.J.M. “I always favored getting it right. But in the current pandemic, that balance may have shifted too far toward getting it fast.”
It’s not just the journal editors who are inundated. Knowledgeable scientists who donate time as peer reviewers are already stretched thin, trying to understand how the coronavirus affects the body, or to find treatments and vaccines. The research is happening at an unprecedented pace.
“I think the academic system is saturated — it’s at capacity,” said Dr. Peter Jüni, a professor of epidemiology at the University of Toronto who has reviewed papers for scientific journals.
“People are tired; they’re working at the edge of their limits,” he said. “They struggle to get good peer reviewers and try to do as well as they can, but the system is at risk of failing, as you see here.”
The retracted paper in The Lancet should have raised immediate concerns, he added. It purported to rely on detailed medical records from 96,000 patients with Covid-19, the illness caused by the coronavirus, at nearly 700 hospitals on six continents. It was an enormous international registry, yet scientists had not heard of it.
The data were immaculate, he noted. There were few missing variables: Race appeared to have been recorded for nearly everyone. So was weight. Smoking rates didn’t vary much between continents, nor did rates of hypertension.
“I got goose bumps reading it,” said Dr. Jüni, who is involved in clinical trials of hydroxychloroquine. “Nobody has complete data on all these variables. It’s impossible. You can’t.”
Both retracted studies were led by Dr. Mandeep R. Mehra, a widely published and highly regarded professor of medicine at Harvard, and the medical director of the Heart and Vascular Center at Brigham and Women’s Hospital.
In a statement last week, Dr. Mehra apologized for the retractions, which he attributed to an eagerness to publish helpful information during the pandemic. He stopped short of calling them fraud, saying only that the data could not be verified by independent auditors.
The data in both studies were produced by a small company outside Chicago called Surgisphere, run by another of the papers’ authors, Dr. Sapan Desai. In an interview with The New York Times in May, Dr. Desai vigorously defended his work and the authenticity of his data registry, which he said included patient records from 1,200 hospitals and other health facilities around the world.
But when the N.E.J.M. and The Lancet demanded independent audits, he refused, citing confidentiality agreements with client hospitals. Following the retractions, Dr. Desai has declined further comment.
“This got as much, if not more, review and editing than a standard regular track manuscript,” Dr. Rubin, the editor in chief of the N.E.J.M., said of the heart study, which was based on a smaller set of Surgisphere data, that appeared in the medical journal. “We didn’t cut corners. We just didn’t ask the right people.”
He said the journal should have tapped outside experts familiar with big hospital data sets to be involved in the peer review. And The Lancet, Dr. Horton said, will demand independent verification of the quality of a database when reviewing studies from now on.
But, both editors said, peer reviewers cannot be expected to detect outright fabrication. Reviewers do not examine the raw data underlying the studies they review, except in exceedingly rare cases. That would be too laborious, and reviewers are not paid for their time.
But editors and reviewers do know who the study authors are, critics note, and that may bias them toward a positive review even when data are suspect. Dr. Mehra is well respected in scientific circles.
“We do tend to trust our authors,” Dr. Rubin said. “All journals do.” Both editors pointed out that Dr. Mehra had signed statements indicating he had access to all of the data and took responsibility for the work, as did other co-authors.
The politicization of the pandemic also may have played a role in The Lancet’s publication, critics charge. Mr. Trump has vigorously endorsed hydroxychloroquine as both preventive and curative treatment for Covid-19. The study’s conclusions at first appeared to rebuke the president.
[Like the Science Times page on Facebook. | Sign up for the Science Times newsletter.]
Dr. Horton is no fan of Mr. Trump, calling his decision to withhold funding from the W.H.O. in April “a crime against humanity.”
“Every scientist, every health worker, every citizen must resist and rebel against this appalling betrayal of global solidarity,” Dr. Horton wrote on Twitter.
On Friday, he said he chose to publish the hydroxychloroquine study only because it showed an immediate danger in widespread use of the drug. The clinical trials should not have been halted, he added.
“Because of the political context, and people using this drug on the basis of minimal evidence in its favor, it seemed very important to publish work that at least gave some sense of whether the drug was safe or not,” he said. “That was the motivation behind the publication.”
Journal editors are caught in a Catch-22 of sorts, said Dr. Hassan Murad, of the Mayo Clinic, who works with a federal project to review medical evidence. “You want to push the information out quickly to practitioners,” he said. “It’s a pandemic, it’s an urgent situation.”
“At the same time, you want quality control.”
