PREDIMED trial of Mediterranean diet: retracted, republished, still trusted?

Retraction and republication

Recent developments questioned PREDIMED at its core. An analysis of reported baseline characteristics in 5087 trials by Carlisle identified trials in which the compared randomised groups were too similar or too dissimilar, raising questions of potential fraud or non-random sampling.10 PREDIMED stood out for implausible P value patterns when comparing the baseline characteristics of the three arms. An audit of the trial found serious irregularities: enrolment of household members without randomisation; assignment of participants to study arms based on clinic site rather than true randomisation; and inconsistent use of randomisation tables. These deviations affected 1588 participants (21% of the total).11

The randomised trial was no longer a randomised trial. The original paper was retracted and replaced with a reanalysis that treated PREDIMED as a non-randomised study and excluded participants who were not truly randomised.11 The reanalyses gave similar point estimates for the primary endpoint.

Is republication justified?

Whether republication is justified in such cases is controversial. Retracting a paper and replacing it with a republication is an uncommon choice. In theory, it can be used when there is an error that significantly affects parts of the study but does not completely refute it. Therefore, the validity of this course of action depends on the nature of the errors and on whether there is really a way to fully correct them.12

In PREDIMED, the detected irregularities may not entirely explain the peculiar baseline characteristics and they also raise questions about the quality of other aspects of the conduct of the trial, such as data collection, data arbitration, and adjudication. Participants, investigators, and assessors were not blinded, and the unmasked design can further compound any bias—for example, if investigators and sponsors favour specific interventions or when assessors are collecting outcome data. Here, it is unclear if the correction of the specific identified errors also corrected all the potential accompanying problems and consequences of these errors.

Additionally, the title of the republication does not make it clear that it is a reanalysis and republication, and many readers may be confused. Showing clearly that the new paper is an amendment and numbering versions would help.13

Continuing follow-up

If randomisation problems had been detected while the study was ongoing, would it have been stopped early because of perceived benefit? The reanalysis excluding improperly randomised patients does not satisfy the P value boundary required for early stopping for each intervention arm. PREDIMED has in fact continued follow-up, and the investigators have published papers with 1.2 additional years of follow-up, during which the number of participants experiencing an event included in the composite primary endpoint increased by 19%. However, information is not provided on the comparison of the three arms for the primary endpoint with this extended follow-up. The full follow-up data should be disclosed and analysed for an intention-to-treat comparison to determine whether there is still benefit.

Full follow-up would also allow more complete assessment of mortality differences. Mortality is linked to a national registry and thus cannot be biased from subjective interpretation by the local investigators who collated data on clinical events before sending them to an adjudication committee. In the original follow-up, the control arm had 114 deaths, versus 116 and 118 in the two experimental arms, respectively.1 More deaths have occurred during the additional 1.2 years’ follow-up. Analysis of the full mortality data would provide the most convincing evidence from this trial and is essential to understand the robustness of its conclusion.

We have asked the corresponding authors of PREDIMED papers and (on their recommendation) the head of the steering committee to provide the number of primary outcome events and deaths in each arm in the extended follow-up. PREDIMED investigators responded that “information you have asked for is the main topic of ongoing analyses on PREDIMED data, as already approved by the steering committee.” We asked them again whether they could share minimal information for this article but have not heard back from them over the past six months.

What about secondary publications?

Importantly, the original PREDIMED paper had already generated 267 secondary publications before its retraction and republication.14 Thirty two of them have already received over 100 citations each in Google Scholar. Most of the publications come from the network of investigators who performed the original trial and their extended teams, with three investigators having each published over 150 articles from PREDIMED. In July 2018 we identified 203 secondary papers with data (excluding reviews, editorials, and commentaries); 194/203 (95%) first authors, 201/203 (99%) last authors, and 223/225 (99%) corresponding authors (some papers have more than one corresponding author) belong to the original PREDIMED investigator team in the 2013 paper or are affiliated with related Spanish institutions (see supplementary data on bmj.com).

The analyses presented in secondary papers use the data that led to retraction of the original. These publications should probably have notices of concern (as has been done for a secondary analysis published in CMAJ15) until they are properly re-evaluated. The PREDIMED authors have started correcting some of their work and have published several letters to this effect (at least five letters pertaining to eight secondary publications1617181920).

Re-evaluation should be truly independent. Given the circumstances, it cannot be done only by the PREDIMED investigators and other sympathetic investigators who hold similar views on the importance of specific foods to modulate disease risk and on the agreement between the results of non-randomised studies (what PREDIMED in perceived to be now) and randomised trials. Involvement of investigators with contrary views is pivotal for this reassessment to be fair and balanced. Even the best intentioned and most knowledgeable investigators in nutrition may still favour their beliefs and thus inadvertently introduce some bias in the reanalyses.21

Instead of trying to correct one paper at a time, it may be more efficient for an independent team to make a centralised effort and report its findings to all relevant journals. Audits of bodies of contested literature are not uncommon. Since the work usually pertains to single investigators they are normally done by university appointed committees. For a major multi-investigator effort like PREDIMED, the independent assessors should have international provenance.

Secondary publications that compare outcomes in the randomised arms are directly affected. However, even when secondary publications deal with the study dataset or subsets as an observational cohort, the clustering of recruited participants (eg, household members co-randomised or a whole village recruited in one step) still affects their results.22 The clustering effect also needs to be incorporated in observational analyses and may lead to different estimates and conclusions.

Inconsistencies in secondary publications

Even without in-depth re-evaluation of the raw data underlying PREDIMED publications, there are some inconsistencies in the reported data in these papers that suggest broader and more generalised problems that are not fixable by a single reanalysis.

To illustrate this point, we searched PubMed using the keyword “PREDIMED” and identified English language PREDIMED publications that included data on over 7000 participants reporting either the primary composite endpoint or mortality (all causes and cause specific). The number of events varies widely even when publications have similar follow-up (see supplementary table on bmj.com).

Some discrepancies may simply reflect missing data and different eligibility criteria in different analyses. However, table 1 shows a sample of contradictions in the reported data in paired papers that cannot be explained in this way and point to errors in one or both of the papers. The original publication1 reported 87 cardiovascular deaths for the full trial population and follow-up, while Henríquez-Sánchez and colleagues’ secondary analysis of dietary antioxidants and mortality24 reports 102 such deaths despite a more limited sample and follow-up. Two publications2324 have identical follow-up, but one23 has a larger number of total deaths, while the other24 has a higher number of cardiovascular deaths. Two other publications2526 have identical total number of deaths, but one has five more cardiovascular deaths (a component of the primary endpoint)26 while the other has five more deaths from other causes (not a component of the primary endpoint).25 The original publication reported only cardiovascular deaths and total deaths.1 However, if the 166 deaths from cancer reported by Henríquez-Sánchez and colleagues24 are added to the 117 deaths from non-cancer, non-cardiovascular causes reported by Martínez-González and colleagues25 and 87 deaths from cardiovascular causes in the original paper,1 the total deaths are 370, exceeding the total deaths (n=348) reported in the original publication despite identical length of follow-up.1

These discrepancies may point to poor reporting, erroneous or inconsistent statistical analyses, or deeper problems related to problematic data collection and curation. They are superimposed on a PREDIMED literature that shows all the hallmarks of data dredging, given the huge number of secondary publications. Most analyses are not prespecified or are specified imprecisely. In fact, it is unclear which (if any) secondary analyses were clearly and unambiguously prespecified. For example, one secondary publication states that invasive breast cancer was a prespecified outcome,27 but the published trial protocol states that all cancer—not just invasive breast cancer—was a secondary outcome. Data are reported on one cancer subtype (claiming huge benefits) but not on dozens of others.

Publicly available data and reanalysis needed

PREDIMED investigators have stated that the data that went into the recent republication can be requested by interested parties, but the concerns identified make a strong case for the complete PREDIMED dataset, not just the data that went into the recent republication, to become publicly available. A truly independent audit should examine the original data records, adjudication, and statistical analyses that underlie this voluminous published literature. Updated follow-up results should be independently assessed and reported, including all-cause mortality data (the most objective outcome). Box 1 summarises some proposed actions. Some may wonder whether it is worth investing so much effort auditing a single study when there is a vast literature on nutrition with potential errors and biases. However, PREDIMED is an iconic example of using randomisation in the field (despite all the caveats discussed above) and it has major repercussions. An independent reanalysis effort may be more efficient and convincing than piecemeal attempts by the authors to defend each secondary publication separately.

Lessons for future pivotal multicentre trials

PREDIMED may offer useful lessons about how to run future large multicentre trials that aim to revolutionise an entire field. The original PREDIMED publication1 had 18 authors and 223 collaborators. Many secondary papers also feature impressive numbers of coauthors. However, studies with hundreds of investigators may still have blind spots where no one really is responsible or knowledgeable enough to avert major mistakes and protocol deviations (box 2).

PREDIMED investigators have a unique opportunity to disclose details on how the process failed, so that other trialists may avoid similar problems. Strong methodological expertise at all pivotal steps and function, both site specific and centralised, is essential. Also the inclusion of people who do not support the study hypothesis in the monitoring board is key to ensure balance and avoid bias.

PREDIMED offers useful lessons that can be applied to many other trials.28 The problems should not lead to a reduction in funding of diet related research to improve health. Conversely, the same or even higher funding should be diverted to well executed large trials in nutrition. This will require getting together people with different expertise and skill sets. We also have an opportunity to probe how large volumes of published literature can be reassessed and corrected, as appropriate, when serious problems are identified. Finally, we can ponder on what expertise and better safeguards are needed to run such important multicentre trials reliably in the future.