The primary goal of evidence-based medicine is to guide the management of patients. The controversy surrounding the diabetes medication rosiglitazone has challenged the medical community to resolve discrepancies between different approaches used to synthesize evidence. Although meta-analysis to evaluate the cardiovascular safety of a drug using data from trials testing for a noncardiovascular outcome is a helpful exercise, the rosiglitazone controversy has made us keenly aware of the limitations.

At the time of going to press, five rosiglitazone meta-analyses conducted by seasoned investigators have been published, including the first report by Nissen and Wolski. The common theme emerging from these reports is that rosiglitazone does not seem to have cardioprotective effects. Most experts believe that the estimates of risk for cardiovascular events in the original Nissen and Wolski 42-trial meta-analysis are exaggerated.

The strongest rebuttal of the Nissen findings came from a meta-analysis by Diamond and colleagues. They pooled the same trials as Nissen but came up with contrasting results depending on the method of meta-analysis used, and whether or not trials without cardiovascular-related events were included. Although applicable when events are rare, the statistical method used by Nissen is controversial when the pooled trials are heterogeneous with regards to patient eligibility and design. The 42-trial meta-analysis by the FDA that used patient-level data and the systematic review by the Cochrane Collaboration could not confirm a statistically significant difference in odds ratios for rosiglitazone versus controls with regards to major cardiovascular events. Singh et al.'s meta-analysis that incorporated only four of the larger trials by limiting eligibility to trials of at least 12 months duration showed a significant increase in myocardial infarction risk but no significant increase in cardiovascular mortality.

If meta-analyses are used to address regulatory and clinical questions they should be conducted with rigor and interpreted with caution...

To cultivate a balanced approach to understanding results generated by meta-analysis of data from small trials it is important to accept the limitations implicit in this method. The rosiglitazone trials pooled were heterogeneous with regards to eligibility criteria and outcome measures and were not designed to individually address relatively uncommon events. As cardiovascular-related events were not the primary outcome in any of the trials, any pooling project should be interpreted as exploratory at best.

Another major limitation of meta-analysis of small trials is the interpretation of the P value. Following post-hoc analysis, values hovering around 0.05 should be regarded with extreme caution. Some experts believe that values of 0.01 or lower should be adopted. For myocardial infarction excess with rosiglitazone, the P values from the four meta-analyses ranged from 0.02 to 0.06 and higher. It is widely accepted that the reliability of the findings of a meta-analysis is linked to the number of overall events accrued.

So was the recent FDA conclusion premature—that rosiglitazone is associated with a greater risk of myocardial ischemic events when compared with placebo, metformin or sulfonylurea? In the absence of conclusive evidence, we recommend that health-care professionals exercise great caution when interpreting meta-analyses of small trials and, in the case of rosiglitazone, carefully balance the associated risks and benefits. We believe that the only definitive approach to quantifying this risk lies with the RECORD trial, prospectively designed to address the cardiovascular safety issue.

While meta-analyses generate hypotheses related to the direction of treatment effects, large well-designed, sufficiently powered, randomized controlled trials are the mainstay of evidence-based medicine. If meta-analyses are used to address regulatory and clinical questions they should be conducted with rigor and interpreted with caution, particularly in terms of the magnitude of the effect size. Let us hope that these meta-analyses of small trials are the beginning of a fruitful discussion—further debate should proceed with caution.