Elsevier

Academic Radiology

Volume 10, Issue 1, January 2003, Pages 25-31
Academic Radiology

Validation of the Summary ROC for Diagnostic Test Meta-Analysis: A Monte Carlo Simulation

https://doi.org/10.1016/S1076-6332(03)80784-5Get rights and content

Abstract

Rationale and Objectives

The author performed this study to test a technique for validating the logit regression method for summary receiver operating characteristic (ROC) meta-analysis, perform initial validation studies, and identify areas for further investigation.

Materials and Methods

Monte Carlo simulation was performed by using a custom macro program for a personal computer spreadsheet. The program creates simulated data sets based on user-specified parameters, performs a meta-analysis on the data sets, and logs the results so the accuracy and variability of the method can be measured. The program can also be used to measure the effects of changes in study design and meta-analysis parameters.

Results

For the base case of a small meta-analysis (10 studies) of small trials (mean, 50 patients), the meta-analysis results closely matched the input sensitivity and specificity when they were less than 80%. Systematic errors, if any, were small. At sensitivities and specificities greater than 80%, the true sensitivity or specificity was underestimated by up to 2% in the meta-analysis. Confidence intervals calculated with the summary ROC curve were reasonably conservative, although they too fell below the true results when sensitivity or specificity was greater than 80%. The underestimation was eliminated when the simulation was repeated for a much larger trial (mean, 1,000 cases per study)\Meven with a sensitivity and specificity of 98%.

Conclusion

The Littenberg-Moses method for summary ROC meta-analysis is effective for obtaining an accurate summary estimate of diagnostic test performance, although the continuity correction introduces a small downward bias in the meta-analysis of small trials.

Section snippets

Materials and Methods

To examine the performance of the logit regression method under a variety of conditions, simulated data sets of known characteristics were created and meta-analyzed. A program was written in the macro language of Excel (Office 2000; Microsoft, Redmond, Wash) to create the simulated data sets. An Excel spreadsheet was programmed to perform the meta-analyses and record results. The spreadsheet was validated by recalculating previously published meta-analyses results and comparing its results with

Results and Discussion

In this initial validation study, the base scenario was a meta-analysis of 10 trials, with a mean sample size per trial of 50 patients (SD, 20 patients). This is consistent with the typical evidence base for emerging diagnostic imaging applications assessed at my institution. The effects a larger study size and larger number of trials have on the meta-analysis will be examined in detail in future experiments. Future experiments will also examine the effect of random variability from trial to

Conclusion

The logit regression method for summary ROC meta-analysis of diagnostic clinical trial results enables accurate estimation of the sensitivity and specificity of the test being studied. Although the method systematically underestimates the sensitivity and specificity for very high levels of test performance, the underestimation is no more than 2% for each parameter. An increase in the size of the clinical trials being meta-analyzed virtually eliminates this bias, which indicates that it is a

References (6)

There are more references available in the full text version of this article.

Cited by (25)

  • Assessment of 3DTV-related fatigue with resting-state fMRI

    2018, Signal Processing: Image Communication
  • Diagnostic accuracy of osteopontin plus alpha-fetoprotein in the hepatocellular carcinoma: A meta-analysis

    2017, Clinics and Research in Hepatology and Gastroenterology
    Citation Excerpt :

    The overall diagnostic sensitivity, specificity, the pooled PLR and NLR were 0.71 (95% CI: 0.69–0.74), 0.80 (95% CI: 0.78–0.82), 3.55 (95% CI: 2.58–4.90) and 0.26 (95% CI: 0.17–0.40) for OPN; 0.61 (95% CI: 0.58–0.63), 0.92 (95% CI: 0.91–0.94), 7.57 (95% CI: 5.00–11.46) and 0.41 (95% CI: 0.33–0.51) for AFP; 0.82 (95% CI: 0.79–0.84), 0.77 (95% CI: 0.74–0.80), 3.68 (95% CI: 2.45–5.53) and 0.19 (95% CI: 0.11–0.32) for OPN + AFP, respectively (Figs. 2–4). In addition, the summary receiver operating characteristics (SROC) curve was employed to obtain an accurate summary estimate of diagnostic test performance [39]. The area under the curve (AUC) was used for summarizing SROC.

  • Improvement of the lipid profile with exercise in obese children: A systematic review

    2012, Preventive Medicine
    Citation Excerpt :

    Pooled data were calculated for the studies grouped into two categories: aerobic programs and combined programs. If the study did not provide the data for the mean, standard deviation, standard error, or standard error of the mean, then the mean and standard deviation were computed by Monte Carlo simulation (Mitchell, 2003), a technique which uses mathematical modeling to imitate the random behavior of real non-dynamic systems. The simulation allows the researcher to specify many parameters that affect the simulated data sets and explore the effects of those parameters on the accuracy of the results.

  • Meta-analysis of B-type natriuretic peptide's ability to identify stress induced myocardial ischemia

    2011, American Journal of Cardiology
    Citation Excerpt :

    Although random and fixed effect models were used, owing to significant heterogeneity we reported only results from random effect models. A summary receiver operating characteristic curve9 was constructed from pooled data for studies included in the final analysis. Subgroup analyses were performed to explore the significant heterogeneity observed.

  • The diagnostic accuracy of CT and MRI in the staging of pelvic lymph nodes in patients with prostate cancer: a meta-analysis

    2008, Clinical Radiology
    Citation Excerpt :

    To address this problem, SROC analysis was used in this meta-analysis. SROC analysis corrects for variation due to differences in test thresholds in the original studies.9–11,14,40 The present study had the following limitations: first, some of the studies included in the meta-analysis were subsets from a larger study as only a limited number of patients fulfilled the inclusion criteria.

View all citing articles on Scopus
View full text