Original Article
Simulation-based power calculation for designing interrupted time series analyses of health policy interventions

https://doi.org/10.1016/j.jclinepi.2011.02.007Get rights and content

Abstract

Objective

Interrupted time series is a strong quasi-experimental research design to evaluate the impacts of health policy interventions. Using simulation methods, we estimated the power requirements for interrupted time series studies under various scenarios.

Study Design and Setting

Simulations were conducted to estimate the power of segmented autoregressive (AR) error models when autocorrelation ranged from −0.9 to 0.9 and effect size was 0.5, 1.0, and 2.0, investigating balanced and unbalanced numbers of time periods before and after an intervention. Simple scenarios of autoregressive conditional heteroskedasticity (ARCH) models were also explored.

Results

For AR models, power increased when sample size or effect size increased, and tended to decrease when autocorrelation increased. Compared with a balanced number of study periods before and after an intervention, designs with unbalanced numbers of periods had less power, although that was not the case for ARCH models.

Conclusion

The power to detect effect size 1.0 appeared to be reasonable for many practical applications with a moderate or large number of time points in the study equally divided around the intervention. Investigators should be cautious when the expected effect size is small or the number of time points is small. We recommend conducting various simulations before investigation.

Introduction

What is new?

  • Interrupted time series designs are increasingly used in health policy research, but there are no accepted methods for performing power calculations.

  • We conducted simulations to estimate the power for segmented autoregressive models with effect sizes, sample sizes, and autocorrelations in the ranges found in previous studies.

  • Power increased when the number of data points periods increased, decreased when autocorrelation increased, and increased as the expected effect size increased.

  • Researchers can use such simulation results to design statistically sound interrupted time series studies.

  • Investigators should be cautious about the statistical adequacy of a study when the expected effect size is small (<0.5) or the number of time points is small (<24).

Quasi-experimental interrupted time series (with comparison series) designs are the most appropriate to study the effects of health policy interventions because randomized controlled intervention trials of policy interventions are rarely feasible [1]. Interrupted time series studies allow health systems researchers to estimate whether and how much an intervention changed an outcome of interest, given the level and trend of the outcome before the intervention; whether effects occurred immediately after the intervention or with delay; whether effects were transient or long term; and whether other factors than the intervention could explain observed changes [2]. Such designs have been widely used in assessing the effects of health services and policy interventions, such as payment restrictions or prior authorization for medications in state programs [3], [4], state law to establish a minimum delivery hospital stay [5], or educational and administrative interventions to improve prescribing [6], [7].

Analysis of time series data requires the use of segmented regression analysis methods with autoregressive (AR) error models that account for correlations of the data across time points. To design interrupted time series studies, one must know how many time points and how many observations to sample at each time point to obtain stable estimates of intervention effects. To date, most methods for time series studies have focused on economic applications, where forecasting is the major goal. In health policy analysis, hypothesis testing concerning intervention effects is the primary goal. To plan efficient policy evaluations using time series designs, investigators need methods to estimate power and sample size. McLeod and Vingilis [8] proposed a method to estimate power and sample size for a Simple Intervention Analysis (SIA) Model, with only one change (level or trend). Their method assumes a fractional AR integrated moving average process (ARIMA), an extension of the well-known ARIMA. ARIMA models usually require more than 50 time points, more than are usually available in health policy evaluation studies. Furthermore, because the SIA Model is based on a Wald-type statistic, it cannot be easily generalized to test simultaneous level and trend changes or control for possible covariates. In policy analyses, researchers are interested in level and trend changes of the outcome, and frequently need to account for phase-in periods during policy implementation, often by excluding a number of time points between the prepolicy period and the start of the postpolicy evaluation period. It is difficult to accommodate such sample size loss in a nonsimulation-based (Wald-type statistic) approach. In addition, the SIA Model will not allow one to estimate power and sample size when error variability changes over time. AR conditional heteroskedasticity (ARCH) models are needed in this case. In summary, a simulation-based approach to power and sample size estimation is a good supplement when traditional Wald-type methods apply and a necessary tool when they do not.

Section snippets

Methods

We developed a simulation-based method to estimate power and sample size for time series studies of policy interventions, focusing on AR error models. To develop a basis for these simulations, we examined the ranges and variability of effect sizes from our previous time series studies of pharmaceutical policy effects [4], [5], [9], [10], [11], [12], [13]. We focused on the effect size (expected intervention effect over its standard deviation) rather than parameter effect and variance as in the

Single parameter (AR)

Table 1, Table 2, Table 3 present the estimated power of segmented time series regression models to detect a single parameter (either a level change or a trend change) for effect sizes 0.5, 1.0, and 2.0 with equal numbers of pre- and postintervention time periods, statistical significance level 0.05, and an AR error model with lag 1. As anticipated, power increased when the sample size increased, holding other parameters constant. In addition, power tended to decrease when the autocorrelation

Discussion

Interrupted time series is the strongest quasi-experimental research design to evaluate the impacts of health policy interventions in situations where randomization is not feasible. Using simulations, we provide estimates of the power for various scenarios of first-order AR models in practice to illustrate which parameters are important determinants of power. As anticipated, power increased when the number of time points included in the time series analysis increased, tended to decrease when

Conclusion

Segmented time series regression models are increasingly used to estimate the impact of health policy interventions. On the basis of the reported simulation results, these models have more than 80% power to detect effect sizes of 1.0 or greater in a range of situations with 24 or more time points, depending on the degree of autocorrelation and whether a level change, trend change, or both are estimated. In situations where it is known a priori that the underlying standard deviation of the data

References (18)

  • T.D. Cook et al.

    Quasi-experimentation: design and analysis issues for field settings

    (1979)
  • A.K. Wagner et al.

    Segmented regression analysis of interrupted time series studies in medication use research

    J Clin Pharm Ther

    (2002)
  • S.B. Soumerai et al.

    Use of atypical antipsychotic drugs for schizophrenia in Maine Medicaid following a policy change

    Health Aff (Millwood)

    (2008)
  • S.B. Soumerai et al.

    Payment restrictions for prescription drugs under Medicaid. Effects on therapy, cost, and equity

    N Engl J Med

    (1987)
  • J.M. Madden et al.

    Effects of a law against early postpartum discharge on newborn follow-up, adverse events, and HMO expenditures

    N Engl J Med

    (2002)
  • A.C. Feldstein et al.

    Reducing warfarin medication interactions: an interrupted time series evaluation

    Arch Intern Med

    (2006)
  • D.H. Smith et al.

    The impact of prescribing safety alerts for elderly persons in an electronic medical record: an interrupted time series evaluation

    Arch Intern Med

    (2006)
  • A.I. McLeod et al.

    Power computations for intervention analysis

    Technometrics

    (2005)
  • K.Z. Bambauer et al.

    Physician alerts to increase antidepressant adherence: fax or fiction?

    Arch Intern Med

    (2006)
There are more references available in the full text version of this article.

Cited by (183)

  • Postmenopausal bleeding after COVID-19 vaccination

    2024, American Journal of Obstetrics and Gynecology
View all citing articles on Scopus
View full text