Time Series Designs to Evaluate Policy Change and Other Interventions: Analysis and Interpretation

2. Time Series Designs to Evaluate Policy Change and Other Interventions: Analysis and Interpretation Anita Wagner April 2, 2004

3. Outline • Challenges and opportunities in pharmaceutical policy and intervention research • Interrupted time series designs • Definitions • Data structure • Absolute and relative effect estimates • Minimizing biases and examples • Summary

4. Challenges and Opportunities Challenges: • No randomized controlled trials • Secular trends Opportunities: • Routinely collected data • Longitudinal data

5. Time series • Series of measurements of a single characteristic in different time intervals • Health facility utilization rates • Prescribing indicators • Prescription cost • Reasons for collection • Description of levels and trends • Prediction of future values

6. Segmented Time Series • Segments • Specific event causes a change in the series, dividing it into distinct segments • Estimating the change in the series allows you to assess the impact of the event • Validity • Strongest non-experimental research design • Pre-event level and trend serves as a built-in “control”

7. Segmented Time Series with Comparison Series • Two or more related series are compared • Populations exposed or not exposed to the event of interest • Other characteristic in the exposed population that is not expected to change • Potential substitute behaviors for the one affected by an event • Increases ability to interpret the impact of the event of interest

8. Impact of the NY TPP on Monthly Benzodiazepine Prescribing New Jersey New York Source: Ross-Degnan et al, under review

9. Hypothetical Changes in Level and Slope of a Time-Series Analysis of an intervention effect using segmented linear regression Utilization rate Intervention immediate level change projected change slope slope=0 Time Assumption: Extrapolating the pre-intervention level and trend correctly reflects the (counterfactual) outcome that would have occurred had the intervention not happened. Adapted from Schneeweiss et al, Health Policy 2001

10. intervention before after Possible Intervention Effects intervention before after intervention intervention before before after after

11. Segmented Regression Analysis • Segmented regression analysis • Used to model the series before and after the event to estimate its impact • Discontinuities can be in level or trend • Can include other time dependent covariates • Other methods • Box-Jenkins models (need >50 data points) • Proportional hazards regression models

12. Basic Data Structure

13. Impact of the NY TPP on Monthly Benzodiazepine Prescribing New Jersey (Outcome with policy – Outcome without policy)/ Outcome without policy (Outcome with policy – Outcome without policy) New York Source: Ross-Degnan et al, under review

14. Post-TPP Reductions in BZD Use Among 1988 Recipients by Group

15. Observed & Predicted Use of Essential Drugs With Change in Cost Sharing Source: Tamblyn R et al, JAMA 2001; 285: 421-429.

16. Minimizing Biases • by factors that are related to the outcome and change at the time of the intervention, e.g., • Co-interventions • Contemporaneous changes in population composition • through • Inclusion of a comparison group • Inclusion of time-varying covariates

17. Impact of the TPP on Use of BZD Indicated for Seizure and Panic Disorder Source: Simoni-Wastila et al, under review

18. Changes in Analgesic Prescribing after Market Entry and Withdrawal of Zomepirac Source: Ross-Degnan et al, JAMA 1993

19. Impact of TPP on Use of Substitute Drugs Among 1988 BZ Recipients New York New Jersey

20. Improving Antimicrobial Prophylaxis for C-Sections in Colombian Hospitals • Study periods • Baseline: MD decision on prophylaxis (Px) • Period I: Px protocol for high risk women • Period II: Px required for all women + OR pharmacy • Process and outcome measures • Women who receive prophylaxis • Prophylaxis episode with correct timing • Surgical site infections • All C-sections, but changing population • Volume • Case mix

21. Controlling for Changes in Population Characteristics Over Time

22. Case-mix Adjusted Prophylaxis and Infection Rates in C-Sections Baseline Period I Period II 20 90 18 80 16 Receive prophylaxis 70 14 Correct timing 60 12 # surgical site infections Infection rate 50 10 40 8 30 6 20 4 10 2 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Month Source: Weinberg et al, Arch Int Med, 2001

23. Complex Intervention to Reduce Use of Injections in Indonesia Interactive Group Discussions to Reduce Injections Pilot Self-Monitoring Evaluation of in 4 Health Centers Routine Injection Quantitative Self-Monitoring in Study Drug Use All 29 Health Focused Study Centers Qualitative Study in 4 Health Centers Aug Dec Apr Aug Dec Apr Aug 1992 1993 1994

24. Trends in Injection Use Comparing IGD and Control Groups Interactive Group Discussions Dissemination Seminar Self-monitoring Implementation

25. Advantages and Limitations • Advantages • Control for important threats to internal validity: • Level and trend before the policy change(s) • Any non-time varying factors • Intuitive visual display • Direct effect estimate • Limitations • Assume linear relationship within each segment • Aggregate analysis, without control for covariates at individual level

26. Summary Points • Time series data allow for strong quasi-experimental designs • Address most threats to validity • Visible effects almost always significant • Use of comparison series • Comparison population or behavior • Market share (within class or indication) • High risk subgroups • Unintended outcomes