Expanding Research and Evaluation Designs…for QII

1. Expanding Research and Evaluation Designs…for QII Carolyn M. Clancy, MD Director, AHRQ September 13, 2005

3. Are We Making Any Progress? • Second annual reports focus on quality of and disparities in health care in America • Quality Report finds that quality is improving and identifies areas which are in need of major improvements • Disparities Report indicates that there are pervasive disparities related to race, ethnicity, and socioeconomic status

4. Preliminary NHQR Highlights 2005 • Health Care Quality Continues To Improve at a Modest Pace Across Most Measures of Quality • 10:1 ratio of measures improved to declined • Overall improvement rate: 2.8% • Health Care Quality Improvement Is Variable, With Notable Areas of High Performance • Patient safety: 10.2% • QIO Measures: 9.2% • Effectiveness measures: 2.8% • High improvement subset: 5.4%

5. RAND Study: Quality of Health Care Often Not Optimal • Doctors provide appropriate health care only about half the time Alcohol dependence 11% 23% Hip fracture Peptic ulcer 33% Diabetes 45% Low back pain 69% Prenatal care 73% Breast cancer 76% Cataracts 79% Percentage of time E. McGlynn, S. Asch, J. Adams, et al., The Quality of Health Care Delivered to Adults in the United States, N Engl J Med, 2003

6. Public Perceptions Percent who say they are dissatisfied with the quality of health care in this country… Has the quality of health care in this country… 2004 2000* Gotten worse Stayed about the same Don’t Know Gotten better * Gallup Poll conducted September 11-13, 2000 with 1,008 U.S. adults. Source: Kaiser Family Foundation / Agency for Healthcare Research and Quality / Harvard School of Public Health National Survey on Consumers’ Experiences with Patient Safety and Quality Information, November 2004 (Conducted July 7 – September 5, 2005).

8. AHRQ’s Mission To improve the safety and quality, efficiency and effectiveness of health care for all Americans

9. Answers must be: • Timely • Valid • Convincing • Practical

10. Balas, 2002 Original research 18% variable Negative results Dickersin, 1987 Submission 46% 0.5 year Kumar, 1992 Koren, 1989 Acceptance Negative results 0.6 year Kumar, 1992 Publication 17:14 35% 0.3 year Poyer, 1982 Balas, 1995 Lack of numbers Bibliographicdatabases Expert opinion 50% 6. 0 - 13.0 years Antman, 1992 Poynard, 1985 Reviews, guidelines, textbook 9.3 years Inconsistent indexing Implementation It takes 17 years to turn 14 per cent of original research to the benefit of patient care

11. Treatments Thought to Work but Shown Ineffective • Sulphuric acid for scurvy • Leeches for almost anything • Insulin for schizophrenia • Vitamin K for myocardial infarction • HRT to prevent cardiovascular disease • Flecainide for ventricular tachycardia • Routine blood tests prior to surgery • ABMT for late stage Breast CA BMJFebruary 28 2004; 324:474-5.

12. Expanding Research and Evaluation Designs…for QII Carolyn M. Clancy, M.D. Director, AHRQ September 13, 2005 DRAFT – Denise D and David Atkins – 9/7/05

14. “Health services research has not yet been sufficiently helpful in meeting the challenge of improving care in part because it has over-constrained both its methods and its favorite topics….HSR should become more effectively part of the solution. To do that will require that we enrich our portfolio of methods and broaden our area of inquiry.” --Berwick, HSR, 40:2 (April 2005)

15. Challenges to the RCT Paradigm • The intervention targets are not patients directly, • Interventions are complex • Interventions change, and • Where the setting is an essential component of the question and intervention.   

16. AHRQ Research Study: Computerized Physician Order Entry Systems • Major Finding: While computerized physician order entry (CPOE) is expected to significantly reduce medication errors, systems must be implemented thoughtfully to avoid facilitating certain types of errors • Study looked at clinicians’ experience in using one CPOE system at a major urban teaching hospital • Implementation problems can be minimized through testing before products are marketed and through adaptation to meet the needs of individual clinical settings R. Koppel, J. Metlay, A. Cohen, et al., Role of computerized physician order entry systems in facilitating medication errors,Journal of the American Medical Association, March 9, 2005

17. Current QI evaluation questions AHRQ and others are asking • Can a regional health information organization improve interoperability of health information technology systems and improve patient safety and quality of care? • Can pay for performance improve quality? • Do changes in hospital culture reduce medical errors? • What QI strategies work for reducing disparities? • Different strategies for different populations and settings? • Same strategies?

18. Current QI evaluation questions AHRQ and others are asking - 2 • When should coverage be linked with development of better evidence (‘decision-based evidence making’)? • [In an action-oriented world how to identify a control group?] • Can evidence for the effectiveness of team training in obstetrics be developed as training is provided? • For which questions is a registry useful? What methods will help us take advantage of new sources of electronic data?

19. Galaxies of Evaluation Designs That May Inform QII Evaluation Designs and Methods Behavioral/Social Science Galaxy Medical/clinical/ HSR Galaxy RCT Action Research— K Lewin GRT ITS/MBS Quasi-exp QII Evaluation Research HSR: 2° data Analysis RD Case reports Systems Science RCT Cohort study Case- Control Qualita- tive methods Case studies PDSA/SPC Deming/ Shewhart [1] For purposes of this chart, intervention evaluation research is defined as studies designed to Answer the Questions: Does the Intervention Work? For Whom or What? Under What Conditions?

21. Questions? Contact CCLANCY@AHRQ.GOV