Risk Adjustment Methods Management Information & Analysis Department

1. Risk Adjustment MethodsManagement Information & Analysis Department June 8, 2007 Patricia Kipnis, Ph.D.

2. Outline • Why Risk Adjust? • Describing Health Care Utilization • Building Risk Adjustment Models • Using the Model • Future of Risk Adjustment

3. Why Risk Adjust? • Health Outcomes Assessment: Measuring and monitoring the outcomes and utilization of medical care is valuable in managing health care delivery. • Meaningful assessment of outcomes and utilization requires: • A measure of the outcome or utilization of interest • A way to risk adjust for patients’ risks for various outcomes • Risk adjustment methods allow us to compare outcomes and utilization of medical care for groups of patients with different risk factors

4. Why Risk Adjust? • Risk adjustment attempts to account for all factors, other than the health care intervention itself or the process of care that could affect outcomes or utilization of services1, for example health of patient before a given intervention. • Risk adjustment methods require the development of statistical models that explain the outcome variables of interest based on patient characteristics we wish to control. 1 Iezzoni, L. Risk Adjustment for Measuring Health Care Outcomes. Health Administration Press, Chicago, IL, 1997.

5. Describing Health Care Utilization • Health care utilization (or outcomes) can be described as: U = ƒ (patient risk factors, practice patterns, random error) • Risk adjustment models estimate the expected utilization as EU = ƒ (patient risk factors) • The difference between U and EU is explained by variation in unobserved patient factors, practice patterns and random error

6. Describing Health Care Utilization • Utilization statistics can then be adjusted for differences in patient risk factors as or where

7. Building Models • Before building a model that may be used for risk adjustment purposes a number of questions must be answered: • Risk of what? • Over what time frame? • What population? • What is the purpose of the model? • What are the risk factors? • What are the data sources? • What tools are available to build the models?

8. Risk of What? • Building models is not a trivial process, maintaining them can be even more daunting. • Health care analysts would prefer to have universal risk scores that denote the illness burden of each patient for all analyses. • However, this is not generally possible. Risk scores that are based on a model that predicts mortality may be very different from risk scores from a model that predicts total cost. • e.g. a terminally ill patient may have a low cost-based score but a high mortality-based cost.

9. Over What Time Frame? • Concurrent models use data from a particular year to predict the outcome for that same year • Predictions are based on episodic and chronic conditions • Used for profiling and outcomes analyses • Prospective models use data from a particular year to predict outcomes the following year. • Predictions are based on chronic conditions. • Used for payment, budgeting, risk stratification, pricing, budgeting.

10. For What Population? • There are many dimensions that help delineate populations with different risk factors for various health outcomes: • Age • Very young babies and very old patients pose a higher health risk • Gender • Models based on a general population model might not be very good predictors of OB/GYN utilization. • Hospitalized patients or patients in the ICU • Persons with disabilities or individuals in long term care setting • Specific diseases: Risk of AMI, CHF or pneumonia mortality

11. For What Purpose? Possible Uses of Model: • Contrasting outcomes for groups of patients • Setting payment levels for individual patients (e.g. DRG codes or capitation payments) • Comparing efficiency of care across providers or health plans • Identifying high risk patients

12. Risk Factors • Risk factors determine the medical meaningfulness of a risk adjustment model. • Medical meaningfulness is crucial since the goal of risk adjustment is frequently to affect physician behavior or gain their cooperation. • We must identify those factors that need to be present and evaluate the impact of factors not considered in the model. • Common risk factors include: • Demographic characteristics: Age, gender, race and ethnicity • Clinical Factors: Acute physiology, Principal Dx, Severity of Principal Dx, Comorbidities, Mental Health. • Socioeconomic factors: Education, Economic Resources, Employment and Occupation, Health Insurance Coverage, Cultural Beliefs. • Health-Related Behaviors: Alcohol, Drug and Tobacco Use, Diet. • Attitudes and Perceptions: Overall Health, Health Care Preferences.

13. Risk Factors • Obtaining a quantifiable measure of these risk factors is a non-trivial task. • Physiology: • Ideally, measured through vital signs but these are rarely available electronically. Lab tests results are generally used. • Illness and Severity of Illness: • Principal or Secondary Dx: • Diagnoses are recorded in the form of diagnosis codes. There are thousands of them. • Generally these are grouped into mutually exclusive categories. • Medications: • Medication utilization has also been used to capture severity of illness. • Drugs need to be grouped into categories. • Drug prescription patterns are continuously changing causing the model to be quickly out of date.

14. Risk Factors • Socio-economic factors, Health Related Behaviors and Attitudes. • Available through survey or as a one time measure. • Not collected on an ongoing basis. • Prior Utilization • Prior utilization might be a poor choice for risk adjustment purposes as it may introduce perverse incentives to increase utilization.

15. Data Sources • Easily accessible data sources are clearly preferred. • Many risk adjustment models use administrative or claims data to build risk adjustment models rather than survey data. • Vendors use large administrative data bases (e.g. Medicare or Medstat) to build models. • They sell a black box that takes administrative data used to define risk factors and the black box outputs a risk score. • These risk scores are calibrated to a particular output (generally total cost or probability of hospitalization)

16. Statistical Tools • Having defined the outcome, time frame, population, purpose of model and data sources we are ready to build a model that explains the outcome as a function of the risk factors. • This is the time to pull out the statistical tool box and model away! • Two very helpful references: • Hastie, Tibshirani and Friedman. The Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer, 2001. • Harrell, F. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer, 2001.

17. Developing Predictive Models • Assemble as much accurate data as possible with wide distributions for predictor values. • Formulate good hypotheses that lead to specification of relevant candidate predictors and possible interactions. • Select methods for dealing with missing values in both the dependent and independent variables. • Consider reducing the number of predictors if necessary. Useful rule is number of predictors < m/10 where m is limiting sample size. • Limiting sample size = N for continuous variable and min(m1, m2) for binary. • Consider possible transformations of outcome and predictors such as: • Log, Square Root, Polynomial • splines

18. Developing Predictive Models • Check additivity assumptions by testing pre-specified interactions. • Check distributional assumptions and overly influential observations. • May conduct stepwise, forward or backwards elimination for variable selection. Ideally, this step would then be part of the model validation process. • Examine collinearity: • Collinearity does not affect predictions if the new data set has the same degree of collinearity as the development data set.

19. Developing Predictive Models • Common Health Care data afflictions: • Utilization data are generally skewed: • Cost of care, Length of hospital stay • May use a transformation (e.g. log) or may truncate. • Large number of observations have a value of zero: • Patient Days, Number of office visits • May fit a two part model or use other GLM approaches

20. Model Validation • Commonly used methods are: • Split validation • Cross-validation • Bootstrap • Evaluate the model’s • Calibration: • Plot observed values vs predicted values (in deciles?) • Hosmer-Lemeshow • Discrimination ability: • R2 • 2 • C-statistic (area under the curve)

21. Applications: Risk Scores • The model can be used to define Risk Scores: Risk Score = Predicted Utilization Average Utilization • Risk scores are used to measure burden of illness at the patient level, the provider level, or Medical Center level. • Score > 1: Higher morbidity than the average population • Score < 1: Lower morbidity than the average population

22. Risk Adjusted Utilization/Cost The risk adjusted cost is defined as: Actual Cost / Risk Score = (Actual/Predicted)*Average For example: Predicted Actual Adjusted Cost Cost Score Cost Medical Center A $100 $100 0.5 $200 Medical Center B $400 $200 2.0 $100 Health Plan $200 $200 1.0 $200

23. Results Risk Adjusted Panel Size The risk adjusted panel size = Actual panel size x Risk Score For example: Actual Adjusted Member Panel Score Panel Status Facility A 10,000 1.1 11,000 Older/sicker Facility B 10,000 0.9 9,900 Younger/healthier

24. Inference • To draw inference on differences across Medical Centers we can estimate the standard error of the score by Rk: Observed to Expected Ratio Medical Center k nk: Number of patients at Medical Center k Ek: Mean expected utilization for Center k Oki: Observed utilization for patient i at Center k Eki: Expected utilization for patient i at Center k

25. Other Methods • Medical Center effects can be estimated as differences rather than the more common ratios. • Medical Centers can be entered as fixed effects into the model • The effect of Medical Center k is the odds ratio or the difference from average. • Alternatively, Medical Centers can be entered as random effects into the model • The effect of Medical Center k is then called the shrinkage estimate (Medical Centers effects with smaller sample size shrink towards the average). • These can be estimated using PROC GLIMMIX, PROC MIXED in SAS, Winbugs or MLWin. • Mixed effects models allow for the possibility of testing facility level effects.

26. Risk Adjustment Future • There is an urgent need to control health care costs. • There is increase pressure to report and compare cost of care across health care providers in addition to the quality of care measures already being reported. • Cost comparisons across health care providers are irrelevant unless these are risk adjusted for differences in the health status of the populations they serve. • Risk adjustment methods and health care predictive models will continue to play an important role in the assessment of health care services.