What You See May Not Be What You Get: A Primer on Regression Artifacts

1. What You See May Not Be What You Get: A Primer on Regression Artifacts Michael A. Babyak, PhD Duke University Medical Center

3. Topics to Cover • Models: what and why? • Preliminaries—requirements for a good model • Dichotomizing a graded or continuous variable is dumb • Using degrees of freedom wisely • Covariate selection • Transformations and smoothing techniques for non-linear effects • Resampling as a superior method of model validation

4. What is a model ? Y = f(x1, x2, x3…xn) Y = a + b1x1 + b2x2…bnxn Y = e a + b1x1 + b2x2…bnxn

5. Why Model? (instead of test) • Can capture theoretical/predictive system • Estimates of population parameters • Allows prediction as well as hypothesis testing • More information for replication

6. Preliminaries • Correct model • Measure well and don’t throw information away • Adequate Sample Size

7. Correct Model • Gaussian: General Linear Model • Multiple linear regression • Binary (or ordinal): Generalized Linear Model • Logistic Regression • Proportional Odds/Ordinal Logistic • Time to event: • Cox Regression • Distribution of predictors generally not important

8. Measure well and don’t throw information away • Reliable, interpretable • Use all the information about the variables of interest • Don’t create “clinical cutpoints” before modeling • Model with ALL the data first, then use prediction to make decisions about cutpoints

9. Dichotomizing for Convenience Can Destroy a Model

10. Implausible measurement assumption “not depressed” “depressed” A B C Depression score

11. Dichotomization, by definition, reduces power by a minimum of about 30% http://psych.colorado.edu/~mcclella/MedianSplit/

12. Dichotomization, by definition, reduces power by a minimum of about 30% Dear Project Officer, In order to facilitate analysis and interpretation, we have decided to throw away about 30% of our data. Even though this will waste about 3 or 4 hundred thousand dollars worth of subject recruitment and testing money, we are confident that you will understand. Sincerely, Dick O. Tomi, PhD Prof. Richard Obediah Tomi, PhD

13. Examples from the WCGS Study:Correlations with CHD Mortality (n = 750)

14. Dichotomizing does not reduce measurement error Gustafson, P. and Le, N.D. (2001). A comparison of continuous and discrete measurement error: is it wise to dichotomize imprecise covariates? Submitted. Available at http://www.stat.ubc.ca/people/gustaf.

15. Simulation: Dichotomizing makes matters worse when measure is unreliable b1 = .4 X1 Y True Model: X1 continuous

16. Simulation: Dichotomizing makes matters worse when measure is unreliable b1 = .4 X1 Y Same Model with X1 dichotomized

17. Simulation: Dichotomizing makes matters worse when measure is unreliable Reliability=.65, .75., .85, 1.00 b1 = .4 X1 Y Contin. b1 = .4 Dich. X1 Y Models with reliability of X1 manipulated

18. Dichotomization of a variable measured with error (y = .4x + e)

19. Dichotomization of a variable measured with error (y = .4x + e)

20. Dichotomizing will obscure non-linearity

21. Dichotomizing will obscure non-linearity

22. Simulation 2: Dichotomizing a continuous predictor that is correlated with another predictor X1 and X2 continuous b1 = .4 X1 Y X2 b2 = .0

23. Simulation 2: Dichotomizing a continuous predictor that is correlated with another predictor X1 dichotomized b1 = .4 X1 Y X2 b2 = .0

24. Simulation 2: Dichotomizing a continuous predictor that is correlated with another predictor X1 dichotomized; rho12 manipulated b1 = .4 X1 r12 = .0, .4, .7 Y X2 b2 = .0

25. Simulation 2: Dichotomizing a continuous predictor that is correlated with another predictor

26. Simulation 2: Dichotomizing a continuous predictor that is correlated with another predictor

27. Is it ever a good idea to categorize quantitatively measured variables? • Yes: • when the variable is truly categorical • for descriptive/presentational purposes • for hypothesis testing, if enough categories are made. • However, using many categories can lead to problems of multiple significance tests and still run the risk of misclassification

28. CONCLUSIONS • Cutting: • Doesn’t always make measurement sense • Almost always reduces power • Can fool you with too much power in some instances • Can completely miss important features of the underlying function • Modern computing/statistical packages can “handle” continuous variables • Want to make good clinical cutpoints? Model first, cut later.

29. Clinical Events and LVEF Change during Mental Stress: 5 Year follow-up Model first, cut later Prob{event} Maximum Change in LVEF (%)

30. Requirements: Sample Size • Linear regression • minimum of N = 50 + 8:predictor (Green, 1990) • Logistic Regression • Minimum of N = 10-15/predictor among smallest group (Peduzzi et al., 1990a) • Survival Analysis • Minimum of N = 10-15/predictor (Peduzzi et al., 1990b)

31. Concept of Simulation Y = b X + error bs1 bs2 bsk-1 bsk bs3 bs4 ………………….

32. Concept of Simulation Y = b X + error bs1 bs2 bsk-1 bsk bs3 bs4 …………………. Evaluate

33. Simulation Example Y = .4 X + error bs1 bs2 bsk-1 bsk bs3 bs4 ………………….

34. Simulation Example Y = .4 X + error bs1 bs2 bsk-1 bsk bs3 bs4 …………………. Evaluate

35. True Model:Y = .4*x1 + e

36. Sample Size • Linear regression • minimum of N = 50 + 8:predictor (Green, 1990) • Logistic Regression • Minimum of N = 10-15/predictor among smallest group (Peduzzi et al., 1990a) • Survival Analysis • Minimum of N = 10-15/predictor (Peduzzi et al., 1990b)

37. All-noise, but good fit

38. Simulation: number of events/predictor ratio Y = .5*x1 + 0*x2 + .2*x3 + 0*x4 -- Where r x1 x4 = .4 -- N/p = 3, 5, 10, 20, 50

39. Parameter stability and n/p ratio

40. Peduzzi’s Simulation: number of events/predictor ratio P(survival) =a + b1*NYHA + b2*CHF + b3*VES +b4*DM + b5*STD + b6*HTN + b7*LVC --Events/p = 2, 5, 10, 15, 20, 25 --% relative bias = (estimated b – true b/true b)*100

41. Simulation results: number of events/predictor ratio

42. Simulation results: number of events/predictor ratio

43. Predictor (covariate) selection • Theory, substantive knowledge, prior models • Testing for confounding • Univariate testing • Last (and least), automated methods, aka stepwise and best subset regression

44. Searching for Confounders • Fundamental tension between underfitting and overfitting • Underfitting = not adjusting for important confounders • Overfitting = capitalizing on chance relations (sample fluctuation)

45. Covariate selection • Overfitting has been studied extensively • “Scariest” study is by Faraway (1992)—showed that any pre-modeling strategy cost a df over and above df used later in modeling. • Premodeling strategies included: variable selection, outlier detection, linearity tests, residual analysis.

46. Covariate selection • Therefore, if you transform, select, etc., you must include the DF in (i.e., penalize for) the “Final Model”

47. Covariate selection: Univariate Testing • Non-Significant tests also cost a DF • Variables may not behave the same way in a multivariable model—variable “not significant” at univariate test may be very important in the presence of other variables

48. Covariate selection • Despite the convention, testing for confounding has not been systematically studied—likely leads to overadjustment and underestimate of true effect of variable of interest. • At the very least, pulling variables in and out of models inflates the Type I error rate, sometimes dramatically

49. SOME of the problems with stepwise variable selection. 1. It yields R-squared values that are badly biased high

50. SOME of the problems with stepwise variable selection. 1. It yields R-squared values that are badly biased high 2. The F and chi-squared tests quoted next to each variable on the printout do not have the claimed distribution