1 / 9

Is a “Discussion” on “Are Observational Studies Any Good” Any Good

Is a “Discussion” on “Are Observational Studies Any Good” Any Good . Don Hoover May 2, 2014. Everyone Already Knows Observational Studies Are Not Perfect … Right?. But who thinks the real type 1 error is 0.55 when the nominal is 0.05? The real coverage of a 95% confidence interval is 25%?

aelan
Download Presentation

Is a “Discussion” on “Are Observational Studies Any Good” Any Good

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Is a “Discussion” on “Are Observational Studies Any Good” Any Good Don Hoover May 2, 2014

  2. Everyone Already Knows Observational Studies Are Not Perfect … Right? • But who thinks • the real type 1 error is 0.55when the nominal is 0.05? • The real coverage of a 95% confidence interval is 25%? • That’s what David Madigan and the OMAP team find • This obviously makes such results meaningless • But how many papers with these properties are being (and will continue to be) published ???

  3. But Does David’s Talk Really Apply to ALL Observational Studies? • They Only Look at Observational Studies of Drug Use and Adverse Consequences • There’s other kinds of Observational Studies … on HIV, Epi, Health Behaviors, Nutrition, etc. • No one has looked at these types of studies • These other studies must have similar problems • Maybe ata smaller magnitude • But there are no “negative controls” for these settings … so no one can check this

  4. The Approach here is Creative and Innovative • Finding Negative Control Exposures or Outcomes to derive empirical distribution of the test statistic somewhat equalizes assumptions and unmeasured confounding • With a given Drug Use as the exposure and agiven Disease the outcome, such negative controls are readily available in many data sets • So maybe something like it should be used when possible • But now some questions ……

  5. Q1- Why were Negative Control Drugs More Associated With Outcomes than by Chance? • People put on Any Drug are Sicker? • Those receiving a negative (control) drug are more likely to receive some other positive drug? • Those apriori more likely to have a given disease outcome are steered to the negative drugs? • Incorrect statistical models used?

  6. Q2- Is this Approach Practical? • A lot more work to fit many models than the standard approach which only fits one • More money as well - A grant application using it would be less likely to get funded • More work also means more chance for error in implementation

  7. Q3 – How does one interpret a positive drug with empirical P < 0.05? Calibrated Normal Scores of Negative Controls Positive Drug with empirical P < 0.05 The use of an “empirical” approach acknowledges we do not know what is going on so maybe the P < 0.05 is from model artifact not causal

  8. Q4 – What is done with “Negative Drugs” more extreme than the Positive One Calibrated Normal Scores of Negative Controls Positive Drug with P < 0.05 Should these Negative Controls all be Examined for Causal Association as their Signal is larger than the positive drug?

  9. Q5 - How to handle Heterogeneity in Denominator of Calibration Statistic From Schumie … Madigan Stat Med 2014 33; 209-18 Variance may introduce Apples to Oranges comparisons especially if although such does not appear to be the case in the examples David used

More Related