Amy M. Cohn, James Grice, Brett Hagman , and Liz Schlimgen November 17, 2012 ABCT

1. Amy M. Cohn, James Grice, Brett Hagman, and Liz Schlimgen November 17, 2012 ABCT Using Observation-Oriented Modeling to Examine Daily Patterns and Predictors of Post-traumatic Stress Symptomatologyin a Sample of Female Rape Victims

2. Aim of Presentation • Test a mediation model with daily diary data • Compare multi-level modeling (MM) to Observation Orientated Modeling (OOM) Daily negative affect Daily post-traumatic stress symptoms Daily alcohol involvement

3. Assumptions of MM have limitations • Homogeneity between individuals • Within-person fluctuations in behavior represented as aggregated, over-time association (slope) • Linear monotonic changes in behavior over time • Associations between variables are additive, not (necessarily) dynamic • Random sampling • Normal population distribution (for differences) • Abstract population parameters that have little (or no) empirical basis • The population is completely theoretical

4. What’s wrong with the p-value? • Not a new argument • Relies on population statistics that may not represent the data • The sample is different, the way you collect the data is different, the questions you ask are different…. • Creates a “false belief” in the validity and generalizability of findings • Many study results cannot be replicated

5. Observation Orientated Modeling “Why is it that the patterns of phenomena are the way they are?” (Harre, 1986) “Fundamentally incompatible with prevailing research tradition in Psychology” (Grice, 2012)

6. OOM Incompatible with MM • Variable-based approach (such as MM) is linear, causal, and based on aggregate statistics such as betas and variances • OOM approach is integrative and focused at the level of the individual Independent variable Dependent variable

7. OOM • Non-parametric, idiographic • Examines qualitative pattern in the data • Rooted in Aristotle’s notion that most things in nature are not produced by people • The researcher does not control everything in a study • Eschews null hypothesis significance testing (NHST) • Results based on probabilities found within the data, not comparison to population distribution • Variables not described in a cause-effect format • OOM describes how the effect conforms to the cause

8. OOM • To Repeat……. • EFFECTS SHOULD CONFORM TO THEIR CAUSES • What the @#*&$? • We do not always know why participants do what they do • Effects are never truly “causal” • Unmeasured pieces of “error” or “garbage” in the data collection process • With OOM, patterns of observations reveal what are in the data – The EFFECTS

9. Study 1 Hypotheses • NA will be greater on days characterized by greater PTSD • Craving and consumption will be greater on days characterized by more intense PTSD and NA Daily NA a b Daily PTSD symptoms Alcohol Involvement c (c’)

10. Sample characteristics (n = 54) • 54 untreated female rape victims who completed at least one day of daily interactive voice response (IVR) monitoring

11. IVR Assessment • 1x a day in the evening (6pm to 12am) • Alcohol use, negative affect intensity, craving intensity, and PTSD symptoms (presence/absence) • Since previous phone call • 93% compliance rate • 13/14 calls were completed

12. HLM Analysis • DV’s = Number of drinks consumed and intensity of craving (850 observations) • Controlled for day of week • Poisson distribution with log link function for drinking • Examined relationship of one variable EACH DAY to the outcome variable ON THAT SAME DAY

13. Figure 1. Mediation of NA on the PTSD-alcohol link. Daily NA 0.13*** 0.42*** Daily PTSD symptoms -0.14 (-0.02) Number of standard drinks/day *** p < .001 In Cohn, Hagman, Moore, Mitchell, Ehlke, and Bramm (under review)

14. Figure 2. Mediation of NA of the PTSD-craving link. Daily NA 0.13*** 0.39*** Daily PTSD symptoms -0.10 (-0.12) Daily craving intensity Note. Covariates included day of the week, baseline PTSD symptom severity, baseline alcohol use. *** p < .001 In Cohn, Hagman, Moore, Mitchell, Ehlke, and Bramm (under review)

15. OOM Analysis: Mediation Steps Daily NA Daily PTSD symptoms Number of standard drinks/day Step 1: Because the effect conforms to the cause, we first examine the probability that number of standard drinks consumed each day conforms to daily ratings of NA intensity

16. OOM Results • Accuracy rate: % observations correctly classified out of total number of observations • Missing data is not a problem • Randomization test • Out of1000 trials of randomized versions of the same observations, what number of instances do we obtain a result high or higher than percent correct classification? • Binomial p-value or chance value should be small (less than .01) • Indicates pattern is unique • Results for individual and group-level patterns

17. Perfect Ordinal Matches for 14 Occasions Proportion of Matches = 1.00; Binomial p-value = .00012

18. Weak Ordinal Matches for 14 Occasions Proportion of Matches = .15; p-value = .99

19. Aggregate Results for all 54 Women • Overall Results (n = 54 women) : • Number of Matches : 123 • Number of Observations : 399 • Proportion of Matches : 0.31 • Randomization Results : • Observed Proportion of Matches : 0.31 • Number of Randomized Trials : 5000.00 • Minimum Random Proportion of Matches : 0.24 • Maximum Random Proportion of Matches : 0.36 • Values >= Observed Proportion : 1758.00 • Matching c-value : 0.35 • Proportion of matches is unimpressive at .31 • C-value of the Randomization Test indicates that .31 is not an unusual aggregate outcome compared to randomized versions of the same observations

20. Aggregate Results for all 54 Women • Proportion of Matches > .50 for only 9 women(5 of these women had 7 or fewer data points) • Fourteen women (26%) showed no variability in their drinking across the 14 days • An additional 6 women drank on only one day

21. Conclusions • Women showing no variability in drinking and those who did not drink across 14 days are “swept” into HLM aggregates • Should this disturb us? • OOM recognizes women with no variability in their drinking • Since OOM not based on means and variances, impact of these women does not adversely effect the overall percent matches

22. Conclusions • OOM “effect sizes” are proportions of matches that are readily interpretable and linkable to individual women • No need for interpretations- such as Cohen’s effect sizes • Idealized p-values are primary in HLM, even over effect sizes • Even if effect is small, if p < .05 we say “YES”! • Proportions of matches consistent with causal hypotheses are primary in OOM • Distribution free p-values (from binomial and randomization tests) are secondary

23. Summary • Erroneously enticed to posit a mediation mechanism that operates successfully for every woman with HLM • OOM treats the women and their individual observations as primary • Does not rely on p-values, means, or variances estimated from a theoretical population • OOM develops integrated models • More accurately explains patterns of observations

24. Acknowledgments • Participants who dedicated their time and effort • Research assistants: Jessica Mitchell, Stephanie Bramm, Sarah Ehlke, Ruschelle Leone, Joanne Wang • Grants: NIDA P30DA028807; USF 582000 / MHBCSG

25. Thank you! Questions? Dr. James Grice Department of Psychology Oklahoma State University Stillwater, OK 74078 James.grice@okstate.edu

26. Deep Structure Transformation CauseObservations EffectObservations 1. Observations are transformed into their “deep structure” M F 0 1 0 1 1 0 1 0 0 1 2 3 4 5 6 7 8 9 10 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 2. Rotate deep structure effect observations into “conformity” with deep structure observations 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 Conformed EffectObservations EffectObservations 3. Accuracy is our central judgment (not statistical significance) and shows how many observations were correctly classified by the algorithm, or how many observations match the pattern.