Locating Variance: Post-Hoc Tests

1. Locating Variance: Post-Hoc Tests Developing Study Skills and Research Methods (HL20107) Dr James Betts

2. It is easy (i.e. data in  P value out) It provides the ‘Illusion of Scientific Objectivity’ Everybody else does it. Why do we use Hypothesis Testing?

3. P<0.05 is an arbitrary probability (P<0.06?) The size of the effect is not expressed The variability of this effect is not expressed Overall, hypothesis testing ignores ‘judgement’. Problems with Hypothesis Testing?

4. Lecture Outline: • Influence of multiple comparisons on P • Tukey’s HSD test • Bonferroni Corrections • Ryan-Holm-Bonferroni Adjustments.

5. Placebo Placebo Lucozade Lucozade Gatorade Gatorade Powerade Powerade

6. Placebo Placebo Lucozade Lucozade Gatorade Gatorade Powerade Powerade

7. i.e. Placebo vs Lucozade Placebo vs Gatorade Placebo vs Powerade Lucozade vs Gatorade Lucozade vs Powerade Gatorade vs Powerade We accept ‘significance’ and reject the null hypothesis at P0.05 (i.e. a 5% chance that we are wrong) Performing multiple tests therefore means that our overall chance of committing a type I error is >5%. Why not multiple t-tests?

8. Post-hoc Tests • A popular solution is the Tukey HSD (Honestly Significant Difference) test • This uses the omnibus error term from the ANOVA to determine which means are significantly different • T = (q) Error Variance √ n

11. q table for Tukey’s HSD

14. Tukey Test Critique • As you learnt last week, the omnibus error term is not reflective of all contrasts if sphericity is violated Placebo Lucozade • So Tukey tests commit many type I errors with even a slight degree of asphericity. Gatorade Powerade

15. Solution for Aspherical Data • There are alternatives to the Tukey HSD test which use specific error terms for each contrast • Fisher’s LSD (Least Significant Difference) • Sidak • Bonferroni • Many others… • e.g. Newman-Kewls, Scheffe, Duncan, Dunnett, Gabriel, R-E-G-W, etc.

18. Trial 2 Trial 4 Trial 1 Fisher’s LSD Bonferroni Trial 3

19. Bonferroni Correction Critique • Correction of LSD values successfully controls for type I errors following a 1-way ANOVA • However, factorial designs often involve a larger number of contrasts, many of which may not be relevant. Recovery Supp. 1 Recovery Supp. 2 See also Perneger (1998) BMJ 316: 1236

20. Solution for Factorial Designs • An adjustment to the standard Bonferroni correction can be applied for factorial designs • This ‘Ryan-Holm-Bonferroni’ or ‘stepwise’ method involves returning to the P values of interest from our LSD test • These P values are placed in numerical order and the most significant is Bonferroni corrected (i.e. P x m) • However, all subsequent P values are multplied by m minus the number of contrasts already corrected.

21. A Tukey test may be appropriate when sphericity can be assumed Multiple t-tests with a Bonferroni correction are more appropriate for aspherical data Stepwise correction of standard Bonferroni procedures maintain power with factorial designs Best option is to keep your study simple: Pre-planned contrast at a specific time point Summary statistics (e.g. rate of change, area under curve) Just make an informed based on the data available. Summary Post-Hoc Tests

22. Further reading from this lecture… • Atkinson, G. (2001) Analysis of repeated measurements in physical therapy research Physical Therapy in Sport 2: p. 194-208 • Atkinson, G. (2002) Analysis of repeated measurements in physical therapy research: multiple comparisons amongst level means and multi-factorial designs Physical Therapy in Sport 3: p. 191-203

23. Compulsory reading for next week’s lecture… • Batterham A. M. & Atkinson, G. (2005) How Big Does My Sample Need to Be? A primer on the Murky World of Sample Size Estimation Physical Therapy in Sport 6: p. 153-163.