If you think you made a lot of mistakes in the survey project….

1. If you think you made a lot of mistakes in the survey project…. Think of how much you accomplished and the mistakes you did not make…

2. Went from not knowing much about surveys to having designed, deployed, and completed one in 1 ½ months • Actually got people to respond! • Did not end up with 100 open ended responses which you had to content analyze!

3. One Tailed and Two Tailed tests One tailed tests: Based on a uni-directional hypothesis Example: Effect of training on problems using PowerPoint Population figures for usability of PP are known Hypothesis: Training will decrease number of problems with PP Two tailed tests: Based on a bi-directional hypothesis Hypothesis: Training will change the number of problems with PP

4. Sampling Distribution Population for usability of Powerpoint 1400 1200 1000 800 Frequency 600 400 Std. Dev = .45 200 Mean = 5.65 N = 10000.00 0 3.75 4.75 5.25 6.25 7.00 7.25 4.00 4.25 4.50 5.00 5.50 5.75 6.00 6.50 6.75 Mean Usability Index If we know the population mean Identify region Unidirectional hypothesis: .05 level Bidirectional hypothesis: .05 level

5. What does it mean if our significance level is .05? • For a uni-directional hypothesis • For a bi-directional hypothesis PowerPoint example: • Unidirectional • If we set significance level at .05 level, • 5% of the time we will higher mean by chance • 95% of the time the higher mean mean will be real • Bidirectional • If we set significance level at .05 level • 2.5 % of the time we will find higher mean by chance • 2.5% of the time we will find lower mean by chance • 95% of time difference will be real

6. Changing significance levels • What happens if we decrease our significance level from .01 to .05 • Probability of finding differences that don’t exist goes up (criteria becomes more lenient) • What happens if we increase our significance from .01 to .001 • Probability of not finding differences that exist goes up (criteria becomes more conservative)

7. PowerPoint example: • If we set significance level at .05 level, • 5% of the time we will find a difference by chance • 95% of the time the difference will be real • If we set significance level at .01 level • 1% of the time we will find a difference by chance • 99% of time difference will be real • For usability, if you are set out to find problems: setting lenient criteria might work better (you will identify more problems)

8. Effect of decreasing significance level from .01 to .05 • Probability of finding differences that don’t exist goes up (criteria becomes more lenient) • Also called Type I error (Alpha) • Effect of increasing significance from .01 to .001 • Probability of not finding differences that exist goes up (criteria becomes more conservative) • Also called Type II error (Beta)

9. Degree of Freedom • The number of independent pieces of information remaining after estimating one or more parameters • Example: List= 1, 2, 3, 4 Average= 2.5 • For average to remain the same three of the numbers can be anything you want, fourth is fixed • New List = 1, 5, 2.5, __ Average = 2.5

10. Major Points • T tests: are differences significant? • One sample t tests, comparing one mean to population • Within subjects test: Comparing mean in condition 1 to mean in condition 2 • Between Subjects test: Comparing mean in condition 1 to mean in condition 2

11. Effect of training on Powerpoint use • Does training lead to lesser problems with PP? • 9 subjects were trained on the use of PP. • Then designed a presentation with PP. • No of problems they had was DV

12. Powerpoint study data • Mean = 23.89 • SD = 4.20

13. Results of Powerpoint study. • Results • Mean number of problems = 23.89 • Assume we know that without training the mean would be 30, but not the standard deviation Population mean = 30 • Is 23.89 enough smaller than 30 to conclude that training affected results?

14. One sample t test cont. • Assume mean of population known, but standard deviation (SD) not known • Substitute sample SD for population SD (standard error) • Gives you the t statistics • Compare t to tabled values which show critical values of t

15. t Test for One Mean • Get mean difference between sample and population mean • Use sample SD as variance metric = 4.40

16. Degrees of Freedom • Skewness of sampling distribution of variance decreases as n increases • t will differ from z less as sample size increases • Therefore need to adjust t accordingly • df = n - 1 • t based on df

17. Looking up critical t (Table E.6)

18. Conclusions • Critical t= n = 9, t.05 = 2.62 (two tail significance) • If t > 2.62, reject H0 • Conclude that training leads to less problems

19. Factors Affecting t • Difference between sample and population means • Magnitude of sample variance • Sample size

20. Factors Affecting Decision • Significance level a • One-tailed versus two-tailed test

21. Sampling Distribution of the Mean • We need to know what kinds of sample means to expect if training has no effect. • i. e. What kinds of sample means if population mean = 23.89 • Recall the sampling distribution of the mean.

22. Sampling Distribution of the Mean--cont. • The sampling distribution of the mean depends on • Mean of sampled population • St. dev. of sampled population • Size of sample

23. Sampling Distribution Number of problems with Powerpoint Use 1400 1200 1000 800 Frequency 600 400 Std. Dev = .45 200 Mean = 23.89 N = 10000.00 0 3.75 4.00 4.25 4.50 4.75 5.00 5.25 5.50 5.75 6.00 6.25 6.50 6.75 7.00 7.25 Mean Number of problems Cont.

24. Sampling Distribution of the mean--cont. • Shape of the sampled population • Approaches normal • Rate of approach depends on sample size • Also depends on the shape of the population distribution

25. Implications of the Central Limit Theorem • Given a population with mean = m and standard deviation = s, the sampling distribution of the mean (the distribution of sample means) has a mean = m, and a standard deviation = s /n. • The distribution approaches normal as n, the sample size, increases.

26. Demonstration • Let population be very skewed • Draw samples of 3 and calculate means • Draw samples of 10 and calculate means • Plot means • Note changes in means, standard deviations, and shapes Cont.

27. Parent Population Cont.

28. Sampling Distribution n = 3 Cont.

29. Sampling Distribution n = 10 Cont.

30. Demonstration--cont. • Means have stayed at 3.00 throughout--except for minor sampling error • Standard deviations have decreased appropriately • Shapes have become more normal--see superimposed normal distribution for reference

31. Within subjects t tests • Related samples • Difference scores • t tests on difference scores • Advantages and disadvantages

32. Related Samples • The same participants give us data on two measures • e. g. Before and After treatment • Usability problems before training on PP and after training • With related samples, someone high on one measure probably high on other(individual variability). Cont.

33. Related Samples--cont. • Correlation between before and after scores • Causes a change in the statistic we can use • Sometimes called matched samples or repeated measures

34. Difference Scores • Calculate difference between first and second score • e. g. Difference = Before - After • Base subsequent analysis on difference scores • Ignoring Before and After data

35. Effect of training

36. Results • The training decreased the number of problems with Powerpoint • Was this enough of a change to be significant? • Before and After scores are not independent. • See raw data • r = .64 Cont.

37. Results--cont. • If no change, mean of differences should be zero • So, test the obtained mean of difference scores against m = 0. • Use same test as in one sample test

38. t test D and sD = mean and standard deviation of differences. df = n - 1 = 9 - 1 = 8 Cont.

39. t test--cont. • With 8 df, t.025 = +2.306 (Table E.6) • We calculated t = 6.85 • Since 6.85 > 2.306, reject H0 • Conclude that the mean number of problems after training was less than mean number before training

40. Advantages of Related Samples • Eliminate subject-to-subject variability • Control for extraneous variables • Need fewer subjects

41. Disadvantages of Related Samples • Order effects • Carry-over effects • Subjects no longer naïve • Change may just be a function of time • Sometimes not logically possible

42. Between subjects t test • Distribution of differences between means • Heterogeneity of Variance • Nonnormality

43. Powerpoint training again • Effect of training on problems using Powerpoint • Same study as before --almost • Now we have two independent groups • Trained versus untrained users • We want to compare mean number of problems between groups

44. Effect of training

45. Differences from within subjects test Cannot compute pairwise differences, since we cannot compare two random people We want to test differences between the two sample means (not between a sample and population)

46. Analysis • How are sample means distributed if H0 is true? • Need sampling distribution of differences between means • Same idea as before, except statistic is (X1 - X2) (mean 1 – mean2)

47. Sampling Distribution of Mean Differences • Mean of sampling distribution = m1 - m2 • Standard deviation of sampling distribution (standard error of mean differences) = Cont.

48. Sampling Distribution--cont. • Distribution approaches normal as n increases. • Later we will modify this to “pool” variances.

49. Analysis--cont. • Same basic formula as before, but with accommodation to 2 groups. • Note parallels with earlier t

50. Degrees of Freedom • Each group has 6 subjects. • Each group has n - 1 = 9 - 1 = 8 df • Total df = n1- 1 + n2 - 1 = n1 + n2 - 2 9 + 9 - 2 = 16 df • t.025(16) = +2.12 (approx.)