STAT131 Week 2 Lecture 1b Making Sense of Data

1. STAT131Week 2 Lecture 1bMaking Sense of Data Anne Porter

2. Review • Learning and Writing • what why how when • Statistics is a study of variation throughout a process

3. Review Statistical Process Process Ethics The nature of the question to be answered Expertise Design Sampling Measurement Description and Analysis (Making sense of data) Conclusions & Decision Making

4. Where we do what! • In lectures the focus - • What are we doing? • Why are we doing it? • When do we do it? • In labs the focus - • How do we do it? • Check definitions, • Do by hand (simple) and SPSS • Making choices about what to use

5. Making sense of raw data A shoe seller sets up on campus & collects some data about what size shoes students wear. What do you see in this data?

6. Making sense of raw data What might we do to to make sense out of the shoe size data?

7. What might we do to make sense? • Order the data • Calculate the centre • Mean average score • Median middle score of ordered values • Mode most common score • Find the spread • Range from minimum to maximum • Look for outliers unusual values

8. Descriptive Statistics (mean, range) What do these statistics tell us? Is this what the shoe seller needs to know?

9. Descriptive Statistics (mean, range) What do these statistics tell us? There is an error in the data! Minimum size 4, Maximum 42 Average is 9.81 Is this what the shoe seller needs to know? Range= Maximum less minimum =42-4 =38 No

10. Five number summary • SHOESIZE Five number summary • Minimum • Maximum • Lower quartile or 25th Percentile: shoe size with 25% of shoe sizes below it • Median, 5oth percentile or middle shoe size • Upper quartile 75th Percentile with 75% shoe sizes below it (ie 25% above it) • The interquartile range shoe size 75th percentile-shoe size 25th percentile • What is a percentile? • How do you calculate quartiles? • And is this what the shoe seller wants?

11. Five number summary • SHOESIZE Five number summary • Minimum 4 • Maximum 42 • Lower quartile , 25% of shoe sizes below = 8 • Median, 50% of shoe sizes below it = 9.5 • Upper quartile, 75% of shoe sizes below it =11 • The interquartile range 75th percentile-shoe size 25th percentile 11-8 • (50% of sizes between 8 and 11) What is a percentile? Does the shoeseller have what is needed?

12. Percentiles - definition • The kth percentile is a number that has k percent of the scores at or below it and (100-k)% above it • The lower quartile has 25% of scores at or below that score

13. Quartiles • Q1 is value of the (n+3)/4th observation, • and Q3 is the value of the (3n+1)/4th observation. • Interpolate if necessary. • There are other approaches to calculating which may give different answers. If the answers are similar there is no problem • The interquartile range=Q3 - Q1 • If we have 17 heights what observation • do we need to get the upper and lower quartile? • What observation will give the median?

14. Quartiles - 19 The upper quartile is? The lower quartile is? The interquartile range is? 166 147

15. What other statistics or graphs might inform the shoe-seller? • Centre - mean, median • Spread • Maximum-Minimum = Range • Upper Quartile-Lower quartile = Interquartile range • 75th percentile-25th percentile= Interquartile range • Outliers

16. Shoe Size 4 5 5 6 : 42 Ordering is often useful but we can do better Ordering the data

17. What is wrong with this display? Frequency or relative frequency table

18. What is wrong with this display? Frequency or relative frequency table The data has been treated as if it were continuous. Some packages will do this but we want the data to be treated as discrete data

19. Frequency distribution (order plus count) We still have an error (42) But we have the frequency (count) of each shoe size. What might be better for the shoeseller?

20. Percentages of each size Why might this be useful rather than frequency?

21. Percentages of each size Why might this be useful rather than frequency? • We only had a sample so this would suggest the percentage or even proportion of each size. • Is this all the shoeseller needs?

22. There are better ways of looking at distributions • What else might we do?

23. Stem-and leaf plot (with error) • Some packages (SPSS) cut off the outliers and lists them as extremes. • See if you can find a definition for an extreme as used in SPSS and an outlier from the text. • Different packages, different procedures may use different definitions - check

24. What do we do with outliers?

25. What do we do with outliers? • Know the context to see what values are possible • Check the original data to see if it is a data entry error • See if it is in different units and transform to the appropriate unit • If an error and you do not know what it should be delete it and make a note • If there is no reason to conclude it is an error leave it in • Sometimes analyse with the point in and the point out of the the data set

26. Stem-and-leaf plot (42 removed) • What does it reveal? • Could it be better?

27. Stem-and-leaf plot (42 removed) • What does it reveal? • Could it be better? • Change stems to focus on whole and half sizes. • We should have transformed the 42. This is the difference between a lecture and data analysis, I deleted!

28. Stem-and-leaf with different stems • What do we notice now? • Do we have what the shoe seller needs?

29. Stem-and-leaf with different stems • What do we notice now? • There is a distribution within a distribution with fewer half sizes • Do we have what the shoe seller needs? • We need male and female data (Next lecture)

30. Graphical Excellence • Convey the message about the data • Axes, units, variable names, figure labels • DO NOT • Distort the data • Use pie charts (there is always a better chart) • More dimensions than necessary, 3D instead of 2D • Unnecessary pattern, fill, ink, decoration

31. To reveal Centre Spread Outliers Distribution Patterns Anything unusual Comparisons (next lecture) And more But there are choices to be made

32. Centre • Mean • Median • Mode • Trimmed Mean Median, FIRST arrange the sample values from smallest to largest. N odd : Median of 8, 7, 9 is the middle of ordered scores 8 N even: Median of 4,7,8,9 =(7+8)/2=7.5 Mode is the most common score in the data set eg for 1,2,3,3,4,5,6 The mode is 3 Trimmed Mean Eg. Diving at the Olympics is the average of the judges scores after having tossed out the highest and the lowest scores

33. Question: mean vs median • Data A: 60, 2, 3, 5 Data B: 6, 2, 3, 5 • Mean A = 17.5 Mean B = 4 • Median A = 4 Median B = 4 • Which measure best typifies the data A? Why? • Which measure best typifies the data set B? Why?

34. Question: mean vs median • Data A: 60, 2, 3, 5 Data B: 6, 2, 3, 5 • Mean A = 17.5 Mean B = 4 • Median A = 4 Median B = 4 • Which measure best typifies the data A? Why? • Which measure best typifies the data set B? Why? For A the outlier 60 suggests the median (4) as the Mean (17.5) is dragged up by the outlier 60 For B both are the same. The median (4) used 2 points the mean (4) uses all the data

35. Question: mean vs median • In what sense are the mean and median the same? • In what sense are the mean and median different?

36. Question: mean vs median • In what sense are the mean and median the same? • In what sense are the mean and median different? They are both measures of the centre They may give different numerical values and for different data sets one may be better as a measure than the other or both may be required

37. Making Choices between mean & median • The mean uses all the information in the sample, because each value is added in the sum. • mean subject to error if spurious values are entered. • median is less affected by “wild” values, we say it is robust. • If the mean is similar to median • use the mean as it uses all data. • often easier to work with the mean • If they are different because of non-symmetric distribution • Can be useful to report both • The context of what the data are are used for may also determine what is an appropriate measure

38. Measures of Spread • Range= maximum value - minimum value • Interquartile range = Upper Quartile-Lower quartile =Q1- Q3 • Sums of Squares • Variance (S2) • Standard Deviation

39. Use of standard deviation • The mean and std deviation gives information about where most of the distribution of values is to be found. • For many distributions, the range mean - 2 standard dev’s to mean + 2 standard dev’s (mean + 2SD) contains approx 95% of the distribution. • (The very least that this spread can contain is 75% of the distribution.)

40. Criteria for a good measure of spread • Whatever measure of variability (or spread) the measure should not be affected by adding a constant to each value so as to change the centre (or location) • If there is spread in the data it should indicate this • Should make sense in the context used • Should be robust, not influenced by outliers or extreme points

41. Undesirable features of measures of spread • Sensitive to outliers • Does not use all data • eg range based only on two scores • Difficult to understand • Eg sum of squares in this context as the answer is very big and gets bigger with every additional data point. But useful in other contexts

42. Revealing distributions • Frequency Distribution Table • Stem-and-Leaf • Histograms • Box-and-whiskers

43. Box-and-Whiskers plots • Often just calledbox plots,they give a pictorial summary of the data for a single variable. • They use the five-number summary: • minimum value, • Q1, • median, • Q3, • maximum value

44. Example: If minimum = 3, Q1 = 6, median=10, Q3 = 12, maximum = 16, the box plot would look like • You must draw a scale for the box plot. 2 4 6 8 10 12 14 16

45. In a horizontal box plot, a horizontal axis shows the scale. The box’s left and right boundaries are Q1 and Q3, and an inner line shows the median. • Whiskers are drawn outwards from the box to the minimum and maximum values. • Often the sample mean is also shown.

46. What values given rise to the box plot below: • If minimum= , Q1= , • median= , Q3 = , maximum= , • the box plot would look like • You must draw a scale for the box plot. 2 4 6 8 10 12 14 16

47. What do you want to see in data? • Information • Meaning • We must turn data into information in order to have meaning

48. What can we see in data? Location (centre) Spread Shape Outliers Unusual patterns Gaps, clusters How do batches differ

49. Tools for making meaning from data Ordering data Dot plots & jittered dot plots Stem-and-leaf plots Histograms, Boxplots, Bar charts Pie charts Frequency tables Numerical summaries

50. Selecting the tool depends on The question asked How the variable is measured The structure of the data Utility of the tool More in the next lecture and labs