Ask Good Questions

1. Ask Good Questions Allan J. Rossman Dept of Statistics Cal Poly – San Luis Obispo

2. Frank Sinatra • What’s the secret to success as a singer? • “Sing Good Songs” Ask Good Questions

3. My similarly succinct suggestion • What’s the secret to effectiveness as a teacher? • “Ask Good Questions” Ask Good Questions

4. George Cobb • “Judge a textbook by its exercises, and you cannot go far wrong.” Ask Good Questions

5. My corollary • “Judge a teacher by the questions that he/she asks of students, and you cannot go far wrong.” Ask Good Questions

6. What kinds/purposes of questions? • Guide students to develop their understanding and skills • Learning activities • Assess how well students have learned • Assessment questions Ask Good Questions

7. What makes a question “good”? • I respectfully decline to answer • For now • Instead I’ll present some examples of both kind of questions (learning activities, assessment questions) • Most from “Stat 101” (algebra-based) • Some from “Stat 201” (calculus-based) • Then I’ll revisit this question Ask Good Questions

8. Six examples (learning activities) Ask Good Questions

9. George Cobb (again) • “Shorn of all subtlety and led naked out of the protective fold of educational research literature, there comes a sheepish little fact: lectures don’t work nearly as well as many of us would like to think.” Ask Good Questions

10. Example 1: TVs and life expectancy • Is there an association between a country’s life expectancy and its number of televisions per person? Ask Good Questions

11. Example 1: TVs and life expectancy • Does the scatterplot reveal an association? Which direction? How strong? Linear? • Based on these data, would you conclude that sending TVs to Uganda would cause Ugandans to start living longer? • Identify a confounding variable that explains the association • Does a strong association between variables imply a cause/effect relationship? Ask Good Questions

12. Example 2: Reading cancer pamphlets • Are pamphlets containing information for cancer patients written at an appropriate level that cancer patients can understand?

13. Example 2: Reading cancer pamphlets • Explain why the mean cannot be calculated for patients’ reading levels • Determine median reading level of patients, median readability level of pamphlets • Are the pamphlets’ readability levels well-matched to the patients’ reading levels? • What proportion of patients have reading level below that of simplest pamphlet? Ask Good Questions

14. Example 2: Reading cancer pamphlets

15. Example 3: Sleep deprivation Does sleep deprivation have harmful effects on cognitive functioning three days later? 21 subjects; random assignment Identify type of study, observational units, explanatory and response variables 15 15

16. Example 3: Sleep deprivation Students use simulation (tactile, then computer-based) to investigate core question of statistical inference: Is such an extreme difference unlikely to occur by chance (random assignment) alone (if there were no treatment effect)? 16 16

17. Example 3: Sleep deprivation • Summarize conclusion, and explain reasoning process that supports your conclusion

18. Example 4: Which tire? • Legendary campus story • Which tire would you pick? • Research question: Do people tend to pick right front tire more often than random chance? • Again, students investigate the question of how surprising the observed class result would be under the null model with ¼ probability Ask Good Questions

19. Example 4: Which tire? • What if 32% of a random sample selects right front? Is this a significant/convincing result? • Or, what additional information would you need? • Make prediction for how p-value will change (if at all) as sample size increases • Calculate p-value for many different sample sizes • Summarize how sample size affects p-value, strength of evidence • Explain why this makes intuitive sense (in hindsight, if not in foresight) Ask Good Questions

20. Example 5: Minimization criteria • How to choose a measure of center to achieve an optimization criterion? • For a given dataset, evaluate and graph the function SAD(m) = S |xi – m| • Determine the value(s) of m that minimize SAD(m) • Make a conjecture about where SAD(m) is minimized for an arbitrary dataset • Repeat with SSD(m) • Use calculus to prove conjecture • Repeat for functions such as MAXAD(m) Ask Good Questions

21. Example 5: Minimization criteria Ask Good Questions

22. Example 5: Minimization criteria Ask Good Questions

23. Example 6: Comparing procedures • Which 95% confidence interval procedure for a population proportion works better, and under what conditions? • And what does “works better” mean? Ask Good Questions

24. Students conduct simulation analysis Compare capture rates for two procedures For various values of n, p Example 6: Comparing procedures

25. Six examples (assessment items) Ask Good Questions

26. Joan Garfield, Beth Chance • Because students learn to value what they know they will be tested on, we should assess what we value.” - JG • “The number one mantra to remember when designing assessment instruments is: Assess what you value.” - BC Ask Good Questions

27. Assessment example 1 For each of the following quantities, indicate whether it can NEVER be negative or can SOMETIMES be negative. a) Sample size b) Sample proportion c) Standard deviation d) Inter-quartile range e) Difference in sample means f) Odds ratio g) Total sum of squares h) Slope coefficient i) Coefficient of determination j) Correlation coefficient k) ANOVA F-test statistic l) p-value Ask Good Questions

28. Assessment example 2 The U.S. has slightly more than 300 million residents. In order to estimate the proportion of U.S. residents who have a facebook account to within about 3 percentage points with 95% confidence, about how many people should be randomly sampled? 100 1000 10,000 100,000 1,000,000 10,000,000 Ask Good Questions

29. Assessment example 3 Are people more likely to lie with email than with pencil-and-paper communication? A recent study involved 48 graduate students in business at a particular university who participated in a “bargaining” game. Researchers kept track of whether the student lied about the amount of money involved when negotiating with another player. Some of the participants were randomly assigned to use email for their communication, while others used paper-and-pencil. It turned out that 24 of 26 who used email were guilty of lying about the amount of money involved, compared to 14 of 22 who used paper-and-pencil. Ask Good Questions

30. Assessment example 3 (cont.) Use a simulation analysis to investigate whether these data provide strong evidence in support of the research conjecture in the first sentence above. Summarize the conclusions that you draw from your analysis. Be sure to address issues of statistical significance, causation, and generalizability. Also explain the reasoning process and justification for your conclusions. Ask Good Questions

31. Assessment example 4 State a research question, and describe a data collection plan to address it, for which a two-proportion z-test would be the relevant inference procedure. Identify the explanatory and response variables, and also state the relevant hypotheses, defining the parameter values appropriately. Be sure to indicate whether the data collection plan involves random sampling or random assignment. Ask Good Questions

32. Assessment example 5 Create a hypothetical example of two softball players batting in two months for which Simpson’s paradox holds: Amy has a higher proportion of hits than Barb in both June and July, but Barb has a higher proportion of hits than Amy for the two months combined. Ask Good Questions

33. Assessment example 6 (from AP Statistics) • Consider the statistic mean / median. What values of this statistic might indicate that the population distribution is skewed to the right? • Consider simulation results for values of mean / median, based on a normal population, and also the observed value of mean / median for given sample data. Do the simulation results suggest that the underlying population is skewed to the right? Explain. Ask Good Questions

34. Assessment example 6 (from AP Statistics, cont.) • Use only statistics in the five-number summary (min, Q1, median, Q3, max) to propose your own measure of skewness. Indicate values of this statistic that would suggest skewness to the right. Explain. Ask Good Questions

35. Again, what makes a question “good”? • Makes students think (duh!) • Goes beyond their starting point • Challenges without overwhelming students • Engages students’ interest • To put forth effort to learn • To care about the course material being studied • Addresses important ideas • Indicates to students what’s valued Ask Good Questions

36. What else makes a question “good”? • Fits within logical progression • Builds on students’ knowledge • Asks students to make/check predictions • Confronts students’ misconceptions • Inspires students to ask their own questions • About course material • About “real-world” phenomena that can be investigated using intellectual skills being learned Ask Good Questions

37. Bill James • What’s the secret to your success as a baseball statistician? • “I don’t understand the world.” Ask Good Questions

38. An example of students posing and investigating own questions • Students use gapminder software (Hans Rosling) to • Pose research questions about the world • Investigate those questions with animated graphics • Write report of their findings Ask Good Questions

39. Some questions from students’ projects • Are students more likely to recycle water bottles depending on the proximity of recycle bin and waste basket? • Can people better answer math problems if they are presented with Roman letter notation as opposed to Greek letter notation? • Are people who walk into a clothing store more likely to purchase something when the weather is rainy as opposed to sunny? • Is balsa wood less elastic after it has been immersed in water? • Are students’ reaction times affected by whether they’ve just completed exercising? • Are faculty more likely to drive a foreign car than students? • Do guests at a dinner respond more positively to spaghetti sauce that has been sweetened or not sweetened? • Do college students spend more money at a local fast food restaurant if they are under the influence of alcohol? Ask Good Questions

40. David Moore Don’t just answer your own questions. “Wait 30 seconds.” Rossman 40 Ask Good Questions

41. Some final questions for you • Have you ever attended a presentation titled • Ask Bad Questions, or • Don’t Ask Questions? • Of course not! So, … • Why am I wasting your time offering such obvious advice as “Ask Good Questions”?? Ask Good Questions

42. Yeah, why am I wasting your time? • I respectfully suggest that the next time you or I feel compelled to invest some of our valuable time, thought, energy, and creativity on … • Preparing crystal-clear lectures • Writing lucid paragraphs of exposition • Crafting beautiful presentations Ask Good Questions

43. What should we do? Rossman 43 Ask Good Questions

44. What should we do? Resist this temptation! Rossman 44 Ask Good Questions

45. So, what do I suggest instead? • Instead we should invest these precious commodities (time, thought, energy, creativity) on • Developing engaging classroom activities • Preparing thought-provoking assignments • Writing lab or project expectations • Crafting effective assessment items • In other words, we should focus more of our attention on making sure that we … Ask Good Questions

46. What’s my point again? Ask Good Questions! Ask Good Questions

47. Thanks very much! • This talk prepared by: • arossman@calpoly.edu • statweb.calpoly.edu/arossman/AskGoodQuestions.ppt Ask Good Questions