Lecture 2

2. Lecture 2 Principles of Economic Experiments and Experimental Design

3. But 1st…results from last week’s 2nd experiment • Goal was to find out the role our subconscious plays in decision making • General result: • the 2nd guess in not necessarily better than the 1st, but the average of the two is better than the 1st • What this means is that even though you don’t have any more info on the correct answer, subconsciously you know whether you over- or underestimated your 1st guess

4. But 1st…results from last week’s 2nd experiment • Our result: • Remember, the actual distance from Vancouver to HCMC is roughly 11,500km • Circumference of Earth = 40,000km • Distance from Earth to Moon = 400,000km • Average of 1st guess = 228,051km • Average of 2nd guess = 180,498km • Average of both guess = 204,274km

5. But 1st…results from last week’s 2nd experiment • Conclusions: • The subconscious is powerful, and we need to understand it better • Why was our 2nd guess (and average of the two) better than our 1st when our 1st guess was supposed to be our best guess? • Many students need to travel to get a better sense of distance • Yet another reason to look into Study Abroad

6. Tutorials next week! • Tutorials begin next week • We will have our 1st experiment • It is very important to show up on time • If you have issues with the tutorial section you are in, come see me at the break • Issues must be serious (i.e. scheduling conflict with other course/work)

7. Course Schedule

8. Tutorials

9. TA Information

10. Principles of Economic Experiments

11. Big Questions to be Answered • How do you choose and present the rules governing an experimental economy? • How do you choose and motivatesubjects?

12. I. Realism and Models

13. What is the goal of designing an experiment? • To make the lab resemble the real-world as much as possible? • Too complex • The more complex the design of an experiment, the more expensive it is to conduct • Reality has an infinite amount of detail • Need to choose only the most important details relevant to the research question • e.g. rules and rewards not fashion style and scent of air

14. What is the goal of designing an experiment? 2. To replicate the assumptions of the formal, theoretical model? • Even if the observed behaviour of subject is consistent with the implications of the formal model, this only serves as weak support for the model • It would be stronger if you had observed the same behaviour by relaxing some of the more stringent assumptions of the model • e.g. # of sellers in a competitive market

15. What is the goal of designing an experiment? 3. Your goal should be to find a design that offers the best opportunity to learn something useful and answer the questions that motivate your research

16. Analogy to Art • An artist wishes to express a human event, say slavery • He is unable to re-enact the event since it took place so long ago • He finds it undesirable to replicate it closely for moral reasons • He chooses a medium, say canvas or stone • The quality of his painting will be judged by how well it simplifies reality to capture and communicate the essence of being a slave • Likewise, an experiment should be judged by its impact on our understanding, not how close it replicates reality

17. Analogy to Art The Captive Slave (1827) by British portraitist John Philip Simpson

18. II. Induced-Value Theory

19. How does the experimenter gain control of the subjects? • Induced-value Theory: • Proper use of a reward allows an experimenter to induce pre-specified characteristics in experimental subjects • With the proper reward, the subjects’ innatecharacteristics become largely irrelevant • this is extremely important when we want to start analyzing and interpreting the results of an experiment

20. What are the necessary conditions to induce subjects’ characteristics? • Monotonicity • Subjects always prefer more reward • Don’t choose a reward that people are bounded by • e.g. pieces of cake, glasses of juice, or anything people can get full of *NOTE: the best and most commonly used reward is cash • Easy to satisfy

21. What are the necessary conditions to induce subjects’ characteristics? 2. Salience • Relation between actions and rewards implements the desired institution • Fixed payment (e.g. $5 to show-up) • NOT SALIENT because payment does not depend on subjects’ actions • Performance-based payment (e.g. $1 per point of profit earned) • This IS SALIENT • Salience is what differentiates surveys from controlled economic experiments

22. What are the necessary conditions to induce subjects’ characteristics? 3. Dominance • Subjects are only motivated by their reward (i.e. not motivated by what others are getting) • Need for privacy • This is why many experiments are conducted in a lab using computer terminals as the interface

23. What have experimenters learned from induced-value theory? • To create a controlled economic environment, need to motivate subjects by paying them incash • Average payment should exceed the average opportunity cost of the subjects • Find subjects whose opportunity costs are low and whose learning curves are steep (e.g. undergrads!) • Create the simplest possible economic environment in order to promote salience and reduce ambiguities in interpreting the results

24. What have experimenters learned from induced-value theory? 5. Check instructions and verify subjects understand in “dry runs” or quizzes • Avoid “loaded” words in instructions • e.g. Prisoner’s Dilemma  actions A & B vs. Loyal & Betray 7. Do not deceive or lie to subjects • Salience and dominance are lost if subjects doubt the announced relation between actions and rewards

25. ANY QUESTIONS?

26. TIME FOR A BREAK COME BACK AT 1:30

27. Experimental Design

28. Introduction • How we design our experiment dictates the questions we can ask and answer • Last week’s “Battle of the Sexes” experiment was very simple in design, and thus could only make very limited comments on coordinating behavior

29. Introduction • How could we have changed last week’s experiment to make comments on: • The effects of using “hockey” and “ballet” as the labels for the actions • The effects of being punished for not coordinating

30. Introduction • Consider the following experimental design

31. Introduction • Now, consider the following experimental design NO PUNISHMENT PLAYER 2 IS PUNISHED

32. What is the Goal of an Experimental Design? • SHARPEN the “focus” variables and minimizing the BLURRING of “nuisance” variables

33. What is the Goal of an Experimental Design? • Focus variable: • The few variables whose effects you are interested in • This is, in fact, the point of the experiment! • AKA Treatment variable • e.g. the labeling of actions and the severity of punishment

34. What is the Goal of an Experimental Design? • Nuisance variable: • Other variables that are of no direct interest, but may affect your results • Types: • Controllable (e.g. sex, age, education, income, etc.) • Uncontrollable (e.g. subject’s interest, alertness, amount of fatigue)

35. I. Direct Experimental Control Constants and Treatments

36. How do we Sharpen Focus Variables? • Focus variables are controlled for at 2 or more different levels • Need to vary all treatment variables independently to obtain the clearest possible effects • Need to ensure all possible explanations for our outcome of interest (i.e. ability to coordinate) are covered

37. How do we Sharpen Focus Variables? Confounded Treatments Independent Treatments If we notice a difference in behaviour for the treatments we have observations for, it is impossible to know whether it was the labelling or the punishment that caused it.

38. II. Indirect Experimental Control Randomization

39. How do we Blur Nuisance Variables? • Uncontrollable nuisance variables can cause inferential errors if they are confounded with focus variables • A variable is confounded if it is correlated with both the outcome variable and the treatment variables • Independence among controlled variables prevents some confounding problems • Need to ensure that any subject does not have a biased opportunity to be in a particular experimental session based on some controlled variable (e.g. sex)

40. How do we Blur Nuisance Variables? • Randomization provides indirect control of uncontrolled nuisance variables by ensuring their independence of treatment variables • EXAMPLE: Role Assignment by Order of Attendance • Subjects’ personal idiosyncrasies and habits are uncontrollable • Don’t assign early birds to one role and late comers to the other role • Randomizing roles based on order of attendance ensures differences between players is due to their roles, not due to differences in subjects’ personal characteristics

41. How do we Randomize? • Types of randomization techniques: • Completely Randomized • Each treatment is equally likely to be assigned in each period of an experimental trial • Quite effective when you can afford to run many periods • Independence is established after many periods • This can be improved upon (i.e. fewer periods) with the appropriate combination of control and randomization

42. How do we Randomize? • Completely Randomized EXAMPLE

43. How do we Randomize? 2. Random Blocks • Difference from completely randomized design is that 1 or more nuisances are controlled as treatments rather than randomized • Between Subjects • Treatments are only varied across subjects • Subjects only receive 1 treatment • Our original Battle of the Sexes class experiment used this design • Within Subjects • Treatments are varied for each subject • In other words, every single subject is exposed to every single treatment • Subjects receive each treatment in a random order

44. How do we Randomize? • Between Subjects • ADVANTAGES • Avoids carryover effects common in Within Subject Design • Lowers the chances of subjects suffering boredom after a long series of tests • Lowers the chances of subjects becoming more accomplished through practice and experience, and thus skewing the results

45. How do we Randomize? • Between Subjects • DISADVANTAGES • Practicality • Requires a large number of subjects to generate useful data since subjects are only exposed to 1 treatment • Individual variability & Assignment bias • Since subjects are only part of 1 group, it is difficult to control for all possible individual differences • Environmental factors • Usually arise from poor experimental design • Suppose, for time reasons, you test one group in the morning and one in the afternoon • Many studies show that most people are at their mental peak in the morning, so this will certainly have created an environmental bias

46. How do we Randomize? ii. Within Subjects • ADVANTAGES • This gives as many data sets as there are conditions for each subject • Requires far fewer subjects than Between Subjects Design • Provides a way of reducing the amount of error arising from natural variance between individuals

47. How do we Randomize? ii. Within Subjects • DISADVANTAGES • Carryover effects where the first treatment adversely influences the others • e.g. Fatigue and Practice • In a long experiment, with multiple conditions, the participants may be tired and thoroughly fed up of researchers prying and asking questions and pressuring them into taking tests. • This could decrease their performance on the last study.

48. How do we Randomize? 3. Crossover Design • Variation of Within Subject Design • Used when you suspect your treatment variables have carryover effects (i.e. effects that last for some time)

49. How do we Randomize? 3. Crossover Design EXAMPLE • Back to the Battle of Sexes: to punish or not • Suppose we are concerned that being in the “punishment” treatment (P) first will affect behaviour in the subsequent “no punishment” treatment (N) • Simple Design: NP and PN • This design confounds time and learning with the treatment variables • Crossover Design: NPN and PNP • Using this design, the difference in the average outcome for P and N indicates the effect of your focus variables

50. How do we Randomize? 4.Factorial Design • Most important general method when you have 2 or more treatment variables • More efficient than completely randomized design