EXPERIMENTS

1. EXPERIMENTS © Pine Forge Press, an imprint of Sage Publications, 2004

2. Different Types ofExperimental Design • true experiments • quasi-experiments • evaluation research • nonexperimental designs © Pine Forge Press, an imprint of Sage Publications, 2004

3. Example of non-experimental design [before-after/pretest-posttest] The Imaginary Seattle Bike Patrol Study

4. Pretest Posttest O X O Look at change in crime rates Subtract pretest score from posttest

5. O X O Could have been; Change in Economy Change in weather Change in media coverage etc. Instead of X, or combined with X

6. “Confounding” • Combining a second variable with the independent variable is Confounding

7. External events • Such additional variables in the research situation which provide an alternative explanation to the one that X is changing Y, in this case are classified as “external events” because they are occurring outside the study at the same time as the independent variable is occurring.

8. HISTORY • Eternal events • HISTORY - ENVIRONMENTAL CHANGES …all events that occurred during the time of the study that might affect the individuals studied and provide a rival explanation for the change in the dependent variable…all events between the pretest [if given] and the posttest… • Changes in the economy • Changes in the weather • Changes in information provided by the media • War • Natural disaster • Etc.

9. Exogenous events • Note: “history” variables are also known as “exogenous” events. This means that the events are “outside” the study.

10. How can we ensure that the change was due to X and not one or more of the external events?... “History”

11. Classical experimental design: Pretest Bike patrol Posttest • RA • Pretest No bike patrol Posttest

12. We have randomly selected a bunch of cities for the E group and the C group…then… • We compare changes in each group.

13. Simultaneous variation Pretest Bike patrol Posttest Change in economy Change in weather Media coverage Etc. • RA Pretest no bike patrol Posttest change in Economy Change in weather Media coverage Etc

14. Simultaneous variation • Simultaneous variation allows for equal changes in all other variables in the experimental group and control group. • Since randomization would equalize [out] the effects in both groups, the only reason for different results in the experimental and control group should be due to X. • S.V. allows for changes to occur in each group yet it “CONTROLS” for effects of other variables.

15. N.B. • Be able to explain how simultaneous variation and random assignment do their work to “control.”

16. Review • History is controlled by random assignment and allowing the History Variables to make changes in Y equally in both the experimental group and the control group. Hence the only difference in changes in the two groups can be attributed to X. • If this doesn’t make immediate sense ask now, work on it, and get help!

17. True Experiments True experiments must have at least one experimental group (subjects who receive some treatment) and at least one comparison group (subjects to whom the experimental group can be compared). 01 01 © Pine Forge Press, an imprint of Sage Publications, 2004

18. True Experiments True experiments must have at least three things: • Two comparison groups (in the simplest case, an experimental and a control group) • Variation in the independent variable before assessment of change in the dependent variable • Random assignment to the two (or more) comparison groups © Pine Forge Press, an imprint of Sage Publications, 2004

19. True Experiments True experiments must have at least one experimental group (subjects who receive some treatment) and at least one comparison group (subjects to whom the experimental group can be compared). 01 01 © Pine Forge Press, an imprint of Sage Publications, 2004

20. True Experiments All true experiments have a posttest—that is, measurement of the outcome in both groups after the experimental group has received the treatment. Many true experiments also have pretests that measure the dependent variable prior to the experimental intervention. A pretest is exactly the same as a posttest, just administered at a different time. 01 01 © Pine Forge Press, an imprint of Sage Publications, 2004

21. INTERNAL VALIDITY • The issue of X being responsible for the changes to Y is an issue of INTERNAL VALIDITY. • Since we ruled out all the outside factors, history factors, external events which could alternatively explain why the bike patrols worked, we may tentatively assume that the study indicates internal validity. • Why tentatively? RA & INTERACTION

22. Tentativeness • RA…random assignment doesn’t always truly equalize the E group and the C group. • Interaction allows for the possibility that bike patrols work sometimes and not others and we may not “uncover” a hidden relationship that work in different directions.

23. RA doesn’t always equalize E & C group in values of Y. • Or, Interaction may mask actual relationship: • Z1 O1 O2 • Z2 O1 O2 • X1 X2

24. Internal Validity • The 5 subtypes of internal validity examined here are: • 1. External [exogenous outside] events / History • 2. Endogenous inside events / Maturation, Instrumentation, Testing, Regression • 3. Selection bias / Selection Mortality • 4. Treatment misidentification / Reactivity, Experimenter Bias, Demoralization, Compensation, Placebo effect • 5. Contamination / Experimental Diffusion, Contamination

25. Causal (Internal) Validity There are four basic sources of noncomparability (other than the treatment) between a comparison group and an experimental group. They produce four of the five sources of internal invalidity: • When characteristics of the experimental and comparison group subjects differ SELECTION BIAS • When the subjects develop or change during the experiment as part of an ongoing process independent of the experimental treatment ENDOGENOUS EVENTS. • When something occurs during the experiment, other than the treatment, which influences outcome scores HISTORY. • When either the experimental group or the comparison group is aware of the other group and is influenced in the posttest as a result (Mohr, 1992)CONTAMINATION.

26. Causal (Internal) Validity The fifth source of internal invalidity can be termed TREATMENT MISIDENTIFICATION: Variation in the independent variable (the treatment) is associated with variation in the observed outcome, but the change occurs through a process that the researcher has not identified. © Pine Forge Press, an imprint of Sage Publications, 2004

27. Review • We have done history and found that it is controlled by • E • RA • C • SIMULTANEOUS VARIATION

28. Endogenous Change • The next set of variables which may interfere with internal validity are exogenous variables. “When the subjects develop or change during the experiment as part of an ongoing process independent of the experimental treatment.” These include: Maturation, Instrumentation, Testing, & Regression artifact.

29. Maturation • The bike patrol period was the beginning of a baby bustlet a small baby bust for folks growing into their teen age years. Fewer teenagers…probably fewer crimes. Folks grow out of lots of crimes when they get older, marry, get decent jobs etc.

30. Maturation • MATURATION – GROWING OLDER, WISER, TIREDER, DEVELOPMENTAL CHANGES…biological, psychological, or social processes that produce changes in the individuals or units studied with the passage of time that are not produced by the independent variable(s) under study in the experiment. • Changes in the age distribution of the population • Changes in the interest in participating in the dependent variable [getting tired of it]

31. How is it controlled? E O X O • RA Bike patrol Age changes • C O O • Age changes • SIMULTANEOUS VARIATION

32. INSTRUMENTATION • Police could have learned to be more vigilant…thus may have caught more crooks. • Alternatively, police may have grown tired and thus been less vigilant.

33. INSTRUMENTATION • INSTRUMENTATION – INSTRUMENT “DECAY” OR IMPROVEMENT…changes in the measuring instrument between the pretest and posttest. • Note changing springs on a scale… • Note changing experience of interviewers… • Note changing ability of coders…

34. How to control? • O X O • Bike patrol • Police Reporting changes • RA • O O • Police Reporting changes • SIMULTANEOUS VARIATION

35. TESTING • The study of crime may have made the potential victims more careful or may have made the criminals more careful and thus less likely to be caught.

36. TESTING • TESTING – PRACTICE OR AWARENESS CHANGES THE SUBJECT…the possible reactivity of measurement …testing itself may change the phenomenon being measured…through awareness & reflection, through experience & practice…

37. How to control? • E X • & changes in testing • RA • C changes in testing • SIMULTANEOUS VARIATION

38. REGRESSION ATRIFACT • Crime scores were unreliably measured at the top & the only way they could go was down as they varied on second test.

39. 300 [297] 300 [400] 300 [295] 300 [298] 300 [296] ____ 1500 1500/5=300 292 300 293 294 300 ____ 1479 1479/5=295.8 300-295.8=4.2 Example using unreliable scale“Extreme” scores can only go down.

40. REGRESSION ATRIFACT • REGRESSION ATRIFACT – UNRELIABLE MEASURES OF GROPUS WITH EXTREME SCORES; THERE’S ONLY ONE WAY TO GO…if only extreme cases are selected, and if the measurement instrument is unreliable, when remeasuring the only way unreliably low scores can go is up and …high scores…down

41. How to control for “regression artifact” • RA and simultaneous variation • Sam Ting w.r.t. changes due to unreliable measures with only one way to go will happen in the E group and the C group

42. Review • All endogenous changes involve changes within the experiment other than X. They are controlled in the same way. SAM TING! • They are controlled by SIMULTANEOUS VARIATION – changing equally in the E group and the C group. RA should place the same kinds of cases in both groups so that change [aside from X] is “equalized/” • Hence SIMULTANEOUS VARIATION. • O X O • Other stuff • RA • O Other stuff O

43. Selection Bias

44. e.g.1 Seattle may have already had a “thing” about crime such that the community was going to reduce it and the police on bikes was just a temporal coincidence. • Seattle was already hell bent on changing crime and so the selection of bikes coincided with an already established tendency. • So Seattle has “selected” to make the change apart from the bike patrol.

45. e.g.2 the film may not be the cause of change when a survey is taken on film effects. Filmgoers may have been more liberal before they saw the film.

46. No X = not going to film Non liberal folks avoid film It isn’t film exposure . Film exposure doesn’t change people. X = film Liberal folks go to such a film It’s the initial liberalism of the audience Confounding by selection

47. How to control for selection • See above • Sam Ting • RA will place the same kinds of cases in the E group and the C group. Folks won’t select themselves.

48. INTERACTIONS WITH SELECTION – SUBJECT SELECTION - FOLKS CHOOSE WHICH GROUP THEY’RE IN SO GROUPS AREN’T REALLY EQUAL subject may be selected with a predilection to change or may appear to change because of how they have selected themselves…

49. EXPERIMENTAL MORTALITY • Perhaps all the crooks were jailed …thus there was a smaller population of criminals committing crime at time 2.

50. EXPERIMENTAL MORTALITY - “DYING” / DIFERENTIAL ATTRITION / DROPPING OUT OF THE EXPERIMENTAL OR CONTROL GROUP…dropout problems that prevent the researcher from obtaining information on all cases.