Basics of Designing Experiments

1. Basics of Designing Experiments Thursday, October 24, 2013 5:00pm - 7:00pm GLC Room G

2. About Me • Graduate student in Virginia Tech Department of Statistics • Enrolled in Master’s program • Expected Graduation Date: December 2013 • Future: Job in industry • Lead Collaborator in LISA • On-campus consulting group, led by Dr. Eric Vance and Dr. Chris Franck, with administrative specialist Tonya Pruitt

3. About • What? • Laboratory for Interdisciplinary Statistical Analysis • Why? • Mission: to provide statistical advice, analysis, and education to Virginia Tech researchers • How? • Collaboration requests, Walk-in Consulting, Short Courses • Where? • Walk-in Consulting in GLC and various other locations • Collaboration meetings typically held in Sandy 312 • Who? • Graduate students and faculty members in VT statistics department

4. Requesting a LISA Meeting • Go to www.lisa.stat.vt.edu • Click link for “Collaboration Request Form” • Sign into the website using VT PID and password • Enter your information (email, college, etc.) • Describe your project (project title, research goals, specific research questions, if you have already collected data, special requests, etc.) • Contact assigned LISA collaborators as soon as possible to schedule a meeting

5. Agenda • Introduction to Designing Experiments • 3 Main Principles • Randomization • Replication • Blocking (Local Control of Error) • EX: Nozzle design and water jet performance • EX: Treatment and leukemia cell gene expression • Factorial experiments

6. INTRODUCTIONto Experimental Design

7. Why is Experimental Design important? • MAXIMIZE… • Probability of having a successful experiment • Information gain from results of an experiment • MINIMIZE… • Unwanted effects from other sources of variation • Cost of experiment if resources are limited

8. Experiment vs. Observational • OBSERVATIONAL STUDY • Researcher observes the response of interest under natural conditions • EX: Surveys, weather patterns • EXPERIMENT • Researcher controls variables that have a potential effect on the response of interest Which one helps establish cause-and-effect relationships better?

9. Correlation ≠ Causation

10. EXAMPLE: Impact of Exercise Intensity on Resting Heart Rate • Researcher surveys a sample of individuals to glean information about their intensity of exercise each week and their resting heart rate • What type of study is this?

11. EXAMPLE: Impact of Exercise Intensity on Resting Heart Rate • Researcher finds a sample of individuals, enrolls groups in exercise programs of different intensity levels, and then measures before/after heart rates

12. EXAMPLE: Impact of Exercise Intensity on Resting Heart Rate • What are some factors the experimental study can account for that the observational study cannot?

13. Sources of variation • Sources of variationare anything that could cause an observation to be different from another observation • What are some reasons that measurements of resting heart rate could differ from person to person?

14. Sources of variation • There are two main types: • Gender and age are what are known as nuisance factors: we are not interested in their effects on RHR, but they are hard to control • What we are interested in is the effect of the intensity of exercise: this source is known as a treatment factor

15. Sources of variation • Good rule of thumb: list major and minor sources of variation before collecting data • We want our design to minimize the impact of minor sources of variation, and to be able to separate effects of nuisance factors from treatment factors • We want the majority of the variability of the data to be explained by the treatment factors

16. Designing the experiment: The Bare Minimum • Response: Resting heart rate (beats per minute) • Treatment: Exercise Program • Low intensity • Moderate Intensity • High Intensity

17. Designing the experiment: The Bare Minimum • Some assumptions • We will be monitoring the participants’ diet and exercise throughout the study (not relying on self-reporting) • We will only enroll participants with high (i.e. unhealthy) resting heart rates so that there is ample room for improvement • Participants’ resting heart rate is all measured in the same manner, at the same time (upon waking up)

18. Designing the experiment: The Bare Minimum • Basic Design • 36 participants: 24 males, 12 females • Every person is assigned to one of the three 8-week exercise programs • Resting heart rate is measured at the beginning and end of the 8 weeks What other considerations should we make in designing the experiment?

19. THREE BASIC PRINCIPLES OF DOE: Randomization

20. Randomization • What? • Random assignment of experimental treatments and order of runs • Why? • Often we assume an independent, random distribution of observations and errors – randomization validates this assumption • Averages out the effects of extraneous/lurking variables • Reduces bias and accusations of bias • How? • Depends on the type of experiment

21. Exercise Example • 36 participants are randomly assigned to one of the three programs • 12 in low intensity, 12 in moderate intensity, 12 in high intensity • Like drawing names from a hat to fall into each group • Oftentimes computer programs can randomize participants for an experiment

22. Exercise Example • What if we did not randomize? • Suppose there is some reason behind who comes to volunteer for the study first versus later • If we assigned first third to one intensity, second third to another, and so forth, it would be hard to separate the effects of the “early volunteers” and their assigned intensity level

23. Completely Randomized Design (CRD) • What we just came up with is called a completely randomized design • Note that in our case, treatments were assigned randomly, but in some experiments where there are a sequence of runs performed, the order of runs need to be randomized as well

24. Summary • Randomizing the assignment of treatments and/or order of runs accounts for known and unknown differences between subjects • It does not matter if what occurs does not “looks random” (i.e. appears to have some pattern), as long as the order was generated using a proper randomization device

25. THREE BASIC PRINCIPLES OF DOE: Replication

26. Replication • What? • Independent repeat runs of each treatment • Why? • Improves precision of effect estimation • Allows for estimation of error variation and background noise • Check against aberrant results that could result in misleading conclusions • EX: One person for each treatment. What could go wrong?

27. Experimental Units (EUs) • We now introduce the term “Experimental Unit” (EU) • EU is the “material” to which treatment factors are assigned • In our case, each person is an EU • This is different from an “Observational Unit” (OU) • OU is part of an EU that is measured • Multiple OUs within an EU here would be if we took each person’s pulse at his/her neck, at the wrist, etc. and reported these observations

28. Replication Extension to EU • Thus, a treatment is only replicated if it is assigned to a new EU • Taking multiple observations on one EU (i.e. creating more OUs) does not count as replication – this is known as subsampling • Note that treating subsampling as replicating increases the chance of incorrect conclusions (psuedoreplication) • Variability in multiple measurements is measurement error, rather than experimental error

29. Consequences of Pseudoreplication • Is it bad to take multiple OUs on each EU then? • No, often the solution here is to average the measurements of from the OUs and treat it as one observation • What if we don’t do this? • We severely underestimate error • We potentially overexaggerate the true treatment differences • What if measurement error is high? • Try to improve measurement process • Revisit the experiment and assess the homogeneity of the EUs, thinking of potential covariates

30. Exercise Example • Use formula: • 36 participants, 3 treatments •  36/3 = 12 replications per treatment in the balanced case • The balanced case is preferred because: • Power of test to detect a significant effect of our treatment on the response is maximized with equal sample size

31. Exercise Example • Unbalanced consequences? • Suppose the following: • This would lead to better estimation of the high intensity treatment over the other two • Thus if you have equal interest in estimating the treatments, try to equally replicate the number of treatment assignments

32. Summary • The number of replications is the number of experimental units to which a treatment is assigned • Replicating in an experiment helps us decrease variance and increase precision in estimating treatment effects

33. THREE BASIC PRINCIPLES OF DOE: Blocking (or Local Control of Error)

34. Local Control of Error • What? • Any means of improving accuracy of measuring treatment effects in design • Why? • Removes sources of nuisance experimental variability • Improves precision with which comparisons among factors are made • How? • Often through use of blocking (or ANCOVA)

35. Blocking • What? • A block is a set of relatively homogeneous experimental conditions • EX: block on time, proximity of experimental units, or characteristics of experimental units • How? • Separate randomizations for each block • Account for differences in blocks and then compare the treatments

36. Exercise Example • Block on gender? • This assumes that males and females have different responses to exercise intensity • Would have the following (balanced) design: • Here, after the participants are blocked into male/female groups, they are then randomly assigned into one of three treatment conditions

37. Exercise Example • Block on age? • This assumes that age may influence the effect exercise intensity has on resting heart rate • Would have the following (balanced) design: • Here, after the participants are blocked into respective age groups, they are then randomly assigned into one of three treatment conditions

38. Randomized Complete Block Design (RCBD) • This design is called Generalized RCBD • Generalized merely means there are replications involved • Here, each treatment appears in each block an equal number of times • Benefits of RCBD • We can compare the performance of the three treatments (exercise programs) • We can account for the variability in gender that might otherwise obscure the treatment effects

39. Summary • Blocking is separating EUs into groups with similar characteristics • It allows us to remove a source of nuisance variability, and increase our ability to detect treatment differences • Randomization is conducted within each block Note that we cannot make causal inferences about blocks– only treatment effects!

40. EXAMPLE: Gene Expression in Leukemia Cells

41. Leukemia Cells Background • Suppose we are interested in how different treatment groups affect gene expression in human leukemia cells • There are three treatment groups: • MP only • MP with low dose MTX • MP with high dose MTX • Each treatment group has 10 obs What type of design is this?

42. CRD Assumptions and Background • The simplest design assumes that all the EUs are similar and the only major source of variation is the treatments • Recall: A CRD randomizes all treatment-EU assignments for the specified number of treatment replications • Recall: We want to aim to have a balanced experiment, i.e. equal replications of each treatment

43. Leukemia Cells • As before, we want to randomize which subjects receive which of the three treatments • The data looks as follows:

44. Leukemia Cells – Pre randomization These EUs should be similar MP only MP + LDMTX MP + HDMTX

45. Leukemia Cells – Post randomization

46. Leukemia Cells in JMP • We want to enter the data such that each response has its own row, with the corresponding treatment type • We then choose Analyze  Fit Y by X

47. Leukemia Cells in JMP • Choose “GeneExp” for Y, Response • Choose “Treatment” for X, factor

48. Leukemia Cells Visual Analysis • General comments • Treatment 3 has a smaller spread of data than the other two • Treatment 2 has the highest average “gene expression”, followed by Treatment 1, then Treatment 3 • Are the differences substantial? What do you see from this graph (to the left) here?

49. Leukemia Cells Summary of Fit • In more technical terms, it is the percent of variation in response (gene expression) that can be explained by our predictor (treatment group). Based on this first glance at the summary of fit, what would you conclude? • R-square is a measure of fit. • If it is close to 1, a good model is indicated. • If it is close to 0, a poor model is indicated

50. Leukemia Cells ANOVA • Null hypothesis: The treatments have the same means • Test: Is there at least one treatment effect that is different from the rest? SStotal=SStrt + SSError Variance of observations from their respective treatment means Variance of all observations from the mean of all the data Variance of treatment means from overall mean