Conducting Experiments, Survey Construction and Data Collection

Conducting Experiments, Survey Construction and Data Collection RCS 6740 7/18/05

Practical Considerations of Research • This lecture will focus on some of the practical aspects of conducting research studies such as: • How do you select research participants • What should you consider when deciding how to manipulate an independent variable • How do you measure a variable • What should you consider when constructing a survey • What should you consider when collecting data • How do ensure control in your study

Selecting Research Participants • Define your participants • People with Aphasia • Women with Breast Cancer • Rats with 5 Legs • Define your population • Everyone in the United States • People in RCS 6740 • Consumers at “The Center” (CIL)

Selecting Research Participants Cont. • Define your sampling technique • Probability sampling • Random sampling • Stratified random sampling • Systematic sampling • Cluster sampling • Non-probability sampling • Quota sampling • Determine your sample size • Remember, larger sample sizes may result in more significant data

Manipulating the Independent Variable • Construct the Operational Definition of your variables • Exercise as defined by jogging for 20 minutes, 3 times a week, for 6 weeks • Depression as defined by the Geriatric Depression Inventory • Set the Stage • Provide the participants with Informed Consent • Usually a document that entails the name of the study, purpose(s) of the study, potential benefits and risks, funding sources, contact information, and a statement that they can withdraw at any time • Explain to the participants in more detail why the study is being conducted • Rarely are hypotheses given to participants • Tell them the study looks at employment when the hypothesis looks specifically at bias towards the employment of people with disabilities • Can be completely truthful and precise • Deception may be used • When deception is used, it is imperative that participants are told the truth during a debriefing at the conclusion of the study

Types of Manipulations • Straightforward Manipulations • Presenting written, verbal, or visual material to participants • Examples: • Having participants read material on substance abuse prevention • Read a passage to a participant and ask them questions later to test recall (memory) • Have a computer program that presents different words that someone needs to pronounce to measure the level of aphasia • Most research is done in a straightforward manner • Researchers can vary the difficulty of material, motivational levels, the way questions are asked, or the characteristics of the participants

Types of Manipulations Cont. • Staged Manipulations • Also called Event Manipulations • Used for 2 reasons • The researcher may need to create some sort of psychological state (anxiety) • The researcher may need to stage a manipulation to recreate a real-world scenario • Having a participant do one task and then having them do more tasks at the same time • Staged manipulations usually employ a confederate • Also called an accomplice • A confederate is someone who appears to be another particpant in an experiment but is really a part of the experiment • Example: Someone who purposely insults a participant in a study in order to provoke anger or frustration

Types of Manipulations Cont. • Another example of the use of confederates: • Asch (1956) study on conformity • Which line is bigger? • 1)--------------------- • 2)----------------------------- • 3)------------------- • Right before a participant had to choose which line was the longest, a confederate announced an incorrect answer • Repeatedly, Asch found that people conformed to the confederate’s incorrect response

Types of Manipulations Cont. • Straightforward Manipulations Vs. Staged Manipulations: • It is easier to recreate real-world situations using Staged manipulations • Staged manipulations are more extensive in terms of cost and set up procedures • It is difficult for other researchers to replicate staged manipulations • Data and results of Straightforward manipulations are easier to interpret due to a greater sense of control

Strength of Manipulations • The strength of a manipulation can effect how results are interpreted • Example: In a study on the use of a new medication to treat diabetes, it is important that the researcher uses a sufficient dosage of the drug in order to see an effect • 5 mgs of Diabetex does not reduce blood sugar levels • 10 mgs of Diabetex reduces blood sugar levels by 3% • 20 mgs of Diabetex reduces blood sugar levels by 10%

Strength of Manipulations Cont. • A strong manipulation maximizes the differences between groups and increases the chance that the independent variable will have a significant effect on the dependent variable • Strong manipulations are especially important in the early stages of research (Why?) • Although you want a manipulation to be as strong as possible, ethical considerations should be examined • You do not want to make someone so anxious that they decide to jump off a building

Cost of Manipulations • Most researchers are limited in terms of the funding they have for a research project • Confederates and elaborate manipulations may cost a great deal • The straightforward approach is usually more budget friendly

Measuring the Dependent Variable Types of Measures • Self-report measures • Behavioral Measures • Physiological Measures

Measuring the Dependent Variable Cont. Self-report Measures • Used to measure attitudes, judgments about oneself or another, emotional states, confidence in oneself or others, and other aspects of human thought and behavior • Example: Please use the following 7 point scale to answer this question. I feel that I am the best student in this class: Strongly Disagree _ _ _ _ _ _ _ Strongly Agree

Measuring the Dependent Variable Cont. Behavioral Measures • Direct observations of behaviors such as: • How many errors someone makes on a test • How many times a person can lift a 10lb object • How may free throws a person makes in 30 minutes • Things to consider with behavioral measures • Rate: How many times a behavior occurs • Reaction Time: How quickly a response occurs after a stimulus • Duration: How long a behavior lasts

Measuring the Dependent Variable Cont. Physiological Measures • Recordings of responses of the body • Galvanic Skin Response (GSR): Measures the electrical conductance of the skin to look for arousal or anxiety • Electromyogram (EMG): Measures muscle tension to look for stress • Electroencephalogram (EEG): Measures activity of brain cells such as looking at what area of the brain is active while dreaming • Magnetic Resonance Imaging (MRI): A technique used to gather detailed images of the inner body • Functional Magnetic Resonance Imaging (fMRI): Looking at the inner body while someone performs a physical or a cognitive task

Measuring the Dependent Variable Cont. What Type of Measure(s) to use? • Basically, what type(s) of measure a researcher will use depends on the nature of the study, cost, and possible ethical considerations • If at all possible a researcher should use a combination of all three types: • To study anxiety, a researcher can administer a self-report measure asking how nervous a person thinks they are, a behavioral measure that looks at how many times a person’s voice cracks, and a physiological measure looking at fluctuations in someone’s heart rate • When administering multiple measures, it is important to remember to account for a possible order effect • Example: A self-report measure on how nervous a person is might cause them to become more nervous thus increasing their heart rate

Measuring the Dependent Variable Cont. Sensitivity of the Dependent Variable • The dependent variable should be sensitive enough to detect differences • Do you like your boss (not sensitive) • On a scale of 1 to 10, how much do you like your boss (very sensitive) • Beware of the Ceiling Effect and the Floor Effect • Ceiling Effect: Task is too easy so everyone performs well • Lifting a 1lb object • Floor Effect: Task is too difficult so everyone performs poorly • Lifting a 200lb object

Information on Survey Construction • DESIGNING SURVEYS • A good question is one that produces answers that are reliable and valid measures of something that we want to describe • 2 types of question evaluations: • Those aimed at evaluating reliability • Those aimed at assessing the validity of answers

Survey Construction Cont. • 5 process standards of questions and answers • Questions need to be consistently understood. • Questions need to be consistently administered or communicated to respondents. • What constitutes an adequate answer should be consistently communicated. • Unless measuring knowledge is the goal of the question, all respondents should have access to the information needed to answer the question accurately. • Respondents must be willing to provide the answers called for in the question.

Survey Construction Cont. • Possible steps to assess the extent to which questions meet process standards • Focus group discussions • Intensive or cognitive interviews • Field pre-tests under realistic conditions

Survey Construction Cont. • Some general rules for designing good survey instruments • The strength of survey research is asking people about their firsthand experiences: what they have done, their current situations, their feelings and perceptions. • Questions should be asked one at a time. • A survey question should be worded so that all respondents are answering the same question. • If a survey is to be interviewer administered, wording of the questions must constitute a complete and adequate script such that when the interviewer reads the question as worded, the respondent will be fairly prepared to answer the question. • All respondents should understand the kind of answer that constitutes an adequate answer to a question. • Survey instruments should be designed so that the tasks of reading questions, following instructions, and recording answers are as easy as possible for interviewers and respondents.

Survey Construction Cont. • Ways of addressing validity in surveys • Deriving questions from relevant literature • Expert panel

Data Collection Tips • Use more than one survey or measure • Come prepared • Extra measures, pens, equipment • Be prepared to answer questions about the survey • Important not to sway a response • Follow up to ensure all items were filled out • (999=missing, 888=don’t know) • Know your survey • How to score it, the factors, the subscales • If possible, have a limited amount of researchers collect the data • Enter data as quickly as possible and use a trusting computer program (SPSS, ECXEL) • Have two people enter the data (one reads, one enters)

Control Considerations Participant Expectations • When a participant knows too much about a study, they may expect to perform in a certain manner • Demand Characteristics: any feature of an experiment that might inform participants of the true purpose of a study • Once someone is aware of the hypothesis of a study, the usually do whatever it takes to confirm the hypothesis • Hypothesis: People who take a drug will lessen their anxiety levels. Once a participant knows this, they may choose to do whatever they can do become less anxious (e.g., exercise, see a counselor, meditate).

Control Considerations Cont. Ways to Control Participant Expectations • Use deception to cover up what is really being studied • Administer irrelevant measures • Add filler questions to measures • Filler questions are unrelated questions on a survey that should disguise the true purpose of a study • Include a Placebo group in a study • Placebo effect: When it is not know if results of a treatment can be attributed to the treatment itself or the expectancies of the treatment • Placebos may consist of giving someone a pill that is made of sugar or injecting someone with saline solution • It is important to remember ethical consideration of using a placebo group. For example, if someone in the placebo group becomes very ill, you may want to administer treatment to them

Control Considerations Cont. Experimenter Expectations • Sometimes experimenters who know the purpose of a study may expect a participant to act or perform a certain way • Also known as expectancy bias • There are 2 potential sources of experimenter bias • When a researcher treats participants in the control group differently than participants in the treatment group • Spending more time with the treatment group • Researchers may record or analyze observations in a biased manner • Well, even though he spoke 10 words clearly, I thought he was close on another so I will give him 11/20

Control Considerations Cont. Ways to Control Experimenter Expectations • Researchers should be well trained and highly ethical • Run all conditions simultaneously so the experimenter’s behavior is the same for all participants • Use experimenters who are unaware of the true hypothesis of the study • Double-Blind Study: Neither the participant nor the experimenter knows whether a placebo or the actual treatment is given Once again, it is highly unethical for experimenters to influence participants or manipulate data!!!

Additional Tips for Conducting Experiments • Once a study is designed, there are certain ways to improve it • Write up a thorough Research Proposal • Includes a literature review, purpose of the study, procedures of the study, copies of measures, a budget, and the manner with which the data will be analyzed • Used often for grants, Institutional Review Boards, and Ethics Committees • Conduct a Pilot Study • A trial run of a study with small sample sizes • Used to “get the kinks” out • Used to gather participant feedback

Additional Considerations of Experiments Cont. • Manipulation Checks • A manipulation check is a way to measure if the independent variable has an effect on the dependent variable • Checks if the independent variable manipulation is a successful operationalization of the conceptual variable being studied • Provides evidence for Construct Validity • Allows researchers to change something in a study before it is too late • Allows researchers to see if they are getting significant results • Example: In a study looking at whether people are nicer to attractive people, a researcher will determine whether the person who they chose to be attractive really is being viewed as so by the participants. If they are not viewed as attractive, the researchers can hire another model. • Manipulation Checks may consist of any type of measure (i.e., self-report, behavioral, physiological)

Additional Considerations of Experiments Cont. • Debriefing • Debriefing is the act of meeting with the participants after the conclusion of the study to discuss the true purpose of the study, ethical considerations of the study, and educational implications of the study • Can also be used to ensure that the participants do not share any information with those still involved in the study Can also be beneficial as researchers may gain valuable information about how the study looks and feels from the “inside”

Analyzing Data • After a quantitative study has concluded, all data should be analyzed statistically

Communicating Research to Others • Once a researcher has finished a study, they should communicate their results and findings to others • Professional Meetings and Conferences (NRA) that present the opportunity to share results in a verbal, written, or poster format • Journal Articles also are a good way to share results • Peer Review is the process of a paper being accepted or declined entry into a journal based on reviews from peers • About 90% of papers submitted to the more prestigious journals are rejected  but do not let this get you down 

Conducting Experiments, Survey Construction and Data Collection