1 / 44

Sampling Demystified

Sampling Demystified. Research Methods for Public Administrators Dr. Gail Johnson. Steps in the Research Process. Planning 1. Determining Your Questions 2. Identifying Your Measures and Measurement Strategy 3. Selecting a Research Design 4. Developing Your Data Collection Strategy

thayes
Download Presentation

Sampling Demystified

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Sampling Demystified Research Methods for Public Administrators Dr. Gail Johnson Dr. G. Johnson, www.ResearchDemystified.org

  2. Steps in the Research Process Planning 1. Determining Your Questions 2. Identifying Your Measures and Measurement Strategy 3. Selecting a Research Design 4. Developing Your Data Collection Strategy • Developing the Sampling Strategy 5. Identifying Your Analysis Strategy 6. Reviewing and Testing Your Plan Dr. G. Johnson, www.ResearchDemystified.org 2

  3. Why Sample? • Sometimes it is possible to gather data from every file, every street, every person in the population of interest. • When you can work with the entire population of files, streets, or people—we call it a census. • Everyone is included because everyone counts Dr. G. Johnson, www.ResearchDemystified.org

  4. Why Sample? • When you have the resources to gather data from the entire population: that’s the gold standard. • But—the world is not often organized in a way that makes this easy • Often there is limited time, staff and money to gather data from the entire population. • So researchers use samples Dr. G. Johnson, www.ResearchDemystified.org

  5. Why Sample? It requires a basic faith that the sample is a fairly accurate reflection of the population Think about it: Doctors don’t have to take all our blood in order to analyze it. They take a sample. Dr. G. Johnson, www.ResearchDemystified.org

  6. Two Big Sampling Options • Nonrandom Samples • Quota • Accidental • Snow-ball • Judgmental • Convenience • Random Samples • Based on probability theory where: every item (person, transaction, street, house, whatever) has an equal chance of being selected Dr. G. Johnson, www.ResearchDemystified.org

  7. Option: Nonrandom Sampling • Used when it is not possible or desirable to do a random sample • Can be more focused: specifically chosen • Select 6 executive directors: 3 men and 3 women • Select all the transactions from the busiest day of the month • Can observe traffic through town during rush hour (4:30-6pm Monday through Friday). Dr. G. Johnson, www.ResearchDemystified.org

  8. Types of Nonrandom Samples • Quota • Set a specific number: we will interview 10 men and 10 women at the health clinic • We will call 50 Democrats and 50 Republicans • First come basis and once quota is met, we stop • If we reach 50 Republicans, we then continue to dial for Democrats until we obtain the 50 quota. Dr. G. Johnson, www.ResearchDemystified.org

  9. Types of Nonrandom Samples • Accidental—person on the street: think “Jay-walking” • While chaotic, it does not meet the definition of random • If I survey people outside the big box store on a Saturday morning, not everyone in the community will have had an equal chance of being selected Dr. G. Johnson, www.ResearchDemystified.org

  10. Types of Nonrandom Samples • Snow-ball • This is useful when researchers really do not know who to include. So they start with the few they think have the information and then ask, “who else should we talk to?” • Ideally, they continue until no new names are given Dr. G. Johnson, www.ResearchDemystified.org

  11. Types of Nonrandom Samples • Judgmental (sometimes called a Purposive sample) • Definite choices based on criteria that is meaningful given the situation • I might decide to conduct focus groups with the heads of the largest nonprofits in my county—and I will specifically select them by name • United Way, Community Services, Big Brother and Sister, the Food Bank, Affordable Housing Coalition, Interfaith Works, Early Learning Coalition, etc Dr. G. Johnson, www.ResearchDemystified.org

  12. Types of Nonrandom Samples • Convenience • I might send a link to a cyber survey to everyone in my social network (myspace, facebook, twitter) because it is easy but they do not represent the larger community • I might survey everyone in my classes but they do not represent all the students in the school Dr. G. Johnson, www.ResearchDemystified.org

  13. Limitations of Nonrandom Samples • Risk of bias • Why were these people (files, streets, classes, whatever) selected but not others? • Are they substantially different from the one’s not selected? Dr. G. Johnson, www.ResearchDemystified.org

  14. The Inherent Caveat of Non-Random Samples • The results of non-random samples cannot be generalized to the larger population. • Results are always limited to: • “Of the people who participated in the focus groups….” or • “Of the three classes we observed…..” or • “Of the 100 people we interviewed at the corner of walk and don’t walk…..” Dr. G. Johnson, www.ResearchDemystified.org

  15. Non-Random Samples Can Be Useful Despite Limitations • In a study about teenaged mothers, non-random selection made sense • We wanted a mix of ages and ethnic backgrounds • Because we were selecting only a few teenaged mothers from each program to participate in focus groups, it was unlikely that we would have gotten the desired mix through random sampling. Dr. G. Johnson, www.ResearchDemystified.org

  16. Non-Random Samples • Qualitative research • Can yield very useful and important information • Researchers should explain what they did, their rationale and the limitations of any conclusions based on this data • If other research results are similar, it adds strength to their results Dr. G. Johnson, www.ResearchDemystified.org

  17. Option: Random Sample • A random sample means that each person (or item) has an equal chance of being selected • Note: this is not the same as random assignment we talked about in classic experiments • Freshman in a psychology class may be randomly assigned to two different groups, but the results are not generalizable to all freshman, all college students, or all people Dr. G. Johnson, www.ResearchDemystified.org

  18. Random Sampling: Three Benefits • It eliminates bias in selecting participants • It enables researchers to make estimates about the larger population based on what is learned from the sample (jargon term: generalizability) • It enables the researchers to estimate sampling error (we will get to this in a minute) Dr. G. Johnson, www.ResearchDemystified.org

  19. Random Sampling: The Challenges • To locate a complete listing of the entire population from which to select a sample (sometimes called a complete enumeration) • For example, there is no listing of all MPA students in the United States • To select a large enough sample so the results will be statistically meaningful • As a general rule, the larger the sample, the more resources are needed Dr. G. Johnson, www.ResearchDemystified.org

  20. Sample Concepts: The Jargon • Population • the total set of units • Census • A complete count of the population • Sample • a subset of the population • Sampling Frame • list from which to select your sample Dr. G. Johnson, www.ResearchDemystified.org

  21. Sampling Concepts: The Jargon • Sample Design • methods of sampling • probability or non-probability • Parameter • characteristic of the population • Statistic • characteristic of a sample Dr. G. Johnson, www.ResearchDemystified.org

  22. Types of Random Samples • Simple Random sample • Stratified Random Sample • Proportionate and Disproportionate • Multi-Stage/Cluster Samples Dr. G. Johnson, www.ResearchDemystified.org

  23. Simple Random Sample • A subset of the entire population • Example: • A sample of all graduates of the teachers college • A sample of all state employees Dr. G. Johnson, www.ResearchDemystified.org

  24. Simple Random Sampling Process • Obtain a complete listing of the entire population • Assign each case a number • Randomly select the sample • Given a population of 200 students, randomly select the first 25 whose numbers are between 001 and 200 on a random numbers table • See: http://ts.nist.gov/WeightsAndMeasures/Publications/upload/h133_appenb.pdf Dr. G. Johnson, www.ResearchDemystified.org

  25. Mini-Random Number Table Using the 1st three digits, select all the numbers between 001 and 200 Dr. G. Johnson, www.ResearchDemystified.org

  26. Plan B: Systematic Sampling • If a complete enumeration (list) is not available, use a systematic sample: with a random start • We randomly select the staring point, and then select every 20th or 50th file or street (whatever you population of interest) • If you have 300 files in boxes, you might decide to randomly begin with the 19th file, and then select every 25th until you have a sample of 100 files. Dr. G. Johnson, www.ResearchDemystified.org

  27. Random Sampling for Phone Surveys • Why not use the phone book? • Well, not everyone who has a phone is listed. • HUD estimates that between 30-50% of city dwellers do not have listed phone numbers Dr. G. Johnson, www.ResearchDemystified.org

  28. Random Sampling For Phone Surveys • To conduct telephone surveys, a computer randomly generates phone numbers with the appropriate area codes • Jargon: random digit dialing • Efforts are made to include cell phones Dr. G. Johnson, www.ResearchDemystified.org

  29. Random Sampling For Phone Surveys • Those without phones will be excluded, which is a limitation that might matter • Researchers obtain a substantial number of phone numbers: some won’t be working, some might be fax numbers • In a local community assessment, we obtained 3,000 numbers in order to get a sample size of 400 completed surveys • You have to kiss a lot of frogs before you find the prince Dr. G. Johnson, www.ResearchDemystified.org

  30. More Complexity: Stratified Random Sample • What happens when one group is very small in the population and is therefore not likely to show in the sample in large enough numbers? • A stratified random sampling process is used. Dr. G. Johnson, www.ResearchDemystified.org

  31. Process of Selecting a Stratified Random Sample • Population is separated into strata (or groups) • Each strata is randomly sampled • Example: Male and Female CEO’s • A simple random sample of men and a simple random sample of women are selected Dr. G. Johnson, www.ResearchDemystified.org

  32. Stratified Random Sample • Ensures that we have enough men and women in each group to use statistical techniques, like tests for statistical significance (we’ll get to this later) • Stratified random samples tend to be larger than if a simple non-stratified random sample is used Dr. G. Johnson, www.ResearchDemystified.org

  33. Proportionate Stratified Sample • The sample has the same percent distribution as the population Dr. G. Johnson, www.ResearchDemystified.org

  34. Disproportionate Stratified Sample • The sample has a different percent distribution than the population Dr. G. Johnson, www.ResearchDemystified.org

  35. Disproportionate Stratified Sample • This is used when one group (or strata) is so small that it will not yield statistically useful results • The key point to remember is that when the researchers want to generalize back to all 1,000 employees, they will need to weight the data so the proportions are back in line with population • The weighted data would show the results where 80% are men and 20% are women Dr. G. Johnson, www.ResearchDemystified.org

  36. No Listing, More ComplexityMulti-Stage Sampling • Suppose we want to observe classroom activities to measure the amount of time spent on hands-on learning activities. • Randomly select classrooms and then • Randomly select days of the week and then • Randomly select times of day • Then observe all the children in those classes at those times. Dr. G. Johnson, www.ResearchDemystified.org

  37. More Complexity:Cluster Samples Useful when you don’t have a complete listing of the entire population. • If you want to survey parents of primary school children in your country, you probably don’t have a list. • You will want to select a few primary schools and then select a random sample of students who can bring the survey home to their parents Dr. G. Johnson, www.ResearchDemystified.org

  38. Cluster Samples • In war torn countries and natural disasters, a cluster sample approach is used to estimate the number of civilians who are killed. • Researchers select a specific number of geographic areas and then randomly select streets, and then select a house as a starting point, and then select a set number of homes on that block Dr. G. Johnson, www.ResearchDemystified.org

  39. Combinations • Random and non-random methods can be combined. • Judgmental sample of schools: • Select 2 from the poorest communities and 2 from wealthiest communities. • Then select a random sample of students. Dr. G. Johnson, www.ResearchDemystified.org

  40. How Might We Select a Random Sample to Measure Traffic? • We want to observe amount of traffic on the road from the village to major town. • Randomly times of year? • Randomly select times and days of week? • Randomly select observation points or select a single observation point along the road? Dr. G. Johnson, www.ResearchDemystified.org

  41. Discussion • You want to find out how likely it is that graduates of MPA programs in the U.S. will apply for jobs in the federal government as compared to consulting companies. • Remember: there is no complete list of all MPA students Dr. G. Johnson, www.ResearchDemystified.org

  42. Discussion • What are the likely questions you would ask? What are the likely options for data collection? • Which one do you think is the best given your circumstance? Why? • Given you choice of data collection, how would you construct a random sampling plan? Dr. G. Johnson, www.ResearchDemystified.org

  43. Takeaway Lesson • The decisions about whether to use a sample—and whether it should be random or nonrandom—depends on the situation. • If it is possible to collect data from the population, that avoids concerns about selection bias and errors associated with sampling. • Researchers should fully disclose their sampling procedures, their rationale, any problems in the process and the limitations. Dr. G. Johnson, www.ResearchDemystified.org

  44. Creative Commons • This powerpoint is meant to be used and shared with attribution • Please provide feedback • If you make changes, please share freely and send me a copy of changes: • Johnsong62@gmail.com • Visit www.creativecommons.org for more information Dr. G. Johnson, www.ResearchDemystified.org

More Related