How to Design a Mixed Methods Study

1. How to Design a Mixed Methods Study by John W. Creswell, Ph.D. and Vicki L. Plano Clark, M.S. University of Nebraska-Lincoln Andrews University, July, 2004

3. Qualitative Text Data This is a sample of a text file of words that might be collected on transcripts through interviews, fieldnotes from observations, or from optically-scanned documents. Quantitative Numeric Data 2342543112232132 23322543 3122432432132433 32334441 2222111432143213 22111555 2331432432132433 32135432 How would you combine two types of data?

4. Objectives of the workshop: • Let’s design a mixed methods study • Let’s study how people learn mixed methods research in this room? (or you can work on your own project and follow along at each step) • Let’s start with a title. Write a title. • What data will we collect?

5. Quantitative data Close-ended scales Attitudinal/behavioral scales Behavioral checklists Census, attendance records Qualitative data Open-ended responses Semi-structured interviews Semi-structured observations Records/documents Videotapes What are types of quantitative and qualitative data?

6. Let’s identify our quantitative and qualitative data collection

7. Now let’s consider some reasons for why we are collecting (and mixing) both forms of data • Together quantitative and qualitative data provide both precise measurement and generalizability of quantitative research and the in-depth, complex picture of qualitative research • To validate quantitative results with qualitative data • We do not have an adequate instrument. Thus, we need to explore views and develop an instrument • Our quantitative data provide a general explanation and we need to follow-up with participants and have them explain the quantitative results • In our experiment, outcomes to be measured are not enough; they need to be complemented by understanding the process of participants

8. Let’s identify our reason for mixing

9. So… • There are good reasons for gathering both forms of data • But…there are certain requirements for this to work best

10. Requirement #1: Now let’s consider whether we have the skills, time, and resources? • We need minimum skills in both qualitative and quantitative data collection. What do we need? • We need time and resources for extensive data collection and analysis. How much time and resources do we need?

11. Write down the skills, time, and resources we will need

12. Requirement #2: The audience(s) • Does our audience appreciate both numbers and stories? • Are they familiar with this design? • Do they need to be educated? • Are examples of published studies available in our content area?

13. Let’s identify the audiences

14. But audiences may not recognize it yet because it is so new • Increased use and acceptance of qualitative research from 1990’s to present • The complexity of our research problems today requires understanding trends, differences, as well as individual stories, setting • Individuals advocating for and writing about mixed methods research as a distinct, new procedure (e.g., books)

15. They may think that it is analyzing data separately Quantitative Data Qualitative Data Mixing: converging the data or connecting the data

16. But how do we mix? Converge data: Qual Results Quan Connect data: Qual Quan Results

17. Why our audience may recognize it • The evidence • Books • Methodological articles • Many published research studies using it • Federal agencies • Private foundations

19. Other writings, initiatives on mixed methods research: • Research studies reported in journals • Methodological articles exploring issues and procedures • Website for bringing mixed methods writers together • Conference sessions • Handbook of Mixed Methods in Social and Behavioral Research • Private foundation interest; federal agency interest

20. NIH Guidelines - Mentioned several approaches for combining qualitative and quantitative research - Considerations for deciding what model to use (literature available, prior studies, realistic design, expertise) - Need to describe each method thoroughly

21. Quotes: • “Combining qualitative and quantitative methods has gained broad appeal in public health research. The key question has become not whether it is acceptable or legitimate to combine methods, but rather how they will be combined to be mutually supportive and how findings achieved through different methods will be integrated.” (NIH, Office of Behavioral and Social Science Research, 1999).

22. National Academy of Sciences • Three major research questions in • quality educational research: • What is happening? (qualitative designs) • Is there a systematic effect? • (a quantitative experiment) • Why or how it is happening? • (a qualitative followup)

23. But even if they recognize it, they may not appreciate or understand how to design a mixed methods study • “We are interested in a randomized control trial with a non-experimental approach embedded within it.” (a private foundation officer) • “We accept multi-method studies, but investigators mostly do not sort out the complexity of these projects so that we can understand them.” (a federal projects officer).

24. We need to define mixed methods research for our audiences • Mixed methods researchis a design for collecting, analyzing, and mixing both quantitative and qualitative data in a single study or series of studies to understand a research problem. • The purpose of this form of research is that both qualitative and quantitative methods, in combination, provide a better understanding of a research problem or issue than either method alone.

25. Now we could mix within single studies or multiple studies Single Study: Quan Qual Results Multiple Studies (called multimethod research): Quan Qual Qual Quan Study 1 Study 2 Study 3 Study 4

26. So how do we design a mixed methods study? The model Worldviews, theoretical frameworks, problem and research question, skills, resources • Type of mixed methods design • Procedures for: • designing the title • writing the introduction to a study • writing the purpose statement and research • questions/hypotheses • data collection • data analysis • writing the mixed methods report • evaluating the mixed methods research

27. What is a worldview? • Philosophy about your preferences for how you learn about something through research • You prefer the quantitative worldview: you are the expert, you decide what needs to be learned, you build in objectivity • You prefer the qualitative worldview: participant is the expert, participant helps you build knowledge, you bring personal bias in • You prefer both the quantitative and qualitative worldview

28. The next steps in planning our study • Let’s write the overall research question for our study

29. Then let’s choose a type of mixed methods study to conduct • What designs are possible?

30. Types of mixed methods designs I. Triangulation Mixed Methods Design + QUAN Data and Results QUAL Data and Results Interpretation II. Nested Mixed Methods Design QUAN Post-test Data and Results QUAN Pre-test Data and Results Qual Process

31. Types of mixed methods designs III. Explanatory Mixed Methods Design qual Data and Results QUAN Data and Results Follow-up IV. Exploratory Mixed Methods Design QUAL Data and Results quan Data and Results Building

32. Triangulation Design: Characteristics • Collecting both quantitative and qualitative data • Collecting these data at the same time in the research procedure • Analyzing the quantitative and qualitative data separately • Comparing or combining the results of the quantitative and qualitative analysis • Example: collect survey data (quantitative) and collect individual interviews (qualitative) and then compare the results

33. Triangulation Design: When is it used? • When you want to combine the advantages of quantitative (trends, large numbers, generalization) with qualitative (detail, small numbers, in-depth) • When you want to validate your quantitative findings with qualitative data • When you want to expand your quantitative findings with some open-ended qualitative data (e.g., survey with closed- and open-ended data)

35. Nested Design: Characteristics • Collecting both quantitative and qualitative data • Collecting both types of data at the same time • Having ONE form of data play a smaller role in the study than the other form of data • Also, • Using one form of data to answer one question; the other form another question • Collecting one form of data at one level of analysis and another at another level of analysis • Example: You conduct an experiment and during the experiment you gather qualitative interview data. The outcomes of the experiment assessed quantitatively address different questions than the process of the experiment explored qualitatively.

36. Nested Design: When is it used? • When you do not have time or resources to commit to extensive quantitative and qualitative data collection • When you want to study the process of an experiment as well as the outcomes • When you want to examine different levels in an organization

37. Nested Research Design Experiment Intervention Quan Data collection Post-test Quan Data collection Pre-test Process – collection and analysis of qualitative data

38. Explanatory Sequential Design: Characteristics • Viewing the study as a two-phase project • Collecting quantitative data first followed by collecting qualitative data second • Typically, a greater emphasis is placed on the quantitative data in the study • Example: You first conduct a survey and then follow up with a few individuals who answered positively to the questions through interviews

39. Explanatory Sequential Design: When do you use it? • When you want to explain the quantitative results in more depth with qualitative data (e.g., statistical differences among groups, individuals who scored at extreme levels) • When you want to identify appropriate participants to study in more depth qualitatively

40. Here is an example of an explanatory design: Case Selection Interpretation – based on quan and QUAL results Quant itative Quantitative Qualitative Data Collection (quan) Data Analysis (quan) Data Analysis (QUAL) + Qualitative Data Collection Interpretation Quantitative Data* Quantitative Analysis Case Selection Qualitative Analysis Selected 5 cases maximally varying Identified critical months in which smoking varied Number of cigarettes Graphic plot of CES - Descri ption of each Why did changes in D6 scores over time case smoking occur? CES - D6 for each participant Identification of life Qualitative Data* events occurring Graphic plot of during critical cigarettes/day values Semi - structured months where over time for each interviews, audio smoking increased or participant recorded and decreased transcribed Thematic analysis of life events for each * Data collected 10 times case over the course of a Cross - case thematic calendar year for 40 analysis participants Creswell et al. (in progress)

41. Exploratory Sequential Design: Characteristics • Viewing the study as a two-phase project • Qualitative data collection precedes quantitative data collection • Typically, greater emphasis is placed on the qualitative data in the study • Example: You collect qualitative diary entries, analyze the data for themes, and then develop an instrument based on the themes to measure attitudes on a quantitative survey administered to a large sample.

42. Exploratory Sequential Design: When do you use it? • To develop an instrument when one is not available (first explore, then develop instrument) • To develop a classification or typology for testing • To identify the most important variables to study quantitatively when these variable are not known

43. Phase I Qualitative Research - Year 1 Unstructured Interviews - 50 participants 8 observations at the site 16 documents Qualitative Data Collection Qualitative Data Analysis Text Analysis: Using QSR N6 Development of codes and themes for each site Qualitative Findings Phase II Quantitative Research - Year 2 Create approximately a 80-item instrument plus demographics Quantitative Instrument Development Administer survey to 500 individuals Determine factor structure of items and conduct reliability analysis for scales Quantitative Test of the Instrument Quantitative Results Determine how groups differ using ANOVA test Sequential Exploratory Mixed Methods Design

44. Qualitative analysis Text/image data Coding Themes Description Interrelated themes Types of analysis: Quantitative analysis Numeric data Descriptive trend analysis Hypothesis testing, effect size, interval estimates How will we analyze the quantitative and qualitative data (within the design types)?

45. Triangulation data analysis QUAN data collection QUAL data collection • Separate QUAN • and QUAL data • analysis QUAN data analysis QUAL data analysis • Two options • Data transformation (change • QUAL to QUAN or QUAN to QUAL) • Comparison (keep separate and • compare/contrast) Results

46. Table. Example of Data Transformation of Text Units into Numeric Data

47. Nested data analysis Quantitative Experiment Intervention Quan Data collection Post-test Quan Data collection Pre-test Qualitative Process Data Analysis Pre-test scores Themes/Codes/ Interrelated Themes Post-test scores or gain scores Compare/Describe Results

48. Explanatory sequential data analysis • Qual • data collection • (purposeful sampling) • Select cases based on s.d. variables • Select cases to represent outliers • Select cases to represent extreme cases • Select cases to make group comparisons • QUAN • data analysis • Statistical results • Outlier cases • Extreme cases • Qual • analysis • codes • themes • cases

49. Exploratory sequential data analysis QUAL data analysis Quotes Codes Themes Quan data analysis instrument development Items on a survey Variables on a survey Scales on a survey

50. Let’s identify how we will analyze the data