Deciding the mixed-mode design WP1

1. ROME April 11th | 12th 2019 MIMOD Mixed-Mode Designs for Social Surveys FINAL WORKSHOP Deciding the mixed-mode design WP1 Martina Lo Conte Istat

2. The decisionmakingprocess for mode sequence Mixed-mode strategies used by the NSIs for the main social surveys are based on a great variety of designs The choice of modes and their sequence has an impact on quality and costs. Experiments on large scale on the impact of sequences in literature show results strictly depending on survey specificity and national backgrounds (Wagner et al. 2014, Mauz et al. 2018) It is difficult to define the best sequence that meets all the requirements, national backgrounds and specific survey features. It is important to look for the optimal mixed-mode design, depending on the purposes of the survey design and on its constraints. MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

3. The decisionmakingprocess for mode sequence In order to find the ‘’optimal mixed mode design’’ for a specific survey, it is possible to define a number of steps to follow: Checklist for choiceof mode Estimate the quality/cost trade-off 1 Determine and prioritize aims • Maximizing data quality (minimize the total survey error) taking into account constrains like costs, timing, response burden, etc. • Minimizing costs under the constraint of maintaining the same level of accuracy in terms of total survey error. Different priorities can influence the choice of mode design options MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

4. The decisionmakingprocess: steps 2 and 3 3 2 The main risks when using mixed-mode strategies are mode and selection effects, break in time series, and budget overruns. Determine the candidate modes Identify risks Once priority of aims is established and risks are identified, then the potential modes to adopt can be determined. Other ingredients to take into account: mode adequacy for the survey topics as well as the availability of contact information (telephone numbers, postal addresses, e-mails). MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

5. The decisionmakingprocess: Step4 4 Evaluatedifferent mixed-mode design options a)    Define the sequence of modes b)   Define which devices respondents can use, if CAWI mode is offered c) Designand test the questionnaire to reduce mode measurement effect d)   Evaluate the complexity of the logistics and operations (Data collection IT system and human resources) e)   Evaluate the possibility of enhancing the response rate and/or population coverage (incentives, mode choice) f)    Design the communication strategy: (advance letter, reminders, interviewer instructions and training, survey website) g)  Test the mixed-mode design MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

6. The decisionmakingprocess: Step4 4 Evaluatedifferent mixed-mode design options Concurrent, sequentialorboth? a)    Define the sequence of modes Some general rules, based on the priorities defined in step 1: • If cost reduction has the highest priority a sequential design where the cheapest mode is offered first can be adopted • Overall cost need to be evaluated. Contamination effect • If the aim is to contain non-response and coverage errors concurrent mix of modeswith interviewer-administered modes allow for a higher response rate and coverage, or a sequential design with most “effective” mode offered first • Strategies in between, like party sequential-partly concurrent designs, can help in balancing costs and errors, and keep low organizational complexity MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

7. The decisionmakingprocess: Step4 4 Evaluatedifferent mixed-mode design options a)   Define the sequence of modes WP5 b)  Define which devices respondents can use, if CAWI mode is offered WP4 c) Designand test the questionnaire to reduce mode measurement effect d)   Evaluate the complexity of the logistics and operations (Data collection IT system and human resources) WP3 e)  Evaluate the possibility of enhancing response rates and population coverage (incentives, mode choice) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

8. The use of incentives Literature suggests that, in general, incentives may increase survey participation: monetary and especially cash are more effective than non-monetary incentives and unconditional incentives are more effective than conditional ones. Concern has been raised that incentives might reduce data quality, but there appears to be little empirical evidence supporting this. On the contrary, some studies indicate that incentives may influence respondents to put more effort into completing the questionnaire (Olsen et al., 2012). Offering incentives might raise costs. They depend, therefore, on the priority of aims and on the budget available. A possibility is to offer incentives only to certain sub-groups of respondents, that are in general those ‘harder’ to involve in the survey. MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

9. Mode choice: to giveit or not to giveit? The possibility to choose the mode is frequently given to meet preferences of respondents, so to reduce their burden and better prepare them to the interview. However there are not evidences that it raises response rates, but it seems even worsening because of the overload caused by the pressure to decide which mode The easiest way is not choosing..? • mode choice could add an additional danger of confounding mode effects and measurement effects with self-selection bias • potential increase of the operational complexity, especially in concurrent designs. • switching respondents can also raise planning problems. When the fieldwork is outsourced (entirely or partly), it is necessary to know in advance how many interviews will be realized with each technique. • Pilot studies or previous experiences can help with these problems. Drawbacks: MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

10. The decisionmakingprocess: Step4 4 Evaluatedifferent mixed-mode design options a)   Define the sequence of modes WP5 b)  Define which devices respondents can use, if CAWI mode is offered WP4 c) Designand test the questionnaire to reduce mode measurement effect d)   Evaluate the complexity of the logistics and operations (Data collection IT system and human resources) WP3 e)  Evaluate the possibility of enhancing response rates and population coverage (incentives, mode choice) WP4 f)   Design the communication strategy (advance letter, reminders, interviewers instructions and training, survey website) g)  Test the mixed-mode design MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

11. The decisionmakingprocess : Step 5 5 Implementthe mixed-mode design and collect data During this phase, two viewpoints have to be taken into account: that of respondents and that of interviewers (if they are involved). As to respondents, it is important to always facilitate respondents tasks by setting a contact centre not only to provide respondents with technical and thematic assistance, but also to assure them about the confidentiality of the data they provide. For interviewers, training and motivation are essential. This is particularly true in mixed-mode surveys using web as first mode, where it can happen that the “easiest” respondents participate to the online questionnaire and the “most difficult” cases are left to interviewers. MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

12. The decisionmakingprocess : finalsteps 6 Compute estimates 7 Assessthe results in terms of the defined aims 8 Document These steps conclude the survey process. Step 7 about the assessment has the purpose to evaluate the design performance in terms of data quality and costs, and therefore to provide indications for future survey editions. MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

13. The checklist for the mode decisionprocess 1 1.       Determine and prioritize aims a)   Define the mode sequence/administration 2 b)  Define which devices respondents can use (CAWI) 2.       Identify risks c) Designand test the questionnaire 3 3.       Determine the candidate modes d)   Evaluate the complexity of the logistics and operations (case management system and human resources) 4 4.  Evaluate mixed-mode design options 5 5.       Implementand collect data e)   Evaluate the possibility of enhancing response rates and population coverage (incentives, mode choice) 6 6.       Compute estimates f)   Design the communication strategy 7 7.       Assess the results in terms of the defined aims g)  Test the mixed-mode design Results of Tests (step 4g) and of Assessment (step 7) might activate a ‘re-design’ step. 8 8.      Document MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

14. Thankyou MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019