Research partially supported by contract with the US National Center for Health Statistics,

1. Responsive Design for Household Surveys: Illustration of Management Interventions Based on Survey Paradata Robert M. Groves, Emilia Peytcheva, Nicole Kirgis, James Wagner, William Axinn, University of Michigan, USAWilliam Mosher, US National Center for Health Statistics Research partially supported by contract with the US National Center for Health Statistics, Contract No. 200-2000-07001

2. Definition: Responsive Design Survey designs that: • Preidentify a set of alternative features potentially affecting costs and errors of statistics • Identify a set of indicators of the cost and error properties of those feature • Monitor indicators in initial stages of data collection • Alter the active features of the survey based on cost/error tradeoff decision rules • Combine data from separate phases into a single estimator

3. The Why’s of Responsive Designs • One-off surveys are mounted with large uncertainties (e.g., eligibility of frame elements, effort required to contact, cooperation rate, length of interview) • Most survey budgets are relatively fixed at start of project • Some survey errors are functions of effort during production Hence, quality is out of control of researcher unless designs are permitted to change based on production experience

4. The NSFG Dashboard Effort Active Sample Productivity I’rs working occupied interviews hours eligible cum. interviews % production nonworked hours/interview X calls/day noncontacts calls/interview calls/hour mean calls % peak calls 8+ calls Data Set Balance scrn’r/main calls locked bldgs resistant response rate hard appt. % with kids propensity % sexually active group rates CV group rates

5. Evaluation of Two Interventions Based on Paradata • Survey setting: face-to-face survey screening (3-5 min.) to locate 60% of households with 15-44 year old; one eligible sampled (60-80 min.) • Interventions • Increasing relative effort on screening interviews versus main interviews with selected respondent • Increasing relative effort on a small subset of cases with high selection weights and high propensities to respond

6. Ratio of Screener Calls to Main Interview Calls by Day by Quarter

7. Autoregressive Time Series Coefficients for Model Predicting Daily Number of Screener Calls(p-values for coefficients)

8. Autoregressive Time Series Coefficients for Model Predicting Daily Number of Screener Interviews(p-values for coefficients)

9. Second Intervention: Increased Emphasis on Subset of Active Cases • In the last weeks of Phase 1, a subsample of cases with high propensities and high selection weights are identified • These cases are chosen to improve balance of respondent pool • Interviewers are asked to give greater emphasis to these cases

10. Mean Expected Probability to be Interviewed on Next Call, Screeners (Red) and Main (Green) by Day of Data Collection by Quarter

11. Analytic Approach • Not all interviewers active workloads contain both “intervention” cases and “non-intervention” cases • We limit the analysis to those who have both types of cases • We examine two indicators of success: • Mean number of calls (imperfect) • Response rate in intervention period

12. Comparison of Mean Screener Calls During Intervention Period for Intervention and Nonintervention Cases Little evidence of increased calling on intervention cases

13. Comparison of Screener Response Rate During Intervention Period for Intervention and Nonintervention Cases Little evidence of increased response rate on intervention cases

14. Comparison of Mean Main Calls During Intervention Period for Intervention and Nonintervention Cases Mixed evidence on higher calls on intervention cases

15. Comparison of Main Response Rate During Intervention Period for Intervention and Nonintervention Cases General tendency to higher response rates for intervention cases

16. Logistic Regression For Likelihood of Main Interview

17. Conclusions • Intervention 1: Management direction to focus on screeners vs. main increases calls, sometimes dramatically; screener interviews follow • Intervention 2: Effectiveness at focusing on individual cases greater for main interviews than screener interviews

18. Next Steps on Responsive Design with Paradata • Responsive design requires effective central management direction of interviewer behavior • We’re still learning how to communicate these directives consistently well