1 / 38

Meeting Data Collection Challenges of the Future

Meeting Data Collection Challenges of the Future. James Griffith Ted Socha Thomas Weko Postsecondary Studies Division National Center for Education Statistics. Purpose of the Presentation.

kalista
Download Presentation

Meeting Data Collection Challenges of the Future

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Meeting Data Collection Challenges of the Future James Griffith Ted Socha Thomas Weko Postsecondary Studies Division National Center for Education Statistics

  2. Purpose of the Presentation To show how our “technological innovations” have responded to challenges facing sample survey and longitudinal study data collections.

  3. Overview • Provide overview of postsecondary sample survey and longitudinal studies as prelude to … • Major challenge – Encourage participation of institutions and students • Technological innovations to meet this challenge • Future challenges, concluding remarks

  4. Postsecondary Longitudinal and Sample Survey Studies • National Postsecondary Student Aid Study (NPSAS) • Beginning Postsecondary Students Longitudinal Study (BPS) • Baccalaureate and Beyond Longitudinal Study (B&B) • National Study of Postsecondary Faculty (NSOPF)

  5. Primary Purposes NPSAS • NPSAS:08, includes 125,000 undergraduate and 13,000 graduate/first-professional students enrolled in 1,963 institutions • Describes how students and their families pay for postsecondary education and the role of federal student aid • Describes undergraduate and graduate student populations, including topics such as civic participation, community service, educational aspirations, life goals, disabilities • Provides rich database for postsecondary research and policy analysis, data gathered on over 1,100 variables • Responds to emerging federal policy interests, such as debt level, use of private loans, training in STEM majors, and state-level financial aid issues

  6. Primary Purposes BPS • Includes about 18,500 first-time beginning students identified in NPSAS, followed 2 years and 5 years later • Provides data on student “flow” in and out of postsecondary education, such as persistence, transfer behavior, attainment • Gathers data on initial work experiences of those who obtain certificates or 2-year degrees B&B • Includes about 23,500 bachelor degree completers identified in NPSAS, followed 1 year, 5 years, and 10 years later • Provides important information on graduate education, work, and career through retrospective information on: -- postsecondary experiences -- paths taken to the degree award, time to degree completion • Focuses on “teacher pipeline,” interest in teaching, preparation, and job search

  7. Chronology of PLSSS Studies BPS = Beginning Postsecondary Students Longitudinal Study B&B = Baccalaureate and Beyond Longitudinal Study

  8. Major Data Collections NPSAS occurs in staged sampling periods requiring data collection at several levels: • Institution data collection • Student interview data • Other administrative data

  9. Data Collections by Study

  10. The Challenge

  11. Declining participation of institutions and students • Missing data at the case-level and item-level • Complicated by … -- Diversity in access and use of technology across institutions (about 6,700) and students (about 19 million) -- Data privacy/security concerns Government = Misuse of study data Institutions = Risks of releasing student information Families = Privacy, identity theft -- Tight schedule for data collection, processing, and delivery – from lists of students gathered in May to data delivery in January -- Students as transient population GOAL – To maximize the usefulness and quality of the data vis-à-vis the challenge and complicating factors …

  12. Institutional Data Collection Challenge: Maximize institution participation by acquiring enrollment lists for sampled institutions. Innovations: • Contacted early (fall contact for spring collection) • Established “Study Coordinator” (IRP as initial commitment) • Provided Help Desk (10-12 staff, Mon-Fri work hours) • Provided easy access through Website (information, secure login, CADE) • Ensured confidentiality / security (FERPA documents, IRBs) • Offered multiple options for participation (secure website, encrypted fax, hard copy)

  13. Institutional Data Collection Challenge: Minimize institution nonresponse. Innovations: • Developed real-time monitoring of characteristics of institutions (IMS) • Performed bias analysis for < 85% participation overall and/or within strata, NCES standard • Made weight adjustment to reduce bias at the institution/unit level

  14. Real-time Monitoring System We can see our response statistics in real-time. We then target “under-responding” institutions for participation in near real-time.

  15. Institutional Data Collection Challenge: Maximize student record collection. Innovations: • Provided accessible Web-based instrument (CADE) • Provided Help Desk (Mon-Fri during normal work hours) • Negotiated reimbursement to “incentivize” participation

  16. Example – Web CADE Student Record Abstraction

  17. Student Record Abstraction Method – NPSAS:04 Web CADE preferred choice 1 The total represents the number of students sampled from institutions that completed computer-assisted data entry (CADE) and may include students who were classified as study nonrespondents. 2 Percentage of total number of eligible institutions/students. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).

  18. Institutional Data Collection Challenge: Maximize student transcript collection. Innovations: • Addressed confidentiality/privacy concerns through secure fax, encrypted email, FTP transfer • Negotiated reimbursement to “incentivize” participation

  19. Example – Web Transcript Collection

  20. Student Interview Data Collection Challenge: Maximize student interview collection. Innovations: • Redefined completed case • Employed multiple tracing vendors, e.g., CPS, NCOA, Telematch, Equifax, Experian,TransUnion • Offered response mode options fitting to the population (self-administered via Web, telephone, face-to-face) • Provided accessible Web-based instrument • “Incentivized” for early response • Provided Help Desk (7 days a week)

  21. Cases Have Multiple Data Sources … Primary sources Institution records (CADE) 95% Student interviews 70% Federal aid applications (CPS) 60% Combinations of source All three primary sources 40% Two sources 50% One source 10% Additional sources Federal loans & Pell Grants (NSLDS) 50%

  22. Allowing for Redefining a Case • Student case is a “complete” if valid data exist for: -- Student type -- Birth date or age -- Gender, -- And at least 8 of the 15 variables: • Dependent status, marital status, any dependents, income, expected family contribution, degree program, class level, first- time beginner, months enrolled, tuition, received financial aid, received non-federal aid, student budget, race, and parent education.

  23. Difficult-to-Locate Students

  24. Even so, Some Are Locatable

  25. Example –B&B:08/09 Self-Administered Web Interview

  26. Student Mode Choice for Interview Almost ½ respond via web. 1/3 receive incentive. Growing trend. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, National Center for Education Statistics, 2004 National Postsecondary Student Aid Study (NPSAS:04).

  27. NPSAS:04 Experience with Incentives

  28. “Incentivized” Respondents $20 vs. $30 Web rises in preferred choice

  29. Student Interview Data Collection Challenge: Minimize bias from case-level and item-level missing data. Innovations: • Made weight adjustment at case or student-level • Imputed item-level missing data by: -- Logical imputation or -- Statistical imputation with nearest similar neighbor donor

  30. Who Responds and Who Doesn’t

  31. Nonresponse Procedure • Nonresponse bias was estimated using statistical modeling. • Dependent variable = Responded or Not responded. • Predictor variables = Variables having values for both respondents and nonrespondents, thought to be predictive of response status, e.g.: -- institution type; region; institution enrollment from IPEDS file (categorical); student type; FTB status; etc. • Weight adjustments = Coefficients for predictor variables used for weighting.

  32. Example Weight Adjustments Already seen -- hard to locate and under- respond. So, “double-up” responding case.

  33. Imputation Procedure • Logical – use other data sources to determine missing variable values for given case (e.g., substitute FAFSA for institution CADE). • Statistical –separate cases into dissimilar groups (using a defined set of variables) such that respective group members are alike. • Membership in final group determines donor candidates or “nearest neighbor.” • For cases having missing variable values, such values are “borrowed” from the “nearest neighbor.”

  34. Item-Level Missing Data

  35. Future Challenges to Data Collections • Increased concerns about security / confidentiality -- both institutions and students • Reduced access to students -- Increased liability concerns of institutions to release student-level contact information -- Greater cell phone usage • Raised respondent expectations– offering $ incentives

  36. Products • Data for analysis Data analysis system, http://nces.ed.gov/dasol Restricted data file, http://nces.ed.gov/pubsearch/licenses.asp • Reports On methods … http://nces.ed.gov/pubsearch/pubsinfo.asp?pu bid=2006180 On content … http://nces.ed.gov/pubsearch/pubsinfo.asp?pu bid=2006186

  37. How to be Informed of Products http://ies.ed.gov/newsflash

  38. Contact Information Tom.Weko@ed.gov 202-502-7643 James.Griffith@ed.gov 202-502-7387 Ted.Socha@ed.gov 202-502-7383

More Related