Why Do College Coaching Interventions Work?

1. Why Do College Coaching Interventions Work? Scott Carrell, UC Davis and NBER Bruce Sacerdote, Dartmouth College and NBER

2. 0.6 0.4 0.2 0 1970 1980 1990 2000 Year US NZ UK SWI ITA FIN SWE US College Completion Ranks 11-13thth In OECD Cascio Gordon Clark JEP 2008

3. US Dept of Ed Trio Programs • $879 million in FY 2011 • GEAR up, Upward Bound, Talent Search • Fundamental tenets: • Catch students early..by 8th grade to get them college ready • Avoid discussion of randomized evaluation • Bruce and Scott don’t know what they are talking about

4. US Economist/Ed School Programs • Send letters to top students (Hoxby and Turner) • Auto fill the FAFSA (Bettinger Long Oreopolous Sabonmatsu) • Send text messages (FAFSA reminders or deadline reminders) (Castleman and Page) • Offer remote coaching or application help (Phillips and Reber) • Carrell and Sacerdote: in person help, financial incentives, “likely letters” • Clearly different populations in all of these

5. 2500 2000 1500 Frequency 1000 500 0 1100 1120 1140 1160 1180 math_score No College College 2010 Cohort: Frequency Counts 10th Grade Math Scores for Non College Goers

6. The Mission • Ask whether simple interventions (bonuses, mentoring, likely letters) very late in the game can have a meaningful impact on long term outcomes • Attending, persisting, graduating, income • Use Clearinghouse data to track into College • Are these marginal college going students very different than the average? • For whom does it work? Try to infer mechanisms using baseline interests, preparation, personality measures • SAT Questionnaire, survey data

7. The Answer • Yes • If you are female • Or a guy who has not taken the SATs • If you do not have strong parental and teacher support for completing applications • If you are not extraverted • What doesn’t work • Likely letters • Letters of encouragement (Delaware, NH) • Cash Bonuses alone

8. Hypotheses • Behavioral • Fear of process • Lack of easily obtained information • Procrastination/disorganization • Largely rational: have already optimized • Need expert/ adult help

9. What is the Intervention • Guidance counselors identify high school seniors at risk of not applying to college • Ideally students who have expressed interest in applying • But have taken few steps • Typically identify in mid-December • We randomly choose half for the treatment group • Guidance staff then invites treatment group to participate

10. What is the Intervention (2) • We visit once per week for 3-5 weeks • Pair up each HS student with a Dartmouth student • 1.5-4 hours per session • Start and complete applications • Send transcripts as needed • Pay application fees • Complete common apps, two year schools • Start the FAFSA • Sign up for SATs & send scores • Incentives • Free coaching • App fees paid • $100 cash bonus upon completion • Get out of class • Pizza

11. The Process

12. Mission Accomplished (?)

13. Challenges Faced By HS Students And Guidance Counselors • Horrible internet access and computers • In first year, many schools had BLOCKED the application sites for all NH Community Colleges • When that was fixed, we still had credit card problems • In some cases we brought our own laptops and MiFi cards • Students and parents unfamiliar with Common App, FAFSA, online forms • Students were often selected to be the procrastinators

14. Treatment and Control Standardized Reading Scores

15. Standardized Math Scores: Treatment Group Versus All Non Experimental

16. Table 1: Summary Statistics for Treatment and Control Groups

17. Table 2: Show Treatment Status Unrelated to Pre-Treatment Characteristics Students are randomly assigned to treatment within high school. Data include 2009-2014 cohorts. Regressions include high school*cohort dummies which is the level at which randomization occurred. Standard errors are clustered at the high school*cohort level. Regressions include birthyear*cohort dummies to control for students' age within grade.

18. Effect of Treatment Status on Applying Survey data with 50% response rate. Control mean is .72. Standard errors in parentheses, includes high school* cohort f.e. , age controls, male + significant at 10%; * significant at 5%; ** significant at 1%

19. Baseline Treatment Effects on Enrollment in Any College Outcome variable is a dummy equal to 1 if the student has any enrollment in college including 2 year or four year colleges. Outcome variables are based on the Nation Student Clearinghouse data. Data include 2009-2014cohorts. Regressions include high school*cohort dummies which is the level at which randomization occurred. Standard errors are clustered at the high school*cohort level. ** p<0.01, * p<0.05, + p<0.1

20. Baseline Treatment Effects on Enrollment in A Four Year College

21. Treatment Effects on Persistence in College (Women) Outcome variables are four different ways to measure persistence into the second year of college. Sample is limited to women in the 2009-2012 cohorts.

22. Evidence on Mechanisms Interaction of Mentoring Treatment with Sources of Assistance on Applications Dependent Variable is Enrollment in Any College .

23. Interaction of Mentoring Treatment with Beliefs About Wages/ Tuition Dependent Variable is Enrollment in Any College .

24. Evidence on Mechanisms Interaction of Mentoring Treatment with Sources of Assistance on Applications Dependent Variable is Enrollment in Any College .

25. Interaction of Mentoring Treatment with Personality Traits Dependent Variable is Enrollment in Any College .

26. Interaction of Mentoring Treatment with Personality Traits Dependent Variable is Enrollment in Any College .

27. Some Takeaways • We find a boots on the ground approach is effective • Appears to compensate for lack of or non take up parental or teacher help • No evidence that its more effective for people who struggle with deadlines, planning, organization • Helps students less far along in process, ie non SAT takers • Much more robust effects for the women..lack of effects for men very much related to labor market opportunities • Can’t find evidence for effects from… • Likely letters/ letters of encouragement • Cash bonuses along..Though cash bonuses may be very important for takeup • Texting of CCSNH students has not yielded effects • Nor did letter of encouragement to DE graduated seniors

28. More Takeaways • We find evidence in favor of optimization story and need skilled help story • We don’t find evidence for super naïve behavioral story • Or that failure to attend is driven by forgetfulness or lack of organization • But parents/mentors could indeed be solving this • We suspect that letter based interventions or simple text reminders may only work with specific groups or in specific contexts

29. Also Mention If Time • Huge drop off in participation when we removed the cash bonus • Men very likely to say they have a job they prefer to college • And no evidence in ACS of OLS return to two years of college • Still have low average persistence in both treat and control • Can’t make the Big Five measures work

30. A Very Few Words on Returns to CollegeOreopoulos and Salvanes

31. Caveats on Earnings Returns to College • Large number of papers but sources of ID very different • We haven’t actually shown that the marginal students in current college going interventions have earnings gains • Bettinger, Gurantz, Kawano and I having trouble finding earnings effects from winning the Cal-Grant • The one thing we have learned is just how ridiculously noisy the outcome is

32. Evidence on Mechanisms Interaction of Mentoring Treatment with Sources of Assistance on Applications Dependent Variable is Enrollment in Any College .

33. Do We Affect Type of College Attended?

34. Do We Affect Type of College Attended? (2) Outcome variables measured in IPEDS data. Sample only includes students in college and for whom we have IPEDS data

35. Split Sample By Test Score

36. Interaction of Treatment with Immigration Status Data are from Manchester West 2010,2011 Cohorts. Sample is roughly 9% immigrants.

37. Mechanisms • Bonus? Incentive? • Same or Cross Gender Mentoring? • What do participants say? • More effective in schools with fewer resources?

38. Evidence From 2012 Cohort (Coaching Plus $100 Bonus Versus Bonus Alone)

39. Post-Survey Evidence on $100 Bonus

40. Take Up Rates Within Mentoring Group

41. What Aspects Helped the Most? • 19 of 19 mentioned in person help with applications • 5 of 19 mentioned the $100 bonus • 3 of 7 men versus 2 of 10 women • 12 of 19 mentioned our paying for application fees • Rational bc a lot more than $100 for some students

42. Treatment Effect by High School

43. Average College Going Versus Effect Size For Women

44. Average College Going Versus Guidance Counselors Per Student

45. Mentoring Treatment Interaction with Sources of Disadvantage

46. Mentoring Treatment Interaction with Sources of Disadvantage

47. Mentoring Treatment Interaction with Sources of Disadvantage

48. Stories from the HS Students • > hey Cambell, its Daniel M from west, one of the student you hepled apply for college. If i remember correctly, you told me to let you know what college i get into. Well, i was recently accepted to NHTI in Concord. • There are still some things i need to take care of, but its nothing serious. • > P.S. Thanks for you help. Sorry for all the flirting

49. Cross Tab of Student Male and Assigned Male Mentor Mentors were assigned on a first come first served basis, but when multiple arrivals occurred at the same time, we had a modest bias towards same gender pairings. Regressions include a dummy for being assigned to treatment but not showing up to be assigned a mentor.

50. Is Cross Gender Mentoring More Effective?