1 / 16

David Figlio , Northwestern U Mark Rush, U Florida Lu Yin, American Institutes for Research

Is it Live or is it Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning. David Figlio , Northwestern U Mark Rush, U Florida Lu Yin, American Institutes for Research. Background. Two major trends affecting higher education

adrina
Download Presentation

David Figlio , Northwestern U Mark Rush, U Florida Lu Yin, American Institutes for Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Is it Live or is it Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning David Figlio, Northwestern U Mark Rush, U Florida Lu Yin, American Institutes for Research

  2. Background • Two major trends affecting higher education • Declining state and local appropriations  more fiscal challenges, increasing demands for “efficiency” (new NRC panel) • Rapid improvements in technology • As a consequence, millions of students are now taking classes online

  3. What do online classes look like? • Two major types: • Innovative, highly interactive classes aimed at exploiting special nature of the internet and modern technology • Traditional lectures presented in an online format • While door #1 is the type of class advocated by learning scientists (NB: I’m working with colleagues to develop this type of class in high school science), door #2 is what most universities are doing.

  4. What is the causal question of interest? • Do students learn more in a traditional lecture when it is broadcast on the internet than when it is presented in a standard lecture hall? • In principle, the results could be positive or negative: • Pro: increased flexibility; no need to rely on others’ notes; ability to annotate/rewatch lectures • Con: increased barriers to student-faculty interaction; reduced ability to ask just in time questions; incentives to defer work

  5. What is the causal question of interest? (part 2) • Are there heterogeneous effects of internet-based traditional lectures on students? • Why interesting? • Some groups may face additional communication barriers (e.g., language minority students) • Some groups may have lower self-regulation skills (e.g., college-aged men; relatively lower-achievers)

  6. What is the evidence to date? • Almost nothing • Mainly small-scale case studies; unsurprisingly, almost all studies are based on type 1 of internet-based class rather than type 2 of internet-based class • Somewhat larger studies have poor treatment-control contrast • Considerable need for experimental evidence on both types of internet class, but especially the broadcast-traditional-lecture type  this study

  7. What is the ideal experiment? • Treatment and control students should be taught in tandem • Same instructor, same exams, same lectures, same supplementary material, same readings • Pure randomization to live vs. internet treatment • No opt-out from experiment • No opportunities for contamination of treatment and control • Live students cannot view internet lectures; internet students cannot attend live lectures • Ample opportunities for detecting heterogeneous effects and improving external validity • Dozens of experiments in different courses at different institutions, with within-course randomization • Cluster-randomized design; clustering on subgroups to ensure sufficient sample size to detect subgroup-specific effects

  8. Fidelity to ideal experiment • Treatment and control students should be taught in tandem • Same instructor, same exams, same lectures, same supplementary material, same readings • Pure randomization to live vs. internet treatment • No opt-out from experiment (instead: randomization of volunteers) • No opportunities for contamination of treatment and control • Live students cannot view internet lectures; internet students cannot attend live lectures (live students may potentially watch with friends) • Ample opportunities for detecting heterogeneous effects and improving external validity (nope: just one class, no group cluster) • Dozens of experiments in different courses at different institutions, with within-course randomization • Cluster-randomized design; clustering on subgroups to ensure sufficient sample size to detect subgroup-specific effects

  9. Threats to internal and external validity • (1) Volunteers, rather than pure randomization • How representative are volunteers of the potential study population at the institution? [external validity]  Table 1 • Knowledge that this is an experiment might lead to differential attrition of live vs. internet [internal validity]  Table 2 contrasts; bounding exercise in Table 3

  10. Threats tointernal and external validity • (2) Fidelity of randomization and lack of treatment contamination • Do people drop from experiment post-randomization? [internal validity]  as 15 students assigned to “live” dropped from the experiment, Table 3 compares results treating defectors as “live” versus dropping from study • Do “live” students view internet version and do “internet” students attend live lecture? [internal validity] • Door guards checked IDs so we know that no “internet” students attended live lectures • Live students did not have online access, but could have accessed via friends’ log-ins  Figure 1 shows that “live” students attended substantially more lectures than “live+internet” non-volunteers

  11. Threats tointernal and external validity • Intro economics might be special • The university in question might be unusual • The particular instructor may translate well/poorly to the internet platform • Lots of hand-wringing • And also a call for more experiments in other subjects and settings and with a design aimed at detecting heterogeneous treatment effects

  12. Threats to internal and external validity • (1) Volunteers, rather than pure randomization • How representative are volunteers of the potential study population at the institution? [external validity]  some differences; volunteers had higher GPAs and lower SAT scores and mom was less likely to be a college grad. • Knowledge that this is an experiment might lead to differential attrition of live vs. internet [internal validity]  no evidence of differential attrition: 6 live, 10 online attriters, and no differences between them. Bounding exercise (giving attriters scores of 0 or 100 on missed exam) shows tight bounds when considering attrition.

  13. Threats tointernal and external validity • (2) Fidelity of randomization and lack of treatment contamination • Do people drop from experiment post-randomization? [internal validity]  no differences in results when we treat defecting volunteers as “live” versus when we drop them from the study • Do “live” students view internet version and do “internet” students attend live lecture? [internal validity] can’t know for certain about live students viewing lectures online, but it looks like live students definitely attend more live lectures than those with the choice. Note that there is a “professional” private note-taking and tutoring service that is very popular and available to all students regardless of live/online. Other studies at the institution on cramming indicate that many internet course-takers don’t view all (or most, or sometimes any) of the lectures either

  14. Threats tointernal and external validity • Intro economics might be special • The university in question might be unusual • The particular instructor may translate well/poorly to the internet platform • This is important, and there’s nothing we can do about this except to call for more experiments

  15. Results • Small insignificant positive estimated effects of live-only vs. internet-only instruction; statistically significant (mostly due to larger coefficients rather than smaller standard errors) when conditioning on covariates • Positive estimated effects are largest for Hispanic students (sig) and Asian students (not sig); male students (not sig) and students with relatively low SAT scores (not sig)  larger sample sizes and cluster randomization might have helped detect differences here

  16. Conclusions • There might be efficiencies to exploit with internet-based traditional lectures, but there is no free lunch • The results of this experiment should be interpreted as a first piece of evidence; responsible universities should be slow to implement this policy change despite the momentum and push for this. We need many more – including larger -- experiments!

More Related