Monitoring and E valuation of ICT U se in R ural School s

1. Monitoring and Evaluation of ICT Use in Rural Schools Ok-choon Park Global Symposium on ICT in Education – Measuring Impact: Monitoring and Evaluation of ICT in Education Initiatives November 9 – 11, 2009 Seoul, Korea

2. Should monitoring and evaluation of ICT use in rural areas be different from those in other areas? Probably little for logistics But not for basic procedures and methods Important difference should be the purpose of ICT use. Why ? Rural schools have different contextual characteristics and needs.

3. What are important characteristics of rural schools? • Geographically isolated. • Smaller student populations, smaller class sizes, • and more combined classes • Fewer well qualified teachers • Lower SES of parents • Fewer opportunities for supplemental education • services • Fewer resources and supports inside and outside • schools Fewer Educational Opportunities Lower Educational Quality

4. FewerEducational Opportunities • Fewer advanced courses; fewer enrichment classes; • fewer remedial classes; fewer elective courses • Fewer options for extra curriculum activities • Fewer internship opportunities • Fewer field trip options • Fewer mentoring opportunities • Fewer options for supplemental education • Others

5. Lower Educational Quality • Fewer opportunities Lower quality • Less experienced and less qualified teachers • Fewer resources • Less stimulating interactions with peers • Less parental assistance • Others

6. ICT for Expanding Educational Opportunities • Distance learning, virtual classrooms, and virtual schools • Access to web-based resources • On-line library • On-line mentoring • Virtual field trips • Cooperative learning with students in other schools • Others

7. Enhancing Educational Quality • On-line courses and supplemental instruction materials • ICT-based learning tools and resources • Virtual laboratory experiments, field trips, and others • On-line teaching aids for teachers • Data management tools for teachers • Data-based teaching • Administrative and management tools for teachers • More teaching time • On-line cooperative learning environments • On-line interactions between teachers, students and parents • Others

8. Technology is a double-edged sword! • Technology can expand educational opportunities and • enhance educational quality for all children. • But, technology creates digital gaps between rural and • other poor schools and wealthy schools. Digital gaps widen: Educational opportunity and quality gaps Learning achievement gaps Social and economic gaps • Closing the gaps is a challenging responsibility for • policy makers!

9. Initiative Examples of the US Government for Narrowing the Gaps • Early initiatives for educational technology • - PLATO system – hardware and software development • - Computer Curriculum Corporation – Courseware development • Star Schools – Distance learning • E-Rate – ICT infrastructure building • 21st Community Learning Centers – After-school learning • Many other grants for educational technology These and other government initiatives stimulated private investments in educational technology and its use at schools and homes!

10. Current Access to ICT in the USA Schools • Near universal access to the Internet for all schools and classrooms • One computer for every 3 or fewer students • But, digital gaps and educational opportunity and quality Gaps are • still there! • Limited access to high-speed broadband internet • Fewer computers for students and teachers • Less technical support • More students without computers or Internet connections at home • Fewer resources • Less parental support, and others

11. Recent Trends for ICT Use in the USA • Laptop loan programs – Pennsylvania example • - A laptop for every high school student • - On-site technical support (1/2 time technician) • - Teacher technical training – 30 hours • Virtual schools and classrooms • - Increasing number of state-wide and independent virtual • schools and classrooms • - Widely different between states, districts, and schools: • Course offerings and requirements • Digital textbook programs

12. Needs of Systematic Monitoring and Evaluation for ICT Use • How much do we know about current ICT access and use in • schools and homes? • Do we know what impacts ICT use has had for schools, • teachers, and children? • How do we know what supports schools, teachers and • students need for effective use of ICT? • How can we provide systematic guidance for teachers, parents • and students to choose and use ICT programs and features? • Do we have clear ideas how to improve ICT uses in schools • and homes? • How do we know which programs are good or bad?

13. Two Types of Evaluation for ICT Use • Descriptive evaluation of ICT Use (e.g., NCES survey) • - Information gathering for ICT access and use • - Surveys, observations, interviews, etc. • Impact Evaluation of ITC Use (e.g., IES RCT studies) • - Investigation of impacts of specific ICT features and • programs on student learning • - Experiments, quasi-experiments, regression discontinuity • analysis, etc.

14. Descriptive Evaluation for ICT Access and Use • Access at schools, classrooms and homes for teachers and students • - Internet access • - Computer availability • - Technical support • Use of ICT features and programs • - Instructional and instructional support use • - Administrative and management use • - Other uses – communication • Extent of ICT use – frequencies and lengths • Barriers for ICT use

15. Longitudinal Database of Descriptive Evaluation Data • Trend analysis of ICT access and use • Correlational studies between ICT use and other variables • - Teacher instructional methods • - Student study patterns • - Interaction patterns among teachers, students and parents • - Student learning achievements • - Student attitudes • - Others

16. Needs for Impact Evaluation of ITC Use • Need of scientific evidence-based information for the users • and policy makers • Limited value of past evaluation study results • - Laboratory-based experiments • - Meta-analysis results • - Quasi-experiments and descriptive studies • - Correlational, descriptive, and case studies

17. Methodological Considerations for Impact Evaluation of ICT Use • Selection of specific ICT use or programs for evaluation: • Justifications • - Theory-based • - Demonstrated efficacy • - Practical justification – wide use • - Availability of implementation requirements and guidance • Selection of evaluator • - Objective third party: Firewall between developer/implementer • and evaluator • - Experience and technical competence

18. Selection of evaluation method • - Randomized control study • - Quasi-experiment • - Regression discontinuity analysis • Selection of samples • - Students • - Teachers/classrooms • - Schools

19. Determination of generalization level of the results • - Fixed effect • - Random effect • - Determination of sample size: Power analysis • Selection of outcome measures • - Terminal outcome measures – student achievements • Tailored measures • Standard assessments • - Intermediate outcome measures • Teacher knowledge • Instructional methods • Student attitudes • Others

20. Selection of study scope • - Data collection points • - Number of cohorts • - Selection of implementation feasibility data: • surveys, observations, interviews, etc. • Development of analysis plan • - Analysis models • - Exploratory data analysis; Sensitivity analysis; • subgroup analysis • - Missing data management • - Multiple comparison

21. Recruitment of samples • Assignment of samples to treatment and control conditions • Implementation: as required by developer • - Teacher training • - Classroom implementation • - Technical supports • Data analysis and report generation • - Confirmatory analysis – Summative report • - Exploratory analysis – Formative report • - Report with or without the results and interpretation of • exploratory data