An iLearn Math Program Evaluation Plan

1. An iLearn Math Program Evaluation Plan Claudia Lopez OLIT 508 Barbra Portzline Fall 2012

2. The Highway to Success Learning Communities is at the UNM-Valencia Campus. • We help students transition into College level English & Math courses. • We help all students that tested low in English and/or Math in their Compass or ACT tests. Background

3. New self-paced math program (iLearn) implemented Summer 2011. • Most of our Math classes now are self-paced iLearn classes. • Some students in these classes are not getting through the material in a timely manor therefore leaving them frustrated and not able to pass the final. • Instructors in these iLearn classes do not lecture, but are there along with a tutor for support or questions. • iLearn program has many glitches, and errors. Rationale

4. To find out how effective the iLearn program is. • To find out if this program should be continued or discontinued. • Results will be used to improve our Math Department. Purpose of This Evaluation

5. iLearn Program Designers: • Directly impacted by results whether to keep iLearn program or the need to improve it. • Highway to Success Program Manager: • Makes final decision as to continue or discontinue iLearn. Primary Stakeholders

6. Instructors: • Have a vested interest to make sure that students pass the final. • Highway to Success Tutors: • Vested interest in making sure that students understand the concepts in order to pass. • Students: • Have a vested interest since they have to pass the class in order to keep their scholarships, etc. Secondary Stakeholders

7. Other learning institutions: • May be interested in implementing iLearn program. • Community members: • May be considering taking a self-paced Math course. Tertiary Stakeholders

8. Key questions in this evaluation will focus on finding out if there is a need to continue using the iLearn program. Questions will help us understand the effect that the program has on the students as far as retention and being able to continue on to the next level math class. These questions may also help the instructors in planning supplemental worksheets, and/or study guides to help the students understand concepts. Key Questions

9. In what way(s) is the iLearn math program making a difference at the Highway to Success? • How do the instructors make sure that the students are learning the concepts that are needed to pass the final? • To what extent of computer knowledge should the students have in order to use the iLearn program? • Have the pass rates improved or declined since iLearn was implemented in Summer 2011? If pass rates declined, what can be done to improve them? If pass rates improved, how can we keep the rates high? The Key Questions

10. Students will often minimize the iLearn program by getting on Facebook or other social media. How can we make sure that the time being tracked on iLearn is actually being spent on that program? • What learning styles is the iLearn program targeting, and how is this being done? • How effective are the tutors in helping to understand the instruction being presented in the iLearn program? • Would a special training for math instructors and tutors be effective and in what ways? The Key Questions Continued

11. This evaluation is meant to be formative. • Conducted for the purpose of either improving the iLearn program or discontinuing it. • The evaluation will be conducted by one of the Math instructors along with a Master tutor. • Results will shared with all Highway to Success Math instructors as well as the program manager. • Existing data will be used from previous semesters’ iLearn classes, and instructor taught classes. • Students will fill out surveys using mostly open-ended questions. • Evaluations will be given to students in the 10th-12th week of the semester. • Interviews will be conducted with the instructors & tutors with questions similar to the students. Evaluation Design

12. Two models that will be used include 2 levels. • Level 1 will be used to measure the reactions of the students in regards to the information that is being retained with the iLearn program. • A Level 4 evaluation will be used to measure the effectiveness of the iLearn program in relation to how well it works for the department as well as the instructors. • This design will include the case study design using Kirkpatrick’s approach. • We will be using both qualitative and quantitative data in order to answer the key questions. • The main key question that needs to be answered is if iLearn is helping students pass, or are they failing because they are not understanding the curriculum the iLearn program provides? • Instrument validity will be important in this evaluation. Evaluation Design Continued

13. Interview Questions: • Department Manager will analyze questions carefully & decide how to categorize the results. • The results will then be put into different categories according to the key questions. • Each category will be coded, and the frequency of the codes will be counted. • Data will be displayed using tables, matrices, charts and diagrams. • Survey Questions: • Instructors and Department Manager will analyze survey questions. • Data will be carefully categorized and coded. • This data will be compared to the interview questions as well as the archival data. • The frequency of these codes will be counted as well. • This data will be displayed in tables, charts and diagrams. Data Analysis – Qualitive

14. Archival Data: • Pass rates from previous semesters will be compared to the survey and interview questions. • Pass rates will help the math department decide if the iLearn program is improving pass rates compared to the traditional instructor led math class • Data will be analyzed carefully by the Department Manager, the Administrative assistant, and the math instructors. • Data will be recalculated to make sure that there is an accurate pass rate percentage being revealed. Data Analysis – Quantative

15. Monetary: • Highway to Success has lost their Title V grant & is now running under UNM-Valencia’s budget. • Since Instructor & tutor are doing the evaluation there will not be an extra charge, regular pay will be given. • The only charges will be paper, envelopes, and stamps as needed. • Time: • Evaluation will be performed a few weeks before finals, instructors might not be willing to give students time for filling out surveys. • Students might not be willing to fill out surveys due to being behind. • Instructor might not have time to do interview. For this reason interviews may be done over the phone or on Skype. Evaluation Issues & Constraints

16. Attitudes: • Some instructors do not want to be bothered with interviews, or evaluations even when it comes to the iLearn program. • Students attitudes might come from the frustration with the iLearn program, and might not want to fill out the lengthy survey. Evaluation Issues & Constraints Continued

17. Primary Stakeholders: • Will get ongoing reports, either by phone or mail, or ongoing meetings as needed as well as the final report. • iLearn Program Designers: • Will receive ongoing emails as needed, so they may make improvements as needed, they will also receive a final report. • Secondary and Tertiary Stakeholders: • Will be able to see a final report presented as a PowerPoint presentation with graphs on the Highway to Success website. • Instructors will see the final report in a meeting with a PowerPoint presentation. Communication & Reporting of Findings

18. Evaluation Budget

19. Russ-Eft, D., & Preskill, H. (2009). Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change. New York, NY, USA: Basic Books. Images: http://www.langevin.com http://www.transformleaders.tv http://www.tutor2u.net http://programsuccess.wordpress.com http://baneofyourresistance.com http://harveymackay.com/ Bibliography