Download
1 / 18
180 likes | 298 Views
Measuring the Impact of IFLE Grant Programs Evidence-Based Grant Design & Evaluation U.S. Department of Education, International & Foreign Language Education Program. Andy Gluck Technical Director ICF International . KimOanh Nguyen-Lam, Ph.D. Advanced Training & Research Division Director.
E N D
Measuring the Impact of IFLE Grant Programs Evidence-Based Grant Design & Evaluation U.S. Department of Education, International & Foreign Language Education Program Andy Gluck Technical Director ICF International KimOanh Nguyen-Lam, Ph.D. Advanced Training & Research Division Director
WRITING MEASURABLE OBJECTIVES An objective is a ‘foreseen end that gives direction to an activity.” John Dewey, 1977 • A goal is where you want to be and objectives are the steps taken to reach the goal. • Goals are the large statements of what you hope to accomplish but usually aren’t very measurable. • Measurable objectives communicate in specific and concrete terms the observable results to be attained. Measurable means that someone can assess the extent to which the project achieved the objectives.
WRITING MEASURABLE OBJECTIVES • When writing objectives, use ACTION VERBS such as perform, identify, list and match. Avoid vague and immeasurable VERBS (appreciate, realize and be aware of) • Each objective should answer the following questions • WHAT, WHO, WHEN and HOW MUCH • What (measurable) change or benefit is expected? • Whatshould occur? • Who is expected to change or benefit? • Whowill benefit from the project? • How muchchange or benefit is expected? • How much change must occur to prove the project’s success? • When is the change or benefit expected to happen? • When will the change happen?
“If you’re not measuring it, you’re not managing it.”S.M.A.R.T. Objectives
S.M.A.R.T. Objectives • SPECIFIC • Concrete • Use action verbs • What specific change is going to occur? • MEASURABLE • Numeric or descriptive • Quantity, quality cost • How will you know that a change has taken place? • ASSIGNED • Relates to a specific target audience. • Who will change? • REALISTIC • The objective is achievable. It can be accomplished. Measures outputs or results. • What is the baseline (The starting point-Increase from what to what)? How much improvement is anticipated? • TIMELY • Relates to the specific time to achieve the objective. • Identifies target date • When will the change take place?
Developing Clear Goals and Objectives Consider the following when developing measurable objectives and planning data collection: • How do we know whether the project is meeting its goals? • What types and sources of data will best demonstrate the project's progress toward achieving its objectives? • How will the data be collected? Describe access and frequency. • How will the data be analyzed and reported? Describe the methodology and key personnel responsible. • Would the results be able to demonstrate the project’s proposed outcome and impact (e.g., an increase in qualified language instructors, higher graduation rate in international studies, better rate of graduate placement, etc.)?
Examples of possible project-specific quantitative objectives include: • Increase the number of students completing advanced courses in XYZ priority languages; • Increase the number of students from business, health, or science majors graduating with foreign language skills; • Increase the number of study abroad opportunities for students on campus; • Increase the number of certificates and degrees conferred in relevant programs of study.
Examples of possible project-specific qualitative objectives include: • Improved employment opportunities for students who possess advanced language skills and international experience; • Stronger collaboration between foreign language departments, international education, and other disciplines; • Improved quality of assessment tools for priority and/or less-commonly-taught languages.
Progress indicators that relate to the quantitative and qualitative examples cited in the above sections include, but are not limited to, the following: • Number of new faculty positions in priority and/or less-commonly-taught languages, area studies courses, or interdisciplinary courses that are institutionalized after grant support has ended; • Number and type of courses developed, piloted, and subsequently submitted to the institution’s review board for inclusion in the college catalog for the upcoming academic year.
Developing Evaluation Questions Formulate evaluation questions that are of interest to all stakeholders and audiences related to the proposed project, and align questions with appropriate information gathering techniques. • Who/what will change? • When do you expect the change(s) to take place? • How much change is expected? • How will change be measured, recorded, or documented?
Planning Data Collection and Analysis • Establish baseline data • Develop data collection instrument (surveys, standardized tests, exams, focus groups, topic guides, etc. ) • Specify the types and sources of data that will be collected • Describe how the data may be collected (including access and frequency) • Describe how the data will be compiled, analyzed, and reported • Describe the methodology and key personnel responsible for these tasks
Instructions for Completing GPRA PMF All applicants filled in a GPRA PMF according to the following instruction steps 1-6. Step #7 was added to allow grantees to report actual results. Grantees need to edit the original GPRA PMF they submitted at the time of application to include the A1-A4 columns. • Performance Measures – The GPRA measures are already filled in. • Activities – Fill in the major activities your institution will undertake to support achievement of the GPRA performance measure goals. • Data / Indicators – State the data or indicator that will be used to track progress of each activity stated in #2. Note that one of the indicators should be synonymous with the performance measure, while other indicators play a supporting role to track progress toward meeting the performance measure. Since the performance measure might not change from the baseline in the early year(s) of the grant, supporting indicators are used to track and demonstrate progress. • Frequency – State periodicity of measurement (e.g., Quarterly, by Semester, or Annual). • Data Source – State the source of the data (e.g., Grantee records or University registrar). • Baseline (BL) and Targets (T1-T4) – State the “Baseline” number of participants (if any) that meet the GPRA measure requirement prior to the current grant that is being applied for. Please state how many participants you intend to have been of service to at the end of each grant year as your answers for T1 through T4. • Actuals (A1-A4) – State these figures as the actual results that can be compared against the targets in the preceding columns.
Instructions for Completing Project-Specific PMF • Project Goal Statement – Each project will have more than one larger goal such as “Development of an international internship program for foreign language students and business students at minority-serving community colleges,” or “Provide international learning experiences to STEM and Business faculty and students, with a special emphasis on part-time adult, nontraditional students.” Each project goal will require a separate PMF template row to be completed. State the first project-specific goal in the first (header) row of the template. Proceed to steps 2-7 below. Repeat the process for each project-specific goal. • Performance Measures – State the project-specific measure in an objective and time-bound manner. Make sure the units of measure (e.g., # of courses, # of students, etc.) are well defined. • Activities – Fill in the major activities your institution will undertake to support achievement of the project-specific performance measure goal. • Data / Indicators – State the data or indicator that will be used to track progress of each activity stated in #2. Note that one of the indicators should be synonymous with the performance measure, and the other indicators play a supporting role to track progress toward meeting the performance measure. Since the performance measure might not change from the baseline in the early year(s) of the grant, supporting indicators are used to track and demonstrate progress. • Frequency – State periodicity of measurement (e.g., Quarterly, by Semester, or Annual). • Data Source – State the source of the data (e.g., Center records or University registrar). • Baseline (BL) and Targets (T1-T4) – State the level of each measure at the time of application submission as the “Baseline” and provide the target levels at the end of each grant year. Where the measure specifies “new,” the baseline is stated as the existing total at the time of the application and the targets are stated as the incremental increase.
