“Scientifically Based Evaluation Methods”

1. “Scientifically Based Evaluation Methods” Presented by Paula J. Martin COE Conference, September 13, 2004

2. Introduction • US Department of Education Definition • Where is it now? • What does it establish • Why should we be concerned? Potential Impact on TRIO • How can we prepare • Possible evaluations we can do now

3. Scientifically Based Research Methods Definition According to the US Department of Education “Scientifically Based Research Methods” are considered to be a priority for program projects proposing an evaluation plan that is based on rigorous scientifically based research methods to assess the effectiveness of a particular intervention. This priority is intended to allow program participants and the Department to determine whether the project produces meaningful effects on student achievement or teacher performance.

4. Where Is It Now? Currently this stipulation exists within the ESEA or Elementary Secondary Education Act as reauthorized by the NCLB (No Child Left Behind Act). It is intended to ensure that Federal funds are used to support activities and services that work. It is a Proposed Priority for other education programs in Federal Register, Vol. 66, November 4, 2003, pp. 62445-62447. Comments were accepted up to December 4, 2003.

5. What does it propose to establish? • If the priority is used as a competitive preference priority, points awarded under this priority will be determined by the quality of the proposed evaluation method. In determining the quality of the evaluation method, we will consider the extent to which the applicant presents a feasible, credible plan that includes the following: • The type of design to be used (that is, random assignment or matched comparison). If matched comparison, include in the plan a discussion of why random assignment is not feasible.

6. What does it propose to establish continued • Outcomes to be measured • A discussion of how the applicant plans to assign students, teachers, classrooms, or schools to the project and control group or match them for comparison with other students, teachers, classrooms or schools. • A proposed evaluator, preferably independent, with the necessary background and technical expertise to carry out the proposed evaluation. An independent evaluator does not have any authority over the project and is not involved in its implementation. In general, depending on the implemented program or project, under a competitive preference priority, random assignment evaluation methods will receive more points than matched comparison evaluation methods.

7. Why should we be concerned? Potential Impact on TRIO • Proposing that this be implemented by any office as needed for various projects within the Department of Education. • Contains key buzz words: “evaluation”, “consistent with statutory purpose”, “project effectiveness”, “evaluation method”, and “outcomes to be measured”. • Current climate that exists within the Department due to NCLB, increased pressure to hold projects they oversee as being accountable, and increased monitoring of projects. • The fallout from ongoing and completed National Evaluations of TRIO programs such as Upward Bound and Talent Search.

8. Potential Impact on TRIO • It is very conceivable that if this priority is adopted that we will see it or some facsimile appear within our grant applications under the evaluation section. • If an external evaluator is required, there is no guarantee that additional grant money will be set aside for the hiring of such an individual. • Will require many of us to incorporate an evaluation method(s) that most of us know little about. • May require the revamping of the design of our projects in order to be able to meet the evaluation requirements.

9. How can we prepare? • Begin to study what are “randomly-assigned” and “quasi-experimental designs with matched comparison conditions” as well as other alternative experimental designs such as “regression discontinuity” or “single-subject”. • Look at various evaluation plans such as “Logical”, “Theoretical”, and so on to see what best fits our projects and then begin to implement them. • Begin doing internal and external evaluations of our projects. • Begin to look at comparison data. • Begin to educate and solicit the cooperation of outside agencies who will be key to carrying out these designs, such as our target schools, agencies, institutions, participants, and so on.

10. Possible Evaluations We Can Do Now? • Look at the graduation rates from high school of our participants vs. that of students who are not our participants in our target schools. • Look at college placement rates of our participants vs. that of students who are not our participants in our target schools. • Look at college graduation rates. • Set up our own cohorts, if not already mandated as in the case of Upward Bound and now SSS, to follow as to their progress towards graduation and college placement and graduation. • Collect pertinent data that will allow us to conduct comparative analyses.

11. Need to consider cost and whether there will be additional monies for the hiring of an external evaluator and implementation of the evaluation plan and method. Not all grantees are created equal with the same amount of resources at their disposal. On what authority would grantees be able to demand the cooperation of schools, teachers, students to participate in a study or provide data? Randomly assigned design might be in violation of grant regulations that require that project activities/services be available to all participants because of need. Would necessitate a reexamination of performance reports criteria and reporting. Highlights of my comments in response to Federal Register Proposed Rules