Educational Research Today: Current State, Key Challenges and Future Prospects

1. Martin E. Orland, Ph.D., Director Center for Education National Research Council The National Academies August 25, 2005 Educational Research Today: Current State, Key Challenges and Future Prospects

2. Unprecedented visibility regarding the nature and potential contributions of educational research Policymakers increasingly assume that results from educational research can and should more prominently drive policy and practice Some even assume that the absence of good research is largely responsible for lack of progress in raising student achievement The Current Context: Educational Research in the National Policy Spotlight

3. Three Objectives I. The Critique of Educational Research for Informing Policy and Practice II. Why the Focus Now on Improving Educational Research Quality III. Opportunities, Concerns, and Challenges Presentation Goals

4. I. The Critique

5. No one has argued that we produce too few educational research products Each year thousands of educational research papers and reports are produced in hundreds of journals and other outlets It is the quality of the work that is at issue Concern over Quality not Quantity

6. A Three-pronged Test: Scientific Rigor Topic Relevant to High Priority Decision-maker Needs Useful in Aiding “Real World” Decision-making High Quality Educational Research Must Meet All Three Criteria Defining Educational Research Quality (for promoting sound policy and practice)

7. Scientific Rigor Critique Inadequate reliance in education on the methods of science to validate effectiveness claims Relevance Critique The questions animating much of the work of educational researchers bear little relationship to the “real world” problems of policymakers and practitioners Usefulness Critique Even when researchers address an important topic for policy and practice, the guidance they provide is not clear and/or compelling Three Quality Critiques of Educational Research

8. NRP study of effective approaches in reading Fewer than half of the studies reviewed met minimum quality criteria AIR review of research on comprehensive school reform Only about half met minimal quality research design standards CTP review of teacher preparation research Only 57 of 300 published report in refereed journals met minimum quality standards Rigor Critique: Some Well Known and Influential Reviews of Research

9. “…means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs” (No Child Left Behind Act of 2001) Scientifically Based Research

10. Addresses significant questions that can be investigated empirically Links research to relevant theory Uses methods to permit direct investigation of question Provides coherent and explicit chain of reasoning Permits replication and generalization across studies Allows for public disclosure to promote scrutiny and critique National Research Council

11. A variety of methods can be considered scientifically valid provided they are appropriate to the questions being posed Research inferences are compromised when methods are ill-suited to inform their intended questions -National Research Council What are Scientifically Based Research Methods?

12. Educational researchers and decision makers (i.e., policymakers/practitioners) are used to operating independently Academic rewards for researchers (both tangible and psychic) typically not found in serving the needs of policy/practice No real incentive for researchers to investigate (in a scientifically rigorous manner) high need issues of policy/practice Successful work of this type is exceedingly difficult requiring long –term commitments, collaborations, crossing interdisciplinary terrains, and replications of findings in multiple contexts Historic under-investment by policymakers in mechanisms to bring these disparate worlds together or to change incentives Why a Relevance Gap?

13. Goals of research and policy/practice are typically at odds Policymakers desire relatively simple and straightforward answers to address problems Researchers are comfortable with nuance, ambiguity, and conditional statements Different time horizons Different language/communications conventions within their respective subcultures Why a Usefulness Gap?

14. Culture of ideology and faddishness rather than reliance on research insights dominates educational decision making. When employed, educational research perceived as handmaiden to ideological values – ammunition in the ideology wars Cynicism by policymakers and practitioners that results from research can really be informative or depended on Consequences of This Situation

15. II. Why the Concern Now?

16. Workforce/global competitiveness concerns Nationalization of educational policy since A Nation at Risk Manifest in standards/accountability movement Policymakers and school officials under ever-increasing pressure to document gains in student achievement Driving the Demand

17. Increasing desire for high quality educational research to inform and improve policy and practice (i.e., high demand) LEADING TO : concerns over the shortfall in scientific rigor, relevance, and usefulness of most current research to meet this demand (i.e., low supply) LEADING TO: new federal legislative requirements and program initiatives to raise the supply (NCLB requirements, ESRA legislation, NSF and IES program agendas etc.) Mismatch of Demand With Supply

18. III. Opportunities, Concerns, and Challenges

19. Greater attention to issues of research quality rigor, relevance, usefulness Improved funding/support for high quality educational research Improved connections between research and educator policies and practices Improved student learning and other socially desired objectives Opportunities

20. Inflated expectations of educational research contributions (especially in the short-term) leading to disappointment Perceived politicization of educational research Failing to appreciate shortcomings in research quality and inference that are not fundamentally questions of internal validity (e.g., quality of measurement, external validity concerns) An unequal emphasis in addressing the different dimensions of research quality for policy and practice (scientific rigor, relevance, usefulness) Concerns

21. To increase dramatically: the scientific rigor of educational research for addressing decision maker needs Without: promising too much too soon creating a needlessly adversarial political environment Underinvesting in developing the capacity needed to: identify appropriate targets for sustained long-term research investigation, implement scientifically sound, multifaceted, and long-term programs of research for informing these areas successfully, ensure the provision of “usable knowledge” Challenges

22. Raising to prominence issues of educational research quality has been very healthy The “old equilibrium” needed shaking up and “shocking the system” was probably a necessary first step Need now is for a “new equilibrium” that makes neither “too much” or “too little” of the application of scientific research in education for improving research, policy, and practice Prospects