A Process for Analyzing Coherence in Science Instructional Materials

1. A Process for Analyzing Coherence in Science Instructional Materials Presenter: April Gardner Collaborators: Joseph A. Taylor, BSCS; Rodger W. Bybee, BSCS; Liu Enshan, Beijing Normal University 2 May 2010 AERA Annual Meeting Denver, CO

2. Project Background • Asia Society convened meetings of American and Chinese math and science education leaders in mid-2000s • This group encouraged collaborative work to examine and compare curriculum standards and materials in the two countries • In 2006, BSCS received a grant from the Office of Science Education at NIH to collaborate with Beijing Normal University biology educators to compare U.S. and Chinese biology textbooks

3. Initial project goals Compare U.S. and Chinese secondary biology programs for: • Foundational concepts • Role of scientific inquiry • Role of educational technology

4. Refining the goals Goals were refined and expanded based on our recognition that: • both the U.S. and China had developed and adopted national science curriculum standards in the recent past (1996 for U.S. and 2003 for China), • both countries had popular, pre-standards programs in use as well as less-popular, post-standards programs in use, and • it would be interesting to compare not only distinctions between American and Chinese instructional materials, but also distinctions between pre- and post-standards curriculum materials.

5. An emphasis on curricular coherence As we continued this work, we further recognized that: • international comparison studies identified curricular coherence as a key distinction between high- and low-performing countries, • focus and rigor have been described as two components of overall coherence (Schmidt et al., 2005; Schmidt & Prawat, 2006), and • focus and rigor are key features of science curriculum standards in both the U.S. and China. Thus, our emphasis changed to • analysis of the coherence (focus and rigor) of one pre-standards and one post-standards biology program from each country

6. When we completed this work, we recognized that we had developed and tested a method for analyzing the coherence of instructional materials, and that’s the more interesting story I’ll relate today . . . A more interesting story

7. Four programs reviewed

8. Process for evaluating coherence:Background • Adaptation of part of the Analyzing Instructional Materials (AIM) process (Powell et al., 2002) • Uses criteria based on research about how students learn science • These criteria address both focus and rigor

9. Process for evaluating coherence:Analyzing focus Example: • Construct and score a conceptual flow graphic (CFG) • Write the overarching concept of chapter and major concept(s) of each section • Depict strength of connections among these using arrows of varying thicknesses • Score by giving each connecting link a score based on its strength: 2 for strong, 1 for moderate, and 0 for weak or no link • CFG score = sum of all links, divided by sum if all links are strong • Transform % score to a score of 5, 3, or 1

10. Process for evaluating coherence:Analyzing focus, continued • Search instructional materials for evidence of focus based on sequencing, context, and key concepts. • Evaluate evidence for strengths & weaknesses, and score as 5, 3, or 1 (strongly, moderately, or weakly address criterion, respectively).

11. Process for evaluating coherence:Analyzing rigor • Search instructional materials for evidence of rigor based on depth of treatment, engagement of prior knowledge, student metacognition, and abilities and understandings of scientific inquiry. • Evaluate evidence for strengths & weaknesses, and score as 5, 3, or 1 as for the focus criterion.

12. Process for evaluating coherence:Scoring • Focus, rigor, and overall coherence scores determined as indicated in the table:

13. Research question • Can the process we devised for assessing coherence distinguish instructional materials developed before the adoption of national science standards from those developed after the adoption of the standards? Hypothesis: Science instructional materials developed after the adoption of national standards will exhibit greater levels of coherence than those developed before the adoption of national standards.

14. Procedure followed • Researchers were trained in the use of the modified AIM process using an unrelated program. • Project director selected the genetics and ecology chapter(s) to be analyzed from each of the four programs. • Three researchers independently reviewed the chapter(s), identified the overarching chapter concept and section concepts, and collected evidence to support their scores for the focus and rigor criteria. • The research team met to compare and reach consensus on each overarching chapter concept and the major section concepts. • The team then collaboratively constructed a CFG for each chapter. • The team discussed and revised scores on the focus and rigor criteria to reach consensus on the scores.

15. Sample CFG: Program B50% of possible strong linksModerately focused (score = 3)

16. Sample CFG: Program D89% of possible strong linksHighly focused (score = 5)

17. Example of evaluating rigor for “understandings of scientific inquiry” • Weak rigor (score = 1) • The chapter described few or no scientists and their work. There is no opportunity for students to connect how the evidence of scientists’ investigations leads them to their conclusions. • Strong rigor (score = 5) • The chapter described how the understanding of inheritance progressed over time, through the ensuing work of scientists. For example, following a detailed description of Mendel’s experiments, results, and elucidation of the principles of inheritance, the text goes on to explain how some results of genetic crosses conducted by other scientists did not fit Mendel’s patterns. The text presents this puzzle as a rationale for the subsequent work of T.H. Morgan who related the inheritance of genes to newer understandings about chromosomes and their movement. Morgan’s work then led to further studies about gene linkage and mapping.

18. Findings Ecology chapters: • The process distinguished ecology chapters from the programs developed before the adoption of national science standards from the U.S. program developed after the adoption of standards, based on their levels of coherence. • The process did not distinguish the U.S. ecology chapters developed before national standards from the Chinese ecology chapters developed after the national standards were adopted. • Programs developed after the adoption of national science standards exhibited greater coherence than those developed before national standards, for both China and the U.S.

19. Findings Genetics chapters: • The process distinguished genetics chapters from programs developed before and after the adoption of national science standards, based on their levels of coherence. • Programs developed after the adoption of national science standards exhibited greater coherence than those developed before national standards, for both China and the U.S.

20. Conclusions • This adaptation of the AIM process is a promising method for evaluating the coherence of science instructional materials. • This method can be used in research studies, curriculum development, and professional development. For example, it could be used: • to study the relationship between the coherence of instructional materials and specific student learning outcomes, • by curriculum developers to strengthen the coherence of materials they are developing, and • in curriculum-based professional development to identify areas in programs where coherence should be strengthened by the teacher.

21. References cited on slides Powell, J. C., Short, J. B. and Landes, N. M. (2002) Curriculum reform, professional development, and powerful learning. In R. W. Bybee (ed.), Learning Science and the Science of Learning (Arlington, VA: NSTA Press), 121 -- 136. Schmidt, W. H. and Prawat, R. S. (2006) Curriculum coherence and national control of education: issue or non-issue? Journal of Curriculum Studies, 38 (6), 641 -- 658. Schmidt, W. H., Wang, H. C. and McKnight, C. C. (2005) Curriculum coherence: an examination of US mathematics and science content standards from an international perspective. Journal of Curriculum Studies, 37 (5), 525 -- 559.

22. To download this presentation, visit www.bscs.org/sessions