1 / 47

Content Analysis Workshop

Content Analysis Workshop. Minneapolis, MN Nov 7-9, 2007. Describing the Content of. Standards & Assessments. John L. Smithson , Director, Measures of the Enacted Curriculum Alissa Minor , Projects Manager, Measures of the Enacted Curriculum

farren
Download Presentation

Content Analysis Workshop

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Content Analysis Workshop Minneapolis, MN Nov 7-9, 2007 Describing the Content of Standards & Assessments John L. Smithson, Director, Measures of the Enacted Curriculum Alissa Minor, Projects Manager, Measures of the Enacted Curriculum Wisconsin Center for Education Research, University of Wisconsin-Madison johns@wcer.wisc.edu

  2. The Goal To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of these three domains of a standards-based approach to education reform and their relationship to one another.

  3. The Goal To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of these three domains of a standards-based approach to education reform and their relationship to one another.

  4. Teacher Reports Alignment Analyses Content Analyses Content Descriptions Needs Assessment Content Descriptions Curriculum Management SEC Taxonomy Monitoring Change Program Evaluation

  5. Uses of Content Analysis Results Descriptive: (Tile Charts and Content Maps) • Visual, curriculum-based descriptions of Instructional Targets for teacher reflection, discussion and planning. Analytic: (Alignment) • Predict student achievement gains • Control for content to examine other factors • As an outcome measure for change over time • Examine alignment of Standards & Assessments

  6. To Describe Instructional Content SEC utilizes a two-dimensional taxonomy based on: Topic by Cognitive Demand

  7. The Content Matrix

  8. … adding levels of relative emphasis yields a 3-D construct

  9. Content Map Data Displays

  10. To Facilitate Comparisons Assessments Curriculum Alignment Standards

  11. Uses of Content Analysis Results Descriptive: (Tile Charts and Content Maps) • Visual, curriculum-based descriptions of Instructional Targets for teacher reflection, discussion and planning. Analytic: (Alignment) • Predict student achievement gains • Control for content to examine other factors • As an outcome measure for change over time • Examine alignment of Standards & Assessments

  12. Alignment as a Quantity Aligning Tests to Standards 0.27 (Avg. Alignment: Test to Standard) Range of Alignment: Test to Standard) 0.14 0.40 0.00 0.25 0.50 0.75 1.00 State U Grade 8 Mathematics Alignment: Test to Standard (0.23) (Based on results for 10 states, across grades 4, 6 and 8: SEC Collaborative 2003)

  13. Instructional Alignment Instruction to Standards & Assessments Fine Grain Standards 0.05 0.17 0.29 Min. Avg. Max. 0.00 0.50 0.75 1.00 0.25 0.03 0.19 0.31 Min. Avg. Max. Assessments Based upon results for 168 teachers, across 3 states: MSP PD Study 2004

  14. Explaining variation in student learning gains

  15. Alignment Analyses for School Improvement Using alignment as an outcome measure Alignment Index: Instruction to Standards Mathematics Across 4 Districts Counts Treatment 99 Control 124 Leaders 16 (Measuring change in alignment over time)

  16. Content Analysis Procedures Exploring the Dimensions of Content But first…. Let’s take a 10 minute break!

  17. The Two Dimensions Of Content What students should know [Topics] And… Be Able to Do [Expectations for student performance]

  18. Describing the Cognitive Domain How Many Categories? 3 SCASS Science 4 DOK (Webb) SEC 5 6 Bloom’s

  19. Dimensions of Knowing & Inquiry Acquire Use Extend (From: Dimensions of Knowing and Inquiring about Science, State Collaborative on Assessments & Student Standards Science Project, Council of Chief State School Officers, 1997)

  20. Depth of Knowledge Level 1 2 3 4 Recall Skill/Concept Strategic Thinking Extended Thinking * Webb, N. 1999. Alignment of Science and Mathematics Standards in Four State. NISE Research Monograph #18. Madison:Wisconsin Center for Education Research.

  21. Exploring Cognitive Demand Acquire Use Extend Recall Skill/Concept Strategic Thinking Extended Thinking

  22. Cognitive Demand (or Expectations for Student Performance) Acquire Use Extend Recall Skill/Concept Strategic Thinking Extended Thinking Memorize Perform Procedures Demonstrate Understanding Conjecture, Generalize Prove Solve non- routine/ make connections Memorize Conduct Investigations Communicate Understanding Analyze Information Apply concepts /make connections Recall Perform Procedures Generate /Demonstrate Analyze/ Investigate Evaluate

  23. Cognitive Demand (or Expectations for Student Performance) Acquire Use Extend Recall Skill/Concept Strategic Thinking Extended Thinking Memorize Perform Procedures Demonstrate Understanding Analyze Information Evaluate/Apply Recall Understanding Application Analyze Evaluate Create

  24. Exploring Cognitive Demand CgD Immersion Activity • Organize into Groups/Tables • Each Table w/ CgD Pie • Each Person w/ Cgd Descriptors Step 1: Place CgD cards on Pie Slices face-down Step 2: Turn cards over: ID agreements e.g. 2 cards w/ same descriptor in same slice if Group Agrees ... discuss key words if not … Discuss … operational definition to distinguish Step 3: Discuss disagreements if consensus reached put in envelope / if not, set aside

  25. Content Analysis Materials • Cognitive Demand List • Topics Lists • Comments & Suggestions Worksheet • Coding Forms • Documents to be analyzed

  26. Cognitive Demand Lists • Five categories of cognitive demand • Slightly different for each subject • Each category is defined by a list of descriptors • The list of descriptors are not exhaustive • Each category stands on its own • Each category has an associated letter (B-F)

  27. Dimension A: Content Topics Topics List (In your packet of material) Organized at two levels: Content Areas (16 for Mathematics) (27 for Science) (14 for ELAR) Topics (identified by number) (182 Mathematics Topics) (211 Science Topics) (114 ELAR Topics) Plus: non-specific & other

  28. Topics Lists • Topics Lists • Mathematics • Science • English Language Arts & Reading • Cover grades K-12 • Organized into Content Areas • Topics & Content Areas have an associated #

  29. Comments & Suggestion Worksheet • One for each reviewer - more available • Use to: • Record coding conventions/decision rules • Suggest/identify additional topics not listed • Suggest/identify additional CGD descriptors • Provide other comments & suggestions • Be sure to turn in at end of workshop (and with mail-in materials, as necessary).

  30. Coding Forms • Assessment Coding Forms • Benchmark Coding Forms • Each is used to record content descriptions • Each content description consists of • A topic number • A cognitive demand category letter

  31. Assessment Coding Forms

  32. Standards Coding Forms

  33. Standards Coding Forms

  34. Practice Coding Exercise Content Analyzing Assessments (Three code maximum)

  35. Practice Coding Exercise Content Analyzing Standards (Six code maximum)

  36. The Content Analysis Process ‘Coding’ Teams of 4-5 Content Experts Independent Coding by each Analyst w/ Group Discussion Should not be necessary to discuss every item – select by team Goal for Process: Generalizability not Inter-rater Reliability Pick-up and return documents / coding sheets to Alissa Sign & return to Alissa non-disclosure forms

  37. A neutral content language Topics by Cognitive Demand Content Analysis Workshop The intended curriculum: State content standards—What students should learn The assessed curriculum: State (and other) assessments—tested learning The learned curriculum: Student outcomes based on school learning The enacted curriculum: What teachers teach

More Related