Presented by: Mike Russell (May 2013) USE ARROW KEYS TO MOVE FORWARD OR BACKWARD.

1. Quantitative and Scientific Reasoning and Literacy (QSRL) Student Learning Outcome (SLO) assessment discovery committee Presented by: Mike Russell (May 2013) USE ARROW KEYS TO MOVE FORWARD OR BACKWARD.

2. Committee members • Susan Bjerke (Biology) Jane Carpenter (Nursing) • Lisa Sharp Elles (Chemistry) Zach Frank (Allied Health) • Linzi Gibson (Psychology) Brandon Gillette (Philosophy) • Young Sub Kwon (Kinesiology) Hwa Chi Liang (Math) • LinseyModdelmog (PoliSci) Nan Sun (CIS) • Brian Thomas (Physics) myself

3. Committee’s Task • By the end of this semester, recommend one or more instruments that assesses Quantitative and Scientific Reasoning and Literacy (QSRL). • Do NOT reinvent the wheel. There is no need. Numerous instruments for this area already exist. • The instrument will be used to assess the QSRL SLO for exiting/graduating students – the end product of a WU education. • i. Not every graduating student will be assessed. • ii. Only a percentage of our students will be sampled.

4. “Assessment instrument” • It could be an assignment. • It could be an assignment given in a University requirement course (EN 300), or • given to a randomly selected group of students. • It could be a standardized test. • The cost of the assessment instrument should not be considered a factor. • We want the best instrument available.

5. Requirement • We MUST recommend an assessment instrument that allows us to compare our results with those of other institutions and national norms. This is MANDATORY. • We need to be able to draw comparisons between ourselves and others (e.g., peer institutions). • The availability of national norms would be useful.

6. Best-case scenario • We want an assessment instrument that is high in reliability and validity. • Reliability = consistency • We want an instrument that produces a score that is not affected by external events (e.g., weather, time of semester). • Validity = truth • By validity, it is meant an instrument that truly and accurately measures quantitative and scientific reasoning and literacy.

7. Washburn University QSRL SLO definition • “Quantitative and Scientific Reasoning and Literacy. Quantitative reasoning involves the ability to work with numerical data and the higher-order thinking skills required to make and understand mathematical arguments. Scientific literacy involves the acquisition and application of skills and knowledge necessary to understand the nature and content of science, and to evaluate scientific arguments using evidence-based reasoning. Students will be able to understand and develop arguments supported by quantitative evidence, clearly communicate those arguments in a variety of formats (using words, tables, graphs, statistical inference, mathematical equations and functions, etc., as appropriate), and apply mathematical and scientific methods to solve problems from a wide array of contexts and everyday situations.”

8. Your task • The QSRL SLO assessment discovery committee has recommended three (3) potential instruments. • Your task is the same as theirs: Recommend one or more of the following assessment instruments that you believe we could use to assess Quantitative and Scientific Reasoning and Literacy. • At the end of this presentation, exit this PowerPoint and access the link to a survey where you can cast your vote as to which instruments you prefer.

9. Option 1Madison Assessment Test • The Madison Assessment test actually has both a Quantitative Reasoning and a Scientific Reasoning section. • Lets look at the quantitative section and then the scientific reasoning section.

10. Madison Assessment Test: Quantitative section • 26-item multiple-choice test designed to assess the quantitative reasoning of students who have completed their general education requirements, and measures the following learning objectives: • (1) Use graphical, symbolic, and numerical methods to analyze, organize, and interpret natural phenomena. • (2) Discriminate between association and causation, and identify the types of evidence used to establish causation. • The QR is recommended for use with general education program assessment and evaluation. The QR is designed to be content-free, meaning that responding correctly to the items does not require specific content knowledge of any domain of science.

11. Madison Assessment Test: Quantitative section • paper-and-pencil and online options • 25-minute long test • reliability = 0.64, 0.66 • seems to be widely used • Cost: $8.00 per student

12. Madison Assessment Test: Scientific Reasoning section • 49-item multiple-choice test developed by science and mathematics university faculty. This instrument was designed to assess the scientific reasoning skills that college students may obtain through a general education curriculum. • Intended to measure learning in scientific reasoning for undergraduate college students. • Designed to be content-free, this instrument should be appropriate for students in any general education science curriculum.

13. Madison Assessment Test: Scientific Reasoning section • paper-and-pencil and online options • reliability = 0.73, 0.76 • seems to be widely used • Cost: $8.00 per student.

14. Option 2Collegiate Learning Assessment (CLA): Quantitative Reasoning • The assessment is divided into 3 parts—analytic writing, performance tasks, and biographical information. Two types of writing tasks are administered. The first, make an argument, invites students to present an argument for or against a particular position. The second type of writing task asks students to evaluate an argument. • The Collegiate Learning Assessment (CLA) asks students to take three critical thinking tests. One of these, the "Performance Task," contains a series of prompts which ask for short essay response. While not all performance tasks include quantitative information, most do. And the kinds of critical thinking skills involved (e.g. distinction between association and causation) seem to reflect common QR learning goals. The designers of the instrument stress that it is useful for assessment at the institutional level-not at the individual student level.

15. Collegiate Learning Assessment (CLA): Quantitative Reasoning • Written essay exam – completed online. • Time requirement: 2 hrs, 45 mins (165 minutes) • Scored by company (cost seems to vary $28 - $32.50/student). • Presents realistic problems that require students to analyze complex materials and determine the relevance to the task and credibility. Students' written responses to the tasks are evaluated to assess their abilities to think critically, reason analytically, solve problems and communicate clearly and cogently. • Provides national norms – 500+ institutions (ESU, WSU, PSU). • Interrater reliability (.65 - .80)

16. Option 3Test of Scientific Literacy Skills (TOSLS): Scientific Reasoning • 28-items, can be accessed as a link from the VPAA’s web site • "freely available, multiple-choice instrument that can be readily administered and scored in large-enrollment Gen Ed courses" • Reliability for the TOSLS very similar to the Madison test of scientific reasoning (0.731 - 0.748) • Cost: none (free) • Easy to grade • Not sure how popular it is though.

17. So what do you think? • We need to know what you think. • Now it is time to cast your vote. Exit this presentation and review the various rubrics and instruments then complete the survey. The survey will take about 2 – 3 minutes. • Your recommendation(s) will be submitted to the VPAA.

18. “Thank you!!!!” • Please let me know if you have any questions, comments, or concerns. • Contact me via email (mike.russell@washburn.edu).