Developing Scientific Evaluation Abilities in Students

1. Developing Scientific Evaluation Abilities in Students Aaron R. Warren Michael Gentile Alan Van Heuvelen Rutgers University, New Jersey

2. Basic Goal • Instructional Goal: We want to help students develop “evaluative abilities” • Specifically, want students to gain proficiency at using some general strategies to critically judge and check their own thought & work. • Motivation: Evaluation is crucial to learning. Developing evaluation abilities should enhance learning.

3. Evaluative Strategies: Dimensional Analysis • Dimensional Analysis: • Question: Is an equation self-consistent? • Strategy: Check dimensions of each term. • Example: • Do a dimensional analysis to check whether F = mv is self-consistent.

4. Evaluative Strategies: Limit/Special Case Analysis • Limiting & Special Case Analysis: • Questions: How is a given idea related to other ideas? Is a given idea consistent with other ideas? • Strategy: Identify & compare relevant models/equations/results (mathematically & conceptually).

5. Example: Limit/Special Case Analysis • Check the limit case of r = ∞ for the quantitative model Fnet = (-mv2/r)r. • What does Fnet equal in this case? • What does the motion of an object look like when it moves uniformly along a circle of infinite radius? • Are your answers to each of the previous questions consistent with one another? Explain why/why not.

6. Evaluative Strategies: Assumption Analysis • Assumption Analysis: • Questions: Can a given model be used to accurately predict some empirical quantity? What may cause the prediction to be inaccurate, and how? • Strategy: Identify all assumptions that must be true for the model to give accurate predictions for the given situation. Identify factors which may cause violations of these assumptions.

7. Example: Assumption Analysis • You have been hired as an engineer for a candy company. For your first job, you are asked to design a hose and nozzle for spraying chocolate on candy bars. You decide to model the flow of the chocolate using Bernoulli’s equation. What assumptions are required in order for Bernoulli’s equation to accurately model the flow of chocolate through a hose and nozzle?

8. The Role of Evaluation in Learning • Self-evaluating one’s work is an integral part of science. Physicists always question new ideas – is an idea self-consistent, is it consistent with other ideas, and how does it relate to reality? • These questions, and the strategies used to answer them, are crucial to how physicists learn and do physics. • So students are more likely to develop ‘expert-like’ knowledge if they learn to value and use these evaluative strategies. Also, students can learn how to learn (meta-cognition).

9. Developing & Using Evaluation Tasks • Tasks used as formative assessments • Students work on tasks in lecture, labs, recitations, homework. Given feedback from peers and instructors. • Collecting Data: Two types of exams • Lecture: Multiple choice quantitative problems • Labs: Short answer conceptual problems • The lab exams also contained evaluation problems.

10. Research Hypotheses • Expect scores for evaluative abilities to correlate with conceptual and quantitative scores. • Strong evaluative ability necessary for robust understanding of physics. • These correlations should be context-independent • The impact evaluative abilities have on learning should not be specific to any topic.

11. Data Analysis: Coding the Data • Developed, tested, and used scoring rubrics to classify student responses to short-answer exam problems. • Rubrics assign scores of 0-3 to student responses in order to measure their evaluative abilities. • Inter-rater reliability for each rubric varied between 91% and 99%, with average of 96%.

12. Data Analysis: Testing Correlations • Subjects: ~150 life science majors.

13. Discussion • Consistently significant correlations between evaluative scores and exam scores. Supports the theory that evaluative abilities are necessary for robust understanding. • Correlations for the first and fourth exams are not significantly different. Supports hypothesis that the role of evaluation in learning is context-independent.

14. Moral of the Story…. • It is worthwhile to help students learn to evaluate their own thought and work. Evaluation plays a critical role in the construction of knowledge! We need to make students aware of this, and engage them in activities which promote evaluative abilities. • Evaluation activities and rubrics are available online. • email Aawarren@physics.rutgers.edu for information.