Evaluating Research and Impact

1. Evaluating Research and Impact Guri Sohi John P. Morgridge Professor and Chair Computer Sciences Department University of Wisconsin-Madison

2. Process of Evaluation • Metrics • Mechanisms

3. Metrics • A diversity of metrics, depending upon type of research and area. • Funding • Publications • Artifacts • Hardware or software or other

4. Funding • Sometimes necessary but definitely not sufficient • Sometimes confused as an end vs. a means • Questionable metric for longer-term direct impact • May have indirect impact, e.g., more students trained

5. Publications • Quality vs. quantity • Quantity is easily measured; quality harder • Temptation to rely on quantity • Use community to help assess quality • Create and cultivate selective publication venues • Larger community helps with assessment

6. Impact • Short-term vs. long-term • Temptation to use short-term • Biases toward flavor of the day • Measures of impact • Citations • Adoption in commercial products • Citations may be lacking • Success of former members of research group • Much longer term

7. Citations • Number of citations vs. quality of citations • Citation indices, e.g., Science Citation Index or Is Highly Cited • May be less useful for CS due to publication venues considered • Modern versions of citation indices e.g., Google Scholar) • Only one measure: highly dependent upon area