Dr. Chris L. S. Coryn & Carl Westine September 16, 2010

1. EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn & Carl Westine September 16, 2010

2. Agenda Images and Purposes of Evaluation – Individual and small group, in-class activity 1 1. Images and Purposes of Evaluation A Brief History of Evaluation – A very brief overview of the development of the evaluation discipline 2 Basic Concepts and Core Principles – Definitions, purposes, types, and (some) uses of evaluation 3 Initial Discussion of Evaluation ‘Theory’– Is there a ‘theory’ of evaluation? 4 Question and Answer Session – Questions regarding the assigned readings or what we have discussed in class 5

3. 1. Images and Purposes of Evaluation

4. Images and Purposes of Evaluation Activity 1 • Instructional Objectives • Reflect on your assumptions and experiences regarding evaluation • Learn how different people perceive the value of evaluation • Understand the various definitions of evaluation • Two components to the activity • You will draw a ‘picture’ of evaluation • An analysis and discussion of evaluation definitions

5. Images and Purposes of Evaluation Activity 1, Part 1 • Draw the first image that comes to mind when you hear the word evaluation (15 minutes) • In small groups share your images, identify common themes, and write your themes on the flipchart (20 minutes) • Share your group’s themes (25 minutes) • Questions • Why do you think you drew that particular image? What experiences have you had that influenced what you drew? • If you view evaluation as a negative/positive endeavor, why?

6. Images and Purposes of Evaluation Activity 1, Part 2 • See handout • Questions • What are some commonalities among the definitions? • How do these definitions differ? • Which of the definitions resonate with your own evaluation experiences? • Are any of these definitions reflected in your images of evaluation? • How important is it to define evaluation, why?

7. Images and Purposes of Evaluation Activity 1, Debrief • What new insights did you gain from these activities? • How would you now describe evaluation to someone? • How is your current understanding of evaluation different from your understanding before this activity?

8. 2. A Brief History of Evaluation

9. A Brief History of Evaluation • In the beginning God created the heaven and the earth. • Then God stood back, viewed everything made, and proclaimed “Behold, it is very good.” And the evening and the morning were the sixth day. • And on the seventh day God rested from all work. • God's archangel came then, asking, “God, how do you know that what you have created is ‘very good’? What are your criteria? On what data do you base your judgment? Just what results were you expecting to attain? And aren't you a little close to the situation to make a fair and unbiased evaluation?” • God thought about these questions all that day and God's rest was greatly disturbed. On the eighth day God said, “Lucifer, go to hell.” • Thus was evaluation born in a blaze of glory. • —Michael Q. Patton (2008)

10. History of Evaluation Ancient Practice, New Discipline • Arguably, evaluation is the single most important and sophisticated cognitive process in the repertoire of human reasoning and logic • It is a natural, evolutionary process (i.e., its cognitive aspect—everyday evaluation) without which we would not survive • Earliest know examples • Product evaluation (e.g., tools) • Personnel evaluation (e.g., leaders, government officials)

11. History of Evaluation Program Evaluation in the United States • Tyler’s national "Eight-Year Study" (1933-1941), involving 30 secondary schools and 300 colleges and universities, that addressed narrowness and rigidity in high school curricula • Mainly educational assessment during the 1950s and early 1960s conducted by social scientists and education researchers • Johnson’s War on Poverty and Great Society programs of the 1960s (large investments in educational and social programs)—Head Start, Follow Through • Evaluation clause in Elementary and Secondary Education Act (ESEA)—evaluation became part of every federal grant

12. History of Evaluation Program Evaluation in the United States • Two U.S.-based professional evaluation organizations emerged in mid-1970s • Evaluation Network (E-Net) • Evaluation Research Society (ERS) • In 1985, the two merged to form what is now the American Evaluation Association (AEA) • Throughout the 1970s and 1980s, growing concerns were voiced about the utility of evaluation findings, in general, and the use of experimental and quasi-experimental designs, more specifically

13. History of Evaluation Program Evaluation in the United States • In the 1980s, huge cuts in social programs resulted from Reagan’s emphasis on less government involvement • The requirement for evaluation was removed or lessened for many federal grants during this period • During the 1980s, many school districts, universities, private companies, state departments of education, the Federal Bureau of Investigation (FBI), the Food and Drug Administration (FDA), and the General Accounting Office (GAO) developed internal evaluation units

14. History of Evaluation Program Evaluation in the United States • In the 1990s, there was an increased emphasis on government program accountability and organizations’ efforts to be lean, efficient, global, and more competitive • Evaluation was conducted not only to met government accountability but also to enhance effectiveness • In addition, it was during this period that an increasing number of foundations created internal evaluation units, provided support for evaluation activities, or both

15. History of Evaluation Program Evaluation in the United States • This slide was added after the lecture was posted, so not part of what you have! • Don’t know how I could have left this out, but I did! • Introduction of the Government Performance and Results Act (GPRA) in 1993 and its corresponding Program Assessment Rating Tool (PART) • GPRA and PART are still major mechanisms for evaluating government-funded programs to this day

16. History of Evaluation Program Evaluation in the United States • In 2001, the reauthorization of ESEA that resulted in the No Child Left Behind (NCLB) act is considered the most sweeping reform of education since 1965 • It has redefined the federal role in K-12 education by focusing on closing the achievement gap between disadvantaged and minority students • NCLB has had a profound influence on educational evaluation design and methods by emphasizing the use of randomized controlled trials (RCT) • To this day, the RCT debate is one of the most pervasive in evaluation and is beginning to take hold internationally and in sectors other than education (e.g., development)

17. History of Evaluation Professionalization • By 2006, there were more than 50 national and regional evaluation organizations throughout the world, most in developing countries • Although specialized training programs have existed for several decades, graduate degree programs in evaluation have emerged only recently • A few examples • Australasia • Africa • Canada • Central America • Europe (nearly every country) • Japan • Malaysia • United Kingdom

18. History of Evaluation Recent Developments • TODAY (literally), the National Science Foundation (NSF) announced the PRIME program (Promoting Research and Innovation in Methodologies for Evaluation) • This is very likely to be one of the most significant advances related to (funded) research on evaluation in the past several decades

19. 3. Basic Concepts and Core Principles

20. Basic Concepts and Core Principles Definitions • Evaluation is the act or process of determining the merit, worth, or significance of something or the product of that process • In addition, high quality, professional evaluation should be (or strive to be) ‘objective’ (free from distortion or bias), ‘non-arbitrary’, and ‘defensible’ – which has to do with criteria/values and standards (which we will discuss in detail next week)

21. Basic Concepts and Core Principles Decomposing the Classic Definition • The preceding definition consists of three predicates • Merit – Intrinsic quality; absent of context and costs • Worth – Synonymous with value; quality under consideration of costs • Significance – Synonymous with importance; merit and worth in context and with due considerations of other relevant contingencies

22. Basic Concepts and Core Principles Competing Definitions • Rossi, Freeman, and Lipsey define evaluation as “the use of social science research procedures to systematically investigate the effectiveness of social intervention programs” • Proponents of theory-driven evaluation approaches characterize evaluation as explaining “how and why programs work, for whom, and under what conditions”

23. Basic Concepts and Core Principles Competing Definitions • Advocates of the empowerment evaluation movement portray evaluation as “the use of evaluation concepts and techniques to foster self-determination” • The Organisation for Economic Co-Operation and Development designates evaluation as “the systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results...the aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability”

24. Basic Concepts and Core Principles Purposes • Formative: To improve • Summative: To determine disposition/inform decisions and actions • Developmental/proformative: To help develop an intervention or program (ongoing formative) • Accountability: To hold accountable (usually under summative) • Monitoring: To assess implementation and gauge progress toward a desired end • Knowledge generation: To generate knowledge about general patterns of effectiveness • Ascriptive: Merely for the sake of knowing

25. Basic Concepts and Core Principles Types of Use and Misuse

26. Basic Concepts and Core Principles Types • Most basic forms (for program evaluation) • Process: Assessment of everything that occurs prior to true outcomes • Outcome: Assessment of an evaluand’s effects (positive, negative, intended, unintended, and side effects) • Costs: Assessment of monetary and non-monetary costs, direct and indirect costs, and actual and opportunity costs • This includes comparisons with alternative ways for getting the same or similar benefits from about the same resources

27. Basic Concepts and Core Principles The 7 “P”s • Program evaluation • Policy analysis • Personnel evaluation • Portfolio evaluation • Product evaluation • Performance evaluation • Proposal evaluation

28. 4. Initial Discussion of Evaluation ‘Theory’

29. Evaluation Theory Theory Evaluation Theory • A set of interrelated constructs, definitions, and propositions that present a systematic view of phenomena by specifying relations among variables, with the purpose of explaining and predicting phenomena • Not to be confused with natural or biological theories, phenomena, predictions, explanations, principles, or laws, among others • (Normative) Evaluation theories describe and prescribe what evaluators do or should do when conducting evaluations (and their anticipated consequences) • They specify such things as evaluation purposes, users and uses, who participates in the evaluation process and to what extent, general activities or strategies, methods choices, and roles and responsibilities of the evaluator, among others

30. Evaluation ‘Theory’ General Classification Schemas • Shadish, Cook, and Levition’s (1991) five principles that undergird evaluation • Alkin and Christie’s (2004) evaluation theory tree that classifies and describes major theorists’ orientation First two today, final one next week Fournier’s (1995) more general ‘logic of evaluation’ (largely derived from Scriven’s earlier works)

31. Evaluation ‘Theory’ Shadish, Cook, & Leviton’s Classification Framework • Social programming: the ways that social programs and policies develop, improve, and change , especially in regard to social problems • Knowledge construction: the ways researchers/evaluators construct knowledge about social programs • Valuing: the ways values can be attached to program descriptions • Knowledge use: the ways social science information is used to modify programs and policies • Evaluation practice: the tactics and strategies evaluators follow in their professional work, especially given the constraints they face

32. Evaluation ‘Theory’ Alkin & Christie’s Theory Tree

33. Evaluation ‘Theory’ Alkin & Christie’s Theory Tree • The trunk is built on a dual foundation of accountability and social inquiry • These two areas have supported development of the field in different ways • The need and desire for accountability presents a need for evaluation • Accountability is broad in scope • It is not a limiting activity, but rather is designed to improve and better programs (and other things), society, and the human condition

34. Evaluation ‘Theory’ Alkin & Christie’s Theory Tree • The social inquiry root of the tree emanates from a concern for employing a systematic and justifiable set of methods for determining accountability • While accountability provides the rational, it is primarily from social inquiry that evaluation models (i.e., theories, approaches) have been derived • The main branch of the tree is the continuation of the social inquiry trunk • This is the evaluation as research, or evaluation guided by research methods, branch (designated METHODS)

35. Evaluation ‘Theory’ Alkin & Christie’s Theory Tree • Initially inspired by Michel Scriven, the VALUING branch firmly establishes the vital nature of the evaluator in valuing • Those on this branch maintain that placing value on objects is the central task of evaluation • Subsequent theorists extend the evaluator’s role to include facilitating the placing of value by others (e.g., Guba & Lincoln)

36. Evaluation ‘Theory’ Alkin & Christie’s Theory Tree • The third major branch is USE, which originated with the work of Daniel Stufflebeam and JoesphWholey, where evaluation was focused on decision making • Work done by theorists on this branch express a concern for how evaluation will be used and by whom • Michael Patton, more than any other theorist, has developed the most comprehensive, extensive theory of use

37. Evaluation ‘Theory’ Final Note • Despite increasing national and international efforts at establishing a recognized profession and discipline of evaluation, independent of other, more mature disciplines, most evaluators today are trained in psychology or education, and they tend evaluate with a “correspondently disciplinary flavor” (Cook, 1997)

38. This Week’s Vocabulary Entries from Encyclopedia of Evaluation • Assessment • Accountability • Auditing • Campbell, Donald T. • Cook, Thomas D. • Criteria • Evaluand • Evaluation • Evaluation Theory • External Evaluation • Formative Evaluation • History of Evaluation • Independence • Logic of Evaluation • Objectivity • Scriven, Michael • Shadish, William R. • Standard Setting • Standards • Summative Evaluation • Value-free Inquiry • Value Judgment • Values