Evaluation

1. Evaluation Kalilo and Raymond Human Performance Technology May 15, 2017

2. At the end of this seminar you will be able to: Define HPT Define the term evaluation State and discuss the three types of evaluation plus meta evaluation Pinpoint the purpose of evaluation

3. What is Human Performance Technology?

4. Introduction to Evaluation • To evaluate is to place a value on or judge the worth of a person, place, business, or event. In the context of PT, evaluation is a way to "compare results with intentions and delve into the usefulness of methods and resources so that we may move toward the required results." • The International Society for Performance Improvement (ISPI) has long supported "the integral role of evaluation in PT and in the ongoing functioning of any organization." • According to ISPI's HPT Model, evaluation is "one of the basic components of PT,” along with performance analysis, cause analysis, intervention selection and design, implementation, and change management.

5. Three types of evaluation The Meta -Evaluation process to the evaluation component. Formative Summative Confirmative

6. Formative Evaluation • Formative evaluation is diagnostic and is "used to shape or mould an ongoing process...to provide information for improvement...." • Formative evaluation is developmental and continuous; it begins during the analysis stage and running through the selection and design of interventions and sometimes into early implementation. • It’s cheaper

7. Summative Evaluation • Summative evaluation focuses on the effectiveness of a performance intervention after it is implemented. • Effectiveness is measured by immediate reaction and personal acquisition of knowledge and skills. • Summative evaluation takes place during implementation and change management. • Its expensive

8. Confirmative Evaluation • Confirmative evaluation identifies and explains long-term or enduring effects. • It is "the process of collecting, examining and interpreting data and information in order to determine the continuing competence of learning or the continuing effectiveness of instructional materials." • Confirmative evaluation builds on and goes beyond formative and summative to place a value on knowledge or skills transfer to the job, organizational impact, and return on investment.

9. Meta Evaluation • Meta evaluation is a process for "assuring and checking the quality of evaluations." • Through meta evaluation the PT practitioner or evaluator validates the formative, summative, and confirmative evaluations and acquires valuable insight into evaluation processes and products. • The best way to distinguish among the three types of evaluation and the meta evaluation process is to look at why and when each type of evaluation and meta evaluation should occur during the life cycle of the HPT Model.

10. Overview of the evaluation – Type – purpose – & Timing

11. What is the Purpose of evaluation The purpose of evaluation is to generate information that will accomplish outcomes into two folds: • Help the organization to value or judge the results of a performance • Trigger or support a decision regarding the performance, the performer, or, ultimately, the organization itself.

12. Purpose of evaluation conti… • As HPT future practitioners, you would agree with us that the purpose of evaluation is to affect decision making. • Therefore, feedback from evaluation can determine whether to maintain, change, or end the performance—or the performer. • To provide feedback that will trigger or support decision-making, the PT practitioner or evaluator may focus the evaluation on one or more areas of the organisation.

13. Evaluation focus The evaluation feedback will help or support decision-makers, or PT practitioner do the following: • Confirm that a particular procedure or treatment is being carried out as prescribed. • Confirm/dis the effectiveness of some treatment or intervention in order to decide whether to continue, expand, or eliminate the treatment. • Audit the current state of affairs. • Determine whether the cost of an activity is justified by its effects.

14. MATRIX OF EVALUATION DECISIONS What to Evaluate Why to Evaluate Matrix of Decisions When to Evaluate Why to Evaluate

15. Matrix decision • Matrix decision helps us explicitly define the purpose of the evaluation (clear, specific, and detailed but undistorted) • If all these are determined in advance, the resulting when, what, and how will unfold more smoothly. • However, we shouldn't think that evaluation is an afterthought or a one-time event. • Ideally, "evaluation may occur at any time and with any frequency. • It depends on the purpose of the evaluation."

16. Role of the PT Practitioner/Evaluator The PT practitioner or evaluator is responsible for performing five major tasks, irrespective of the type of evaluation he performs such formative, summative, confirmative, or meta.

17. Role of the PT Practitioner/Evaluator These tasks include: • Setting the scene of focus by establishing the purpose, goals, objectives, and scope of the evaluation. • Selecting the appropriate method (s) based on the purpose and the context (economic, political environment, available resources, time, and cost). • Collecting all the information that is feasible, available, and relevant. • Analysing the data using sound qualitative or quantitative methods. • Reporting the findings after first determining who needs to know, what they need to know, how best to inform them, and how often to keep them informed.

18. Formative Evaluation

19. Formative Evaluation • It starts during the analysis stage, continues through the selection and design of interventions and, may extend into early implementation. • Formative evaluation is a set of quality control methods to improve, not to prove...effectiveness. • It can be designed to improve the performance of individuals and/or organizations.

20. What are you testing with Formative evaluation? It test weather:. • Designers/developers are doing what they promise they will do. • All things grounded in the mission and values of the organization. • All are aligned to the objectives performance improvement effort. In other words formative evaluation is called continuous improvement or quality control.

21. Conducting a Formative Evaluation There are four methods for conducting formative evaluation: • Expert Review • One-to-one • Small Group/s • Field Test Although these methods are traditional to the field of instructional systems design, they may also be adapted to the PT environment.

22. These methods are used to review the entire performance intervention package—products and procedures—throughout the entire PT process. 1. Expert Review • A performance expert provides information that help in the selection or design of the intervention before implementation. • The PT evaluator "reviews the review," clarifies any remaining issues, and revises the intervention. 2. One-to-One Evaluation • A performer or user reviews components of the selected or designed intervention before implementation. • The PT practitioner takes part in the review and revises the intervention as needed.

23. Methods are used conti…. 3. Small-group Evaluation • Potential performers or users review components of the selected or designed intervention before implementation. • The PT practitioner may or may not participate directly in the review, but is responsible for establishing what the group will focus on during the review, clarifying issues arisedduring the review, and making the necessary revisions.

24. These methods are used conti…. 4. Field Test Evaluation • The selected or designed intervention is tried out with target performers/users before full-scale implementation. This method is frequently followed by a debriefing session involving the PT practitioner, who then makes any necessary revisions.

25. Alternative evaluation 1-A Self-evaluation • Several members of the design team evaluate the intervention before presenting it to experts for evaluation. • This process is frequently called an internal review done before presenting material to the client or external review.

26. For this to work effectively, the developer must complete the following: • Develop a set of evaluation criteria. • The criteria may be the same or different from the criteria set for the expert, performer, or client review, but it should include all the items the client or external reviewers will focus on. • Set the intervention material aside for several days to gain distance from the intervention's content and intent. • Record both positive and negative feedback.

27. Alternative evaluation Cont… 1-B Panel Reviews • The PT practitioner directs and structures the evaluation process, preparing a set of questions to guide two or more groups of experts. • Ideally the experts review the package before meeting with the PT practitioner so that they can focus on areas of concern during the meeting. • The PT practitioner facilitates the meeting and records the outcomes.

28. Alternative evaluation Cont… 2-A Two-on-One Evaluation • Two performers review the performance intervention package with the PT practitioner. • The performers discuss their reactions as they move through the processes and products that compose the package.

29. Alternative evaluation Cont… 2-B Think-aloud Protocols • This method involves only one performer at a time. The performer walks through the package and verbalizes all of his or her thoughts and reactions. • The PT practitioner or evaluator prompts the performer to continue thinking aloud whenever the performer becomes silent.

30. Alternative evaluation Cont… 2-C Computer Interviewing • Computer interviewing is the visual counterpart of telephone interviewing and a very effective use of e-mail, bulletin board, and chat room technology. • The evaluator can send, retrieve, analyze, and respond to the email, or use a software program that automatically sends questions, collects and analyzes responses to open or closed-ended questions, and even generates and distributes a customized report.

31. Alternative evaluation Cont… 3-A Evaluation Meetings • Evaluation meetings bring performers together to review and discuss the performance improvement package. - For example, trainers may hold an evaluation meeting after a workshop to gather feedback from the participants to update the next session.

32. Alternative evaluation cont… 3-B; 4-A Computer Journals and Networks • In a situation where the performance intervention uses network software, the PT practitioner or evaluator may gather information from online journals. • However, he keeps a journal to record reactions to the software, and makes suggestions for improvement.

33. Alternative evaluation cont… 4-B Rapid Prototyping Rapid prototyping is an alternative development evaluation process. During the development of the performance improvement package, the designer or developer works on one component at a time, and may simultaneously analyze, design, develop, and implement instead of working in a linear fashion. • 1. Analyze, design, and develop one component of the performance improvement package. • 2. Develop the support products required to implement the component. • 3. Field test the component immediately with experts or performers. • 4. Revise as needed. • 5. Repeat the process until the entire performance improvement package is completed.

34. Let’s what this eye breaker

35. Summative Evaluation

36. Summative Evaluation • Summative evaluation is the next logical step in the evaluation process and helps the organization determine whether or not to put the "soup" on its performance improvement "menu." • "When the cook tastes the soup, that's formative; however, when the guests taste the soup, that's summative.“ • Summative evaluation is the most objective way to document the strengths and weaknesses of a performance intervention package. • However, summative evaluation is mostly omitted because it requires time, money, skilled resources, and the commitment and support of senior management.

37. Summative Evaluation Level 1-Reaction Evaluation Question • Did the participants like the training? Timing and Strategy • At the end of each training day the facilitator will use the Audience Response System (ARS) to gather participant reaction to the workshop content, presentation, instructional aids, and their self-reported ability to use the new learnings or skills. • Because the Road Show is a work in progress, the designer will use the information to revise the program as needed.

38. Summative Evaluation Level 2-Learning Evaluation Question • Did the participants learn the information/acquire the skill presented in the training? Timing and Strategy • At the end of each module, the facilitator will use performance tests to test the participants' knowledge.

39. Summative evaluation Level 3-Behavior Evaluation Question • Can the participants apply what they have learned to create business as "unusual"? Timing and Strategy • At the end of each module, participants will complete an action plan describing how they will use the new knowledge or skill. The evaluator will follow up within 30–60 days by interviewing participants, sales managers, co-workers, customers, and dealer principals.

40. Summative evaluation Level 4-Impact - Evaluation Question • Did the road show have a positive impact on the dealerships and division? Timing and Strategy • Six months after the training the evaluator will interview participants, sales managers, co-workers, customers, dealer principals, and division stakeholders. • The evaluator will also perform a before-and-after comparison of sales figures and customer service documentation.

41. Confirmative Evaluation

42. Confirmative Evaluation • Confirmative evaluation is "a new paradigm for continuous improvement." • Twenty years ago Misanchukintroduced confirmative evaluation as a logical next step to formative and summative evaluation of instructional materials or Iearners. • Today, practitioners within the instructional and PT environments are beginning to accept the fact that "Quality control requires continuous evaluation including extending the cycle beyond summative evaluation ." • Confirmative evaluation builds on the findings and recommendations generated during formative and summative evaluation. It enables evaluators to confirm the continuing worth of performance improvement packages and performers and helps organizations decide how to manage continuous performance improvement efforts.

43. Purpose • To provide continuous quality control over the life cycle of the performance improvement package. • Placing a value on the endurance of the performance improvement package and by helping the decision makers establish what to do next.

44. Purpose cont… • Confirmative evaluation identifies, explains, and confirms the value of the performance improvement intervention over time. "Enduring or long-term effects refer to those changes that can be identified after the passage of time and are directly linked to the performance improvement package."

45. EQUATION FOR CONFIRMING THE VALUE OF A PERFORMANCE INTERVENTION How the value of a performance intervention is equal to the continuing competence of the performers who participate in the intervention and the continuing effectiveness of the entire performance improvement package including products and processes. Confirmative evaluation also helps decision makers select ways to manage the performers and the performance improvement package over time.

46. SIX OPTIONS FOR MANAGING THE PERFORMER AND THE PERFORMANCE IMPROVEMENTPACKAGE The table also suggests that even if the level of performance meets performance standards or the performance package is effective, there is always the option to exceed the standards, increase the effectiveness, and delight the customer.

47. In addition to confirming the competency of the performers and the effectiveness of the performance improvementpackage, there are other reasons why PT practitioners or evaluators should conduct confirmative evaluation: 1. Confirmative evaluation may be used to link the intervention to broader accomplishments that directly affect the organization and to establish actual costs and benefits. 2. Long-term formative evaluation, or confirmative evaluation, not only supports and strengthens continuous improvement efforts within an organization, but can also "feed back valuable hypotheses to researchers in pursuit of a better understanding of human behavior, performance, and accomplishment," helping to build a scientific base for PT.

48. Conducting a Confirmative Evaluation The following guidelines for conducting long-term formative evaluation also apply to the process of confirmative evaluation: 1. Collect data every six months. 2. Be as unobtrusive as possible; for example, incorporate data collection into ongoing activities such as performance appraisals or audits. 3. Focus on the typical effects of the package such as long-term productivity and payoff. 4. Use a form of expert review to verify that the content of the package is still valid, timely, and aligned with the latest corporate policies.

49. Methods for conducting confirmative evaluation may include, but certainly are not limited to, the following: • Checklists (desired or optimal performance) • Interviews (performer, supervisor, coworkers, customers, etc.) • Observation (performance tests) • Rating scales (quantifies level of actual performance) • Assessments (printed tests of knowledge and skill; pre-and post-tests) • Review of existing information (before-and-after data, safety reports, production reports, suggestions etc.) These and other methods may be used separately or together. For example, in the following situation, the performance standards were known and the PT practitioner planned to train observers to use checklists and a rating scale to conduct confirmative evaluation.

50. Confirmative Evaluation Model • In 1997 Performance Improvement published a model of "confirmative evaluation-a new paradigm for continuous improvement.“ • The model divides confirmative evaluation into four phases: planning, doing, assessing, and improving. • The focus of the model is on "continuous improvement, a principle driven by customer expectations."