S519: Evaluation of Information Systems

S519: Evaluation of Information Systems Result D-Ch10

Putting them together • Now we are ready to write our evaluation report. • Basically we are going to fill our content to the checklist boxes we learned in lec2.

The Key Evaluation Checklist (D-p. 6-7) I. Executive Summary II. Preface III. Methodology 1. Background & Context 2. Descriptions & Definitions 3. Consumers 4. Resources 5. Values 10. Exportability 7. Outcome Evaluation 6. Process Evaluation 8 & 9. Comparative Cost-Effectiveness 11. Overall Significance 14. Reporting & Follow-up 12. Recommendations & Explanations 13. Responsibilities 15. Meta-evaluation

KEC Preliminary checkpoints • To orient readers to the basics of : • what the evaluation is and • why the evaluation was done • What the main approach was

I. Executive summary • Shows a short overview (1-2 pages) of what was evaluated and what the main findings were. • You can consider to include the followings: • Very short description of the program • The overall conclusion about the quality or value of the evaluation • The graphical profile of the evaluand‘s performance (see Exhibit7.6) • Several (<7) bullet points about the most important strengths and weaknesses.

II. Preface • Address the following questions: • Who asked for the evaluation and why? • What are the main evaluation questions? • Is this a formative or summative evaluation? • Who are the main audiences for this evaluation report? • Mainly based on your work in Ch2 • Please always provide justifications for your conclusions

III. Methodology • The main methodologies used in the evaluation • How you set up your experiments • How you collect data • How you analyze your data? • How you draw conclusion • And each with why.

KEC foundations checkpoints • To represent all the initial ingredients • Background • Context • What it is • Whom it serves • The nature and limitation of any resources • Where the values come from

1. Background and Context • Incude just enough information for readers to be able to understand the basic retionale for the program • Contents are: • Why did this program or product come into existence in the first place (needs assessment) • How is the evaluation supposed to address the needs and problems? • What are the main context of evaluands (e.g. Physical, economic, political) to facilitate or constrain the evaluation?

2. Description and definitions • Describe your evaluand in enough details • Definition of important terms

3. Consumers • Identify the actual and potential recipients or users of the evaluand • Describe consumers in geographic location, demographics • Downstream impactee

4. Resources • Describe the available and important resources (which helps to understand the conclusion of the evaluation) • Funds/budget • Physical space • Experience • Networks

5. Values • To convey to the audience how the evaluation team determinded what should be considered as „valuable“ or „high quality“ for the evaluand. • Justify the validity

KEC Sub-evaluation checkpoints • Getting evaluation explicitly

6. Process evaluation • The issues of evaluation, content and implementation (see Ch4) • Ethics, the priniciple of equity and fairness • Consistency with professional and scientific standards • Efficiency (i.e, minimal wasted effort or resource) • Needs of consumers (e.g., timeliness, learning style, current knowledge) • Needs of staff (e.g., performance on tasks and activities)

7. Outcome evaluation • The outcomes are what happened to the consumers as a result of coming into contact with the evaluand • Intended and unintended • Short-term and long-term effects • A good way to do • List the main outcome dimensions with a brief explanation of how the list was generated. • Rate each dimension on importance • Explain the used methods (ch7) • Rate the each outcome dimension (ch8) • Provide the justification (evidence, logic, methods) for your rate

8&9. Comparative cost-effectiveness • It is important to know whether the evaluand is cost-effective • See D-p62-63

10. Exportability • Whether your evaluation could also have value outside its current context. • It is always worth considering the possibility

KEC conclusions checkpoints • Combining Checkpoints6-10 to draw an overall conclusion

11. Overall significance • Using synthesis methodology to draw a summary covering Checkpoints 6-10 • Graphical representation (e.g., Exhibit7.6) • Summary of strengths and weaknesses

12. Recommendations and explanations (possible) • Recommendations for furture improvement

13. Responsibilities (possible) • Who is repsonsible for good or bad results • You have to be highly skilled

14. Report and support • Who get the copies of the evaluation reports and in what forms • What are follow-ups to ensure that the findings are used

15. Meta-evaluation • Self review of the evaluation itself • The validity of their conclusions • Utility to relevant stakesholders • The methodology used for evaluations • Credibility • Cost • Lessoned learned during this evaluation.

Exercise • Go back to your evaluation group • Discuss all these checkpoints and how are you going to put all your result together

Exercise • Go through these checkpoints • Write down 3 checkpoints which are most important to you and state why? • Discuss with your partner • Form the group and select 3 most important checkpoints, why and • Present your group conclusion and your suggestions in poster • Hang your group poster and people can shop around poster • Vote for the best poster

S519: Evaluation of Information Systems