Evaluating Media Assistance Programs: What We Have Done and What We Have Learned

Evaluating Media Assistance Programs:What We Have Done and What We Have Learned Lee B. Becker & Tudor Vlad

Background • University of Georgia • Grady College of Journalism & Mass Communication • James M. Cox Jr. Center for International Mass Communication Training and Research • Conducts training programs • Focuses on evaluation Athens and the university in 1840 as depicted in the painting by George Cooke.

My Research Interests • Media effects • Opinion and learning • Organizational behavior and message construction • Characteristics of workers, including educational credentials • Sociology of education and labor markets • Needs assessment • Evaluation of impact

Evaluation of the Knight International Press Fellowship Program • Interviewed 531 persons in 8 European and 3 Latin American Countries • At 31 people in each of the 11 countries • Field work completed in 1999.

Basic Distinction • Assessing and monitoring program process. • Measuring and monitoring program outcomes.

Locus of Impact • On journalists • Attitudes • Behaviors • On organizations • On media system • On society overall

Problems with Self-Reports • Timing is important • Immediately after training is too soon • Duration of effect is variable • Likely to overestimate effect • To satisfy the program provider • To justify own investment of time and effort • Lacks reference • Most participants in training on projectile to change • Participant may not know the answer

Journalists in Philippines • Overestimated the amount of coverage they gave to the issues behind the conflict. • Underestimated the amount of coverage of the Manila media to issues other than the conflict.

Measurement Solutions • Go from the general to the specific • Now I want to start with a very general question. Please tell me some of the things that you learned from your participation in the XXX Fellowship that you consider at this point to have been most important to your work as a journalist. Of course, if you didn’t learn anything just tell me that. • OK. Now I’m going to ask about some specifics. You may already have touched on some of these, but I want to make sure I cover everything. Did your participation in the XXX Fellowship help you develop your skills in findings sources for stories you have reported since you returned? • Use what we can a “jab and probe” questioning technique. • Are there specific sources that you came into contact with through the XX Fellowship that you still use today? • IF YES: Who are some of those?

Validation Measures • Interviews others who also can attest to attitudes, behaviors and changes in both • Colleagues • Supervisors • Examples • Since you completed the XXX Fellowship, have you held any “brown bag” or other information discussions about your experiences in the program with other journalists? • As far as you know, has (NAME OF FELLOW) made any efforts to share her/his experiences at the Fellowship with others at your organization since she/he returned?

Design Solutions • Treat timing as a variable • We have compared short-term programs conducted across time • Timing matters—effect several years • But topic matters too and can offset lag • Develop a “control” group • True experimental design isn’t often possible • Imperfect “control” better than none • Develop before and after design

Some Control Examples • Control group made up of matched respondents • Successful applicants for subsequent ongoing programs • Those who work in similar settings • Can match based on lists • Can get respondent to help create a match • Control groups made up of matched organizations • Best control is the group itself • Before and after has limitation of systemic change

Post-Employment Training Common Part of Media Assistance • When evaluating journalism training component of media assistance, keep in mind • Measurement problems • Design problems • Use tactics to overcome them

Central Concerns of Media Assistance • Concern with Media Freedom or media independence • Concern with media performance • Concern with the connection between Media Freedom and media performance • Concern with the relationship between media performance and democracy

Mapping Media Assistance • Effort by Monroe Price and colleagues • Our purpose was to expand that • Shifted to narrower short-term goal • Rough estimate: $1 billion year spent on media assistance each year • Ultimately, valuable to understand media assistance this way

Two Actor Groups • Media assistance community • Donors • Media assistance providers • Targets of media assistance • Media monitors • Academic community • Political scientists • Communication scientists

Academic Community • Political scientists • Extensive literature on democratization • Extensive debate on meaning of democracy • Communication scientists • Extensive literature on media freedom • Extensive debate on meaning of media freedom • Relatively little empirical testing of contribution of media freedom to democracy or democratization

Different Perspectives of Advocates of Media Assistance and Political and Communication Scientists • Former believe media assistance leads to democratization • Role of the research is to document this • Scientists think all of these questions are open to examination

Model of Impact of Media Assistance Media Assistance Trained, Skilled, Motivated Journalists Media Organizations that Facilitate and Distribute the Work of Journalists Information Needed for Functioning of Democracy, Open Economy Independent Media System Functioning Democracy, Economy ? Institutions of Civil Society Civil Society Assistance

Does Media Assistance Lead to Trained, Skilled, Motivated Journalists? • Yes, probably • Evaluation is spotty • Process evaluation rather than impact evaluation • Often rely on self-reports of impact • Control groups are rarely used • Evaluations often not independent of funder, media assistance provider

Does Media Assistance Lead to Facilitative Media Organizations? • Maybe • Most evidence is indirect, based on observations of participants • Designs are inadequate • Many examples of failed investments • Problems of sustainability paramount

Does Media Assistance Produce Independent Media Systems? • Some limited concrete evidence it does • Comparative study by Steven Finkel and colleagues for USAID • Found that USAID Media investments across 165 countries led to gains in media freedom • Used Freedom House indicators • Considerable controversy over what is meant by media freedom

Do Independent Media Produce Information Needed for Democracy? • Almost no evidence either way • No real explication of what information is needed for democracy

Do Independent Media Produce Civil Society Institutions? • No one knows • Mixed evidence of the effectiveness of civil society assistance as well

Do Independent Media Foster Democratization? • Surprisingly little empirical evidence has been gathered • No understanding of what conditions might be placed on this relationship

Six Hypotheses • The media-supremacist position, which holds that media freedom and independence produce democracy. • The democracy-supremacist position, which holds that democratic reform determines and produces media freedom and independence. • The media-freedoms-are-an-element-of-democracy position, which argues that media freedoms are a part of democracy and, as such, have no causal force leading to democracy. • The null-effect position, which holds that there is no relationship between media freedom and democracy. • A media-freedom-hinders-democracy argument. • A democratization-hinders-media-freedoms stance.

Rozumilowicz Stages of Media Reform Linked to Stage Theory of Democracy

Media Democracy Link

What We Know About Media Freedom Measures? • Strong evidence of reliability across time • Strong evidence of reliability across measures • Some evidence of validity • Criterion • FH measures reflected changes in Warsaw Pact countries • Construct • IREX MSI related in predicted way in analysis of impact of hypercompetition on press performance

Relationship between Market Competition and Press Performance

Limitations of Media Freedom Measures • Focus on structure of media system • Limited evidence of performance • Limited notion of media independence • Focus on independence from government • Do not recognize pressures of commercial dependence • Largely ignore audience • Largely ignore “needs” of democracy

Additional Monitoring • Develop checklist of needs of democracy • A democratic state is a state in which all citizens have access to information about how the state operates. • A democratic state is one in which citizens have the ability to communicate to each other. • A democratic state is one in which conflicts are managed without resort to violence. • A democratic state is one in which representations of members of the state are presented in a way that foster appreciation and understanding.

Relationship between Press Freedom and Confidence in Media

Media Assistance Trained, Skilled, Motivated Journalists Media Organizations that Facilitate and Distribute the Work of Journalists Information Needed for Functioning of Democracy Independent Media System Functioning Democracy ? Institutions of Civil Society Civil Society Assistance Model of Impact of Media Assistance

Evaluating Media Assistance Programs: What We Have Done and What We Have Learned