Developing and pilot-testing a culturally-competent evaluation model for minority health projects in Ohio

1. Developing and pilot-testing a culturally-competent evaluation model for minority health projects in Ohio Manoj Sharma, University of Cincinnati Robert Fischer, CaseWestern Reserve Univ. Carla Clasen, Wright State University Peter Leahy, University of Akron Betty R. Yung, Wright State University Lucinda Deason, University of Akron Fatima Perkins, United Way of Greater Cleveland Session 3369, Board 2 November 6 (M), 2006, 4:30 pm

2. Background of the Ohio Commission for Minority Health (OCMH) • In July 1987, the Ohio General Assembly, passed Amended Substitute House Bill 171, creating the Ohio Commission on Minority Health (OCMH) • The Commission is an independent state agency with a biennial appropriation of approximately $3.5 million from general revenue funds. • Innovative and culturally specific projects are funded for up to $200,000, for a two-year period. • Between 1988-2001 OCMH funded grants totaling approximately $12 million • OCMH has funded organizations in 22 counties within Ohio during the past 18 years

3. Ohio Commission on Minority Health: Mission • The Ohio Commission on Minority Health is dedicated to eliminating disparities in minority health through innovative strategies and financial opportunities, public health promotion, legislative action, public policy and systems change. Website:http://mih.ohio.gov/index.stm

4. Present Priority Areas for Funding of OCMH Demonstration Projects • Cardiovascular disease (including Hypertension) • Cancer • Diabetes • Infant mortality • Substance abuse • Violence

5. Grants Funded by the OCMH • Demonstration Projects • Innovative and culturally specific projects that are funded for up to $200,000, for a two-year period in one or more of the six diseases/ conditions/ risk factors, • Minority Health Month • Grants for conducting this celebration in April for an award of $2,000. • Lupus • Grants for lupus programs with a maximum of $14,000. • Health Priorities Trust Fund • Resulted from the Tobacco Master Settlement Agreement. Supports Community Health Grants for up to $200,000 and Scientific and Community Partnership Grants for up to $400,000. Focus of this project has been to improve the evaluation of demonstration projects.

6. Major Program Activities of OCMH Funded Projects: 1988-2001 • Providing health education • Focusing on “hands-on” skill building • Providing health screenings • Primarily focusing on health screenings for cancer • Increasing access to health care • Providing public awareness campaigns • Providing referral & follow-up • Improving relations with primary care providers • Building cultural competence in health care providers

7. Distribution of OCMH Grants by Race: 1988-2001

8. Distribution of OCMH Grants by Type of Organization: 1988-2001

9. Ohio Commission on Minority Health: View on Evaluation • Evaluation methods should be appropriate to the intervention model. • Each evaluation should include measures of both process outcomes (implementation) and behavioral/health status outcomes (changes in participants). • Evaluation is seen as a vehicle for program improvement (internal use) and program accountability (external use). • Evaluation findings from OCMH-funded projects should contribute to the knowledge base of what works for minorities.

10. Purpose of this Project • OCMH identified a need to streamline the evaluation process being used by demonstration projects that could be adopted by all funded projects, thereby allowing for cross comparisons between projects. • The purpose of this project was to develop a culturally-competent and practically-feasible evaluation model specific to projects serving minority populations in Ohio and make recommendations for its implementation.

11. Evaluation Focus • Cultural competence: Behavioral (knowledge, attitudes, and skills) and environmental (policy & organizational change) level initiatives that help communication and effective relations across and among different cultures • Culturally-competent evaluation: Emphasis on community participation before, during, and after evaluation

12. Methodology for Streamlining Evaluation of OCMH Projects: 2005-2006 • State-wide search & selection of six evaluators with expertise in working with minorities & program evaluation: May 2005 • Review of existing models: June-July 2005 • Selection of outcome measures & model: August-September 2005 • Evaluation Guidelines including changes to RFP & Reporting developed: Sept. 2005-June 2006 • Projects begin implementation of new guidelines at next funding cycle (FY 2006 – 2007)

13. Recommendations for Specific Changes Related to Evaluation • Pre-proposal period (modifications to the RFP, technical guidance, pre-identified pool of evaluators, writing support) • Proposal review process (technical comments on evaluation plan by evaluation professionals) • Post award support (technical assistance) • Grant period (annual technical assistance meeting) • End of grant period (clear guidelines for evaluation report)

14. Salient Highlights of Evaluation Guidance Document • Presents a practically-feasible model • Focuses on culturally-competent evaluation • Addresses issues of health literacy • Specifies key considerations in selecting an evaluator, planning for an evaluation budget, choosing a design and instruments, and securing permissions, as needed • Provides examples of outcome indicators and sample instruments

15. Guidance Document’s Content Highlights • Rationale for Evaluation: Evaluation helps to determine if programs really make a difference in the health and lives of people in Ohio. • Cultural Issues in Evaluation: Culturally competent evaluation engages the community being evaluated with the evaluation itself. This evaluation approach places an emphasis on community empowerment or community participation is necessary for a culturally competent evaluation.

16. Guidance Document’s Content Highlights Cont’d • Given the increasing size of the ethnically/racially diverse populations and various cultures, intervention programs and evaluation designs must be culturally competent. Nearly all work involves communication or the act of sharing information, yet we are rarely taught how to communicate effectively, in a multicultural setting. Program staff and evaluators must develop the skills necessary for obtaining cultural information from participants, yet bearing in mind that each participant is different and may not share all of the cultural values of her/his cultural group. • Prior to designing a program and evaluation that will help to reduce health disparities, the project director and evaluator must take into consideration the cultural context for the intervention and the participants’ past experiences with the health care system. • Various strategies can be employed when conducting a culturally competent evaluation, including Pre-Evaluation and Post-Evaluation work which are very crucial.

17. Guidance Document’s Content Highlights Cont’d • Health Literacy: Health literacy is the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions (National Library of Medicine, 2000). • Although low health literacy is more prevalent among people who completed fewer years of education, persons of certain ethnic/racial groups, the elderly, and individuals with low cognitive ability, the Institute of Medicine (2004) report noted that people of all ages, races, income and education levels are challenged by low health literacy. • Instruments have been developed for measuring health literacy levels. • Some interventions that have helped to improve knowledge for the low health literacy population include using videotaped educational tools, illustrated materials, and designing pamphlets for easy readability.

18. Guidance Document’s Content Highlights Cont’d • Evaluation Logistics: The evaluation logistic section of the Guidance Document provides information about selecting an evaluator, the evaluation budget/contract, and the institutional review (IRB) board process.

19. Guidance Document’s Content Highlights Cont’d • Evaluation Approaches: This section of the Guidance Document consists of information about evaluation designs, instrumentation, reliability, and validity as well as data collection methods.

20. General Logic Model: Basic Depiction

21. An Example of Outcomes and Outcome Indicators

22. Concluding Thoughts • The evaluation model will be implemented beginning with grantees in the 2006-2007 OCMH funding cycle. • The model offers great potential for streamlining data collection, data analysis, and reporting of evaluation results from the health promotion projects serving minority populations in the state of Ohio. • The model contains many exportable components that can be replicable in Ohio and other states for projects pertaining to ethnic minority populations.