BY: JULIE TROCCHIO, BSN, MS
The most significant challenge in the community health needs assessment process, according to many community benefit leaders, is evaluating and reporting the impact of their activities. Starting with the current cycle of CHNAs, tax-exempt hospitals must describe the impact of activities they have undertaken to respond to needs identified in previous assessments.
Although meeting regulatory requirements is an important reason to establish a strong community benefit program evaluation process, charitable, tax-exempt hospitals also need to show that they are good stewards of the public and philanthropic funds entrusted to them to improve the health of their communities. A strong evaluation program allows these hospitals to improve and fine-tune their community benefit programs and to show the impact that they are having on targeted community health needs.
TAKING A CUE FROM PUBLIC HEALTH
The Catholic Health Association has developed two resources to assist tax-exempt hospitals: A guide titled
Evaluating Your Community Benefit Impact and a five-part webinar series. Both resources were developed with Vizient, a longtime partner of CHA's in community benefit education, and are modeled on the Centers for Disease Control and Prevention's six-step framework for evaluating a public health program:
- Engage stakeholders
- Describe the program
- Focus the evaluation design
- Gather credible evidence
- Justify conclusions
- Ensure use and share lessons learned
CHA has tailored the CDC evaluation framework to help community benefit leaders gather information for improving their programs and understanding the programs' impact on community health needs.
Engage stakeholders: Identify who has a stake in the program. The first step in program evaluation is engaging stakeholders. Questions to ask include: Who — as in providers and users — has been involved with the program? Who funded the program? Who will use the evaluation findings? The answers reveal the stakeholders in the evaluation process, and they should be involved to some extent in designing, implementing and reporting the evaluation.
Describe the program: Understand the what and why. Prior to the evaluation, there needs to be a clear understanding of what the program is all about. What are its goals and objectives? How will you know if these have been met? What will be needed in terms of personnel, facilities and materials? What activities will be carried out? Once there is a shared understanding of the program, the evaluation can be designed.
Focus the design: Decide what you want to know. Because there are many facets to most community benefit programs, it is necessary to decide what it is about the program you want to evaluate. Stakeholders can be helpful here, and they may have suggestions such as: Did participants find the program helpful? Did the program change attitudes and/or behaviors? Was the program cost-effective? Were anticipated outcomes reached?
The evaluation can answer questions about both the program's implementation and its impact. Implementation questions could include: Were activities carried out as planned? Was the target audience reached? What problems were encountered?
Impact questions could include: Was knowledge gained? How many were screened and referred for treatment? Were there measurable changes such as reduced weight, blood pressure or cholesterol levels?
Gather credible evidence. Collect actionable information. This step involves compiling reliable and valid information that can be used to inform program decisions and actions. For each question, the evaluation team needs to determine what data is needed to answer the question, how and when it will be collected, and who will be responsible.
CHA's Evaluating Your Community Benefit Impact advises: "Those who will use the information to make program decisions should have confidence that the information gathered is believable and trustworthy and that your recommendations are based on credible data."
Data can come from existing record-keeping systems, such as hospital admissions or school attendance records, or data can be collected specifically for the evaluation through a participant survey or interview notes. The evaluation's design will dictate when to collect information — to assess impact, for example, data could be collected at the conclusion of the program (known as post-test design) or before and after the program (known as pre-test/post-test design). Similarly, to measure the impact of counseling on weight management, data may be collected from participants after counseling to show current attitudes and self-management skills, or data may be collected about participant knowledge before and after the counseling to show if and how much knowledge has changed.
To assess implementation of the program, information may be gathered in the planning stages to ensure that the program is feasible to implement, appropriate to the priority audiences and acceptable to program stakeholders. This is known as a formative evaluation. Information also may be collected while the program is being implemented to ensure it is being carried out as planned (for example, that the target audience is being reached or that the planned activities are taking place).
Justify conclusions. Understand what your data says. The data collected should indicate whether goals and objectives were met. Were admissions reduced? Did school attendance increase? Were clinical indicators improved? Behavior changed?
The evaluation also should examine how the program was conducted. Was the program implemented as planned? Were unexpected problems encountered? Were there intervening factors that might have altered the outcomes? Such factors might have influenced whether objectives were met.
Share and use findings. Put results into action. The primary purpose of program evaluation is to make decisions — should the program be continued, replicated, expanded? How could the program be improved?
Recommendations and results should be shared with stakeholders identified earlier in the process, people such as program providers, funders and others who can use the information to make program decisions.
The Evaluating Your Community Benefit Impact guide provides a comprehensive description of the evaluation framework described above, an overview of key concepts for community benefit program evaluation and important planning considerations. One of the most important key concepts is to start planning the evaluation when the program is being planned and to evaluate the program throughout implementation. The guide notes, "Building evaluation into the design of new programs gives you the opportunity to specify the information you want to collect before you've implemented the program and to gather the information at appropriate times." This is vital to gathering credible information about the program.
The five-part webinar series on community benefit program evaluation starts with an overview of the evaluation framework and how it can be applied to a single program. The series then examines how the framework can be used to evaluate a multicomponent program, community initiatives and policy/system/environment changes. The guide, webinar series and other evaluation resources can be found on CHA's website at www.chausa.org/communitybenefit.
As health care organizations grapple with how to do more with less, it can be difficult to bring leadership attention and resources to community benefit program evaluation. But investing in the effort can help hospitals better direct their population health management and community health improvement efforts, which in the long run might be the only sustainable approach to addressing these issues. For faith-based, mission-driven organizations, such evaluation efforts get at the heart of why they were established. Results can be used to reaffirm that the programs serve all with respect, are targeted to the most vulnerable and improve the health and well-being of the community.
JULIE TROCCHIO, BSN, MS, is senior director, community benefit and continuing care, the Catholic Health Association, Washington, D.C.