BY: RYAN EHRENSBERGER, Ph.D., and JULIE TROCCHIO, RN, MA
Dr. Ehrensberger is administrative director, community benefit services, Bon Secours Richmond Health System, Richmond, Va., and Ms. Trocchio is senior director, community benefit and continuing care ministries, Catholic Health Association, Washington, D.C.
Evaluating community benefit programs and activities involves asking many questions. Answers to those questions enable health care leaders to make decisions about ways to improve services and plan future programs. While many leaders focus on the program impact, other equally important questions need to be answered to help explain the effectiveness of a community benefit program. Even the most scientifically valid program may not demonstrate the impact desired by community benefit leaders. Knowing what questions to ask and seeking answers to those questions will strengthen the quality of both the program and the evaluation.
There are several important questions to ask at the beginning of the evaluation process before addressing the question of most interest to leaders — are we making an impact? Although the following questions are listed sequentially, in reality, these questions are often asked simultaneously and are more cyclical in nature than linear.
Does The Overall Program Have What It Needs to be Successful?
To succeed, any community benefit program needs full organizational commitment and dedicated resources. This often includes a specific community benefit goal or objective in an organization's strategic plan, and an assessment of the human and financial resources available to fulfill those goals. To this end, the first set of evaluation questions relates to whether sufficient human and financial resources are available, appropriate policies and procedures are in place, and whether effective community partnerships exist.
The amount of resources needed for the community benefit program will vary for each organization, depending on the scope of services. However, every organization involved in community benefit should ask the following questions:
- Have we committed enough resources to accomplish our goals?
- How does this program align with our organization's overall strategic plan?
- Does the community benefit leader(s) have specific job responsibilities for these programs stated in her/his position description?
- What are the financial assistance policies and how are they applied?
- How are community needs identified and prioritized?
- How are other departments within the organization (finance, planning and communications) integrated with community benefit services?
The collaborative nature of the organization in the community is bedrock to an effective community benefit program. Health care organizations should engage in candid discussions addressing the compatibility of its community benefit partners and whether they are adequate partners to achieve the outlined goals. However, collaboration should be reciprocal and it is equally important for the organization to candidly examine itself with regard to its reputation in the community for being open and approachable by others to address community need, and whether it has effective working relationships with the key community groups.
Are We Implementing the Right Programs to Meet Community Needs?
Every health care organization serves communities with multiple needs. Even the most well-intentioned community benefit program may not achieve the desired impact if the provided services are not meeting a community need, do not fit within the organization's overall strategic plan, or use organizational and community assets inefficiently.
Conducting a needs assessment will identify what needs exist in the community the organization serves. Community benefit leaders can then assess the degree to which their existing programs address those needs and opportunities for new services. Several factors influence a hospital's decision to continue and/or establish new community benefit programs. However, the following questions that organizations can ask during the planning and evaluation of their community benefit program will help address the degree to which the program(s) meet the need in the community:
- What evidence do we have that a need exists for this community?
- What is the strength of that evidence?
- Does the program duplicate or detract from an existing community program?
- Does the program primarily serve those in need?
- Is the program easily accessible for those most in need of the service?
- Would other organizations provide this service if we did not?
- Why have we chosen to address these needs? (This assumes that an organization cannot address all the needs identified in the needs assessment.)
Answering these questions can be a challenge and health care organizations must be prepared to accept that some programs they provide are not appropriate. A long-standing program that has continued even though the need for it no longer exists, or a health education program targeting only insured persons excluding low-income persons are examples of community benefit programs that should not continue.
An important question to ask about the quality of community benefit programs is: are the programs evidence-based? Evidence-based programs use approaches that have been successfully implemented elsewhere and have research to back up their effectiveness. Community benefit resources should use evidence-based programs when possible because, in most cases, these programs have already demonstrated desired outcomes and increase the likelihood of similar success in your community.
Are Individual Programs Being Implemented Correctly?
Once a community benefit program(s) is selected, an initial evaluation of the intervention should begin. The evaluation of program implementation is referred in textbooks as "process evaluation." It gathers information about how the program is being carried out and whether it is implemented as planned. It looks at the resources used (Were the teaching materials effective? Were staff members knowledgeable, and did they connect with participants? What was the cost per person?), participation information (Did the program reach the target group? Was attendance as expected? How many sessions did each participant attend?) and methods (Was the program implemented as designed? Were the participants satisfied with the program?).
For evidence-based programs, the evaluation should examine whether the program was faithful to the original program design in critical areas. Although exact replication is usually not possible, evidence-based programs will have critical elements which should be followed. For example, a successful chronic disease management education program might have a set curriculum that should be presented in a certain order, using standardized teaching materials.
One way to assess how programs are being implemented is to pilot the program before full implementation use in the community. Gathering information from this pilot would indicate whether the methods used were workable, the staff and participant perceptions of the program, and problems encountered that could be avoided when implementing it with a larger group.
A pilot evaluation should always be done whether using an evidence-based program or testing the effectiveness of a new program. It is critical to gather this information on an ongoing basis beginning at the start of the program, rather than waiting until the end to review the data. The information obtained in a process evaluation is extremely valuable to program managers so they can make modifications to the program as needed.
Are We Making an Impact?
All community benefit programs start with a basic theory: "if we implement X-program in this way, then Y-outcome will happen." Evaluating the impact of a program examines whether the intended outcomes actually occurred and to what degree those outcomes can be attributed to your program.
Evaluating impact begins with asking stakeholders, such as the program staff, funders and organizational leaders, the following questions: What do they expect to happen? What outcomes do they hope the program achieves? You may find their most pressing interests relate to how many persons were immunized or whether the program resulted in solidified partnerships with community groups. Stakeholders may want to know if there was a cost savings as a result of the program, or if the program worked for the intended population. Community benefit programs do not always have the evaluation resources needed to monitor all the possible outcomes for every intervention. Therefore, it is important to identify which indicators are of most interest to the program's key stakeholders. This will not only reduce the logistical frustration of data collection associated with impact evaluations, but will also provide a strategic direction for implementation of the program.
Most community benefit programs have three types of program outcomes: immediate, intermediate and long-term.
Immediate outcomes are the short-term, observable effects of the program. These could include a change in knowledge, attitude or skills of the participants. For example, a school-based asthma program might examine the immediate outcome of whether the children enrolled in the program know what environmental triggers might prompt an attack.
Intermediate outcomes link the immediate outcomes with the long-term outcomes. They usually involve risk factors or clinical changes in participants' health status, change in utilization rates of health services, or cost savings. For example, if the ultimate goal of a program is to decrease tobacco related illness, an intermediate outcome could be completion of a smoking cessation program and being tobacco-free for a set period of time. If a program has been designed to reduce diabetes among obese teens, an intermediate goal might be weight loss of those enrolled in the program.
Long-term outcomes are the ultimate goals of the programs, such as change in morbidity, mortality or quality of life. These outcomes can take several years to achieve and are often beyond the scope of most community benefit programs. For example, a smoking prevention program might have a long-term outcome of decreases in the rate of tobacco-related illness and death in the community. This type of evaluation would require a timeline of several years to observe these changes at the community level. It also would require professional assessment of whether this specific program was responsible for, or contributed to, the outcome. These types of evaluations are often done by trained epidemiologists.
Therefore, in assessing impact, most community benefit programs concentrate on evaluating immediate and intermediate outcomes.
What Does All This Information Mean?
Evaluating community benefit programs can be a daunting task for a health care organization and its leaders. The true purpose of evaluation is to gather information that helps in decision-making about the merit or worth of a program or service. Judging the merit or worth of a program is influenced heavily by the interests of the key stakeholders and the goals of the program. Once programs are evaluated, findings should be turned into action. Questions to ask include:
- How can we use this information to improve program quality?
- What other resources are needed?
- Should programs be discontinued? Modified? Replicated?
- How can we use findings to better relay our community benefit story?
Evaluation is a complicated practice and although this article cannot provide solutions to all evaluation questions, it can be used as a guide when planning community benefit programs. It is critically important to ask these questions at the beginning of the planning process and not waiting until the program ends. Planning for the evaluation of the community benefit program should be done as a part of the planning for the program itself to ensure that appropriate questions are being asked.
Share Your Views
The Catholic Health Association is working with system community benefit leaders, Saint Louis University School of Public Health, and the Disease Management Association of America to develop a comprehensive guide for program evaluation. Ask for a draft copy to test in your organization and provide us with comments.