Author: Bryan P. White
*Note: This is a sample program evaluation developed for a graduate level course in public health program development and was not actually intended to be used as a real program evaluation method, although it could be used as a design guide.
Original Publication: 03/30/2020
Environmental health is a core objective of the Healthy People 2020 (DHHS, 2020) goals, a set of national public health standards employed by the United States (Dept of Health and Human Services). Nationally, this goal is broken down into six core categories: 1. Outdoor air quality, 2. Surface and ground water quality, 3. Toxic substances and hazardous wastes, 4. Homes and communities, 5. Infrastructure and surveillance, 6. global environmental health. In order to understand how these goals are reflected in the environmental programs governing health in Corvallis, Oregon, I have created the following evaluation proposal for the Benton County Environmental Health Department and its constituent programs. The evaluation proposal is broken down into 7 core sections. First, stakeholders, a brief description and listing of the stakeholders involved in regulating environmental health in Corvallis, as well as the community members themselves. Second, a detailed description of the current environmental programs implemented in Benton County, which describes the point at which an evaluation program can be implemented as well. Third, a concise description of the design of the evaluation program, including a logic model and expected activities, outputs, and outcomes. Fourth, a guideline for gathering evidence during the evaluation. Fifth, how the evaluation team will justify its conclusions, as well as steps it will take to maintain contextual and cultural competency. Sixth, how the information gained will be used and disseminated, again taking into concern cultural competencies as well as stakeholder propriety. Seventh, a reflection on the standards for “good” evaluations. We believe these sections and the program enumerated below will provide an adequate and useful evaluation of the Benton County Environmental Health Department.
Corvallis, Oregon is a small city located about 60 miles south of Portland, and 20 miles from Salem (the capital). Corvallis’ main industries are food and microbreweries, making access to clean water and sanitation top priorities for public health. Environmental health policies in Corvallis are dictated at the county level by Benton County Environmental Health Department, which employs a mixture of State, County, and local policies to conform with State and Federal health standards.
Table 1. List of Stakeholders and Level/Sector.
|State||Oregon Health Authority – Oregon Drinking Water Services (DWS, 2020)|
|State||Department of Environmental Quality (DEQ, 2020)|
|State||Oregon Water Resources Department (OWRD, 2020)|
|County||Benton County Environmental Health Department (BCEHD, 2020)|
|City||City of Corvallis – Public Works (CCPW, 2020)|
|City/Private/State||Oregon State University, College of Engineering – Well Water Program (OSU CE, 2020)|
|City||Community Members – Corvallis residents, students, and visitors|
In terms of Healthy People 2020 goals, #2 (Surface and ground water quality), #3 (Toxic substances and hazardous wastes), #4 (Homes and communities), and #5 (Infrastructure and surveillance) are most applicable to the local health needs of the people of Corvallis.
The potential health impacts of neglecting environmental health are very serious and poor environmental conditions contributes to overall community health burden. Health impacts of lack of clean drinking water include a, b, and c. Access to clean drinking water is impacted by environmental hazards posed by human run-off and wastewater, a large amount of which are generated by the food-service industry. Monitoring and controlling the production of environmental wastes through the permitting of sanitation activities (e.g., septic tanks and business-grade drinking water), the licensing of food handlers, and food establishments, and restaurants, and the surveying of local drinking sources (e.g., lakes, rivers, and ground water) are all goals of the Benton County environmental health department.
Ensuring sound environmental health policies in Benton County is a complex program wherein multiple State, County, and City level programs are employed to fulfill certain sub-tasks of the environmental program. Some of the key programs include the following:
A septic tank permitting program, issued by the Oregon State Department of Environmental Quality. Septic tanks are a possible source of contamination into both groundwater (even during proper usage) and a risk for contamination of above-ground water (lakes, streams, and rivers during flooding). Proper care, installation, and maintenance of septic tanks can reduce the risk of contamination into groundwater. Permitting involves a pre-installation inspection followed by the creation of test holes surrounding the septic tank for post-installation inspection, both signed off by a certified State or County employee.
Food handling and restaurant licensing is another entry point for environmental contamination in Corvallis, Oregon. There are over 115 restaurants or food establishments in the city (Visit Corvallis, https://www.visitcorvallis.com/restaurants), so the potential for wastewater and run-off contamination of nearby water sources (e.g., the Willamette River) is extremely high. Although Corvallis does not have a great population (58,641 as of 2018), compared to other nearby metropolitan areas (e.g., Portland, 653,115 as of 2018), in terms of population density it is very similar (4,110 people per sq. mile in Corvallis compared to Portland’s 4,895 per sq. mile). With such a high population density, coupled with many food/restaurant establishments, proper licensing of food handlers and restaurant facilities should be a critical goal of the local health departments.
To maintain proper food industry licensing, Benton County has two public health endpoints: The Food Handlers Card and Food Establishment Licenses. Food Handlers Cards include training for proper food handling as well as maintaining basic food sanitation policies. The authority in Oregon for issuing Food Handlers Cards is the State level by Oregon Health Authority, and the training programs are implemented at the County level. In Benton County, all Food Handlers card training can be completed online. Obtaining restaurant licensing is much more complicated depending on the type of food establishment or restaurant, and what type of facilities the establishment will be offering. Similarly, though, the State level authority is implemented by the County, and licensing is issued directly by the county. Food Handlers and Restaurant Owners in Corvallis can follow Benton County guidelines to obtain proper licensing.
Maintaining clean water supply and access is a priority health concern of Oregon state, and programs ensuring clean and safe drinking water are implemented across all levels of the government. In the case of Corvallis, an additional level of program implementation occurs at the State level through Oregon State University, which acts as both a local and statewide authority.
Some of the key clean water programs in Benton County include Protect Oregon Groundwater (Oregon State University, OSU), Oregon Drinking Water Service (Oregon Health Authority, OHA), and the Department of Environmental Quality (DEQ) Drinking Water Protection Program. The goal of the Protect Oregon Groundwater from OSU is to protect drinking water from wells and or aquifers that may be exposed to septic systems. About 23% of Oregonians rely on domestic wells, so preserving and protecting these water sources is a critical aspect of maintain clean drinking water. The Oregon Drinking Water Service is a state level (OHA) service that engages in regulatory activities by enforcing drinking water quality standards in the state.
All of these programs combined contribute to the Benton County Environmental Health program, and serve as entry points for providing clean water access and services.
Figure 1. Logic Model for evaluating the Benton County Environmental Health Program.
During the evaluation program development phase, two key components should first be identified: 1.) The Social Health Problem, which is a representation of the social health problem that the evaluated program is addressing and 2.) The Public Health Goal, which is the ultimate goal in terms of public health outcomes that the program is seeking to achieve.
Social Health Problem: Protecting drinking water from contamination, including either runoff from sewage or pollution through kitchen sinks, is a critical component to ensuring positive health outcomes in local communities. Similarly, proper food handling, along with food sanitation procedures, contributes directly towards improving both water security and food security, which ultimately improves health outcomes.
Public Health Goal: To improve health outcomes through safer water access, cleaner sanitation protocol, and improved food hygiene in Benton County.
After defining the Social Health Problem and Public Health Goal of the evaluation program, the Activities, Outputs, and Outcomes should be described, which includes their timeframe and relationship to stakeholders. Since the Benton County Environmental Health program is really a collection of multiple City, County, and State stakeholders are engaged to various degrees and activities, the stakeholder engagement plan has been simplified into 3 main categories, as depicted in Figure 1 (Activities). These three main categories are Environmental Site Inspections, Food License Training, and Water Quality Monitoring.
The goal of this evaluation program is to understand the scope and nature of Environmental Health in Benton County, and to offer recommendations to stakeholders on how to create a more streamlined, coordinated program between all of the relevant stakeholders.
In order to understand the key endpoints for gathering evidence to evaluate the Benton County Environmental Health program, the Activities, Outputs, and Outcomes are summarized here in Table 2.
Table 2. Key Activities, Outputs, and Outcomes to be evaluated in this proposal. Activity Output Outcome Environmental Site Inspections Env. Site Reports Identify Contaminated Sites Food License Training Training Certificates Location Certification Improve Food & Sanitation Handling Water Quality Monitoring Water Quality Reports Identify Poor Water Quality
Most indicators that can be used for this evaluation are related to the tracking and logging of reports filed by program employees and staff. These include reports of program participation, permits granted, environmental evaluations conducted, incident reports, and inspection reports. In order to gather this evidence, program evaluation staff will have to be hired to physically engage stakeholders in reporting and data collecting activities. In terms of community-level stakeholders, the ultimate health impacts of the program also need to be evaluated. Community-level evaluation can be carried out by surveys, interviews, and similar inspection of environmental reports (e.g., water quality complaints logged by Corvallis residents) and would also require the hiring of program evaluation staff. The recommended time frame for this evaluation is 2 years, broken into four major segments.
The first segment of data collection should involve establishing a baseline, which will involve meeting with all of the government-sector stakeholders and identifying which materials are currently available and which will continue to be available for data collection during a 2 year period. This will also involve establishing baseline community readings through surveys and community organization events (e.g., townhall meetings) to determine which areas of program evaluation the community members feel are most important to focus on.
The second and third segments of the evaluation should be considered “ongoing data collection” during the evaluation process. These time points, 6 months and 12 months into the program, should be used as major stepping stones for the evaluation team to consider the progress of data collection and make any necessary changes to methods as needed to achieve the goals decided on by the evaluation team with the stakeholder feedback. During these two major milestones, preliminary data analysis should also be conducted. That includes developing and validating any statistical methods that will be used to present the final data.
The final segment of the data collection period should be to complete data analysis and ensure that no additional data needs to be collected in terms of follow-up questions. Most of the potential follow-up questions should already be determined before hand during the 2nd and 3rd segment milestones.
Since environmental health is a mixture of formal science, human health, and socioeconomic aspects, care should be taken during the final data analysis to ensure that justification of the results bests serves the stakeholders at their respective levels (community on up through governmental). In environmental health, there is always room for improvement – no society or culture is perfect – so the evaluation team should be careful not to hold too high of a burden on any one stakeholder group. Oregon’s environmental health departments have already maintained a high standard, but as scientific techniques and methodologies improve, our ability to assess the health and quality of the human interface with their environment.
Some of the data analytics methods that could be employed to understand the effectiveness of the programs under the Benton County Environmental Health department include analysis of variance (ANOVA) on factor variables, such as the average value of water quality metrics in one region (e.g., metropolitan) compared to several other regions (e.g., rural or agriculture). Understanding the environmental landscape in the area surrounding Corvallis, OR would be a critical goal of the evaluation program. In the case of rank-ordered data obtained through surveys, the non-parametric version of an ANOVA (Friedman test) could be conducted. It would be critical to understand which responses to surveys are statistically significant, and under which categories (e.g., race, ethnicity, income level), as statistical significance between groups could be indicators of socioeconomic disparity. Another method that might also be employed could be regression analysis of more frequently collected data (e.g., number of food handlers training events per month). This would be useful to determine if training events are increasing, decreasing, or haven’t changed in a statistically significant manner.
The raw data analysis produced during the justification and conclusions phase should be interpreted and adapted for each significant stakeholder group. It would not make sense to report the same data to the Department of Environmental Quality (ie. government-sector experts) as it would be to present to the local community (ie. non-experts). Preliminary reports should be presented to each set of stakeholders to gauge the utility of the reports early in the data analysis (this should also be done during the intermediary phases of data collection as well), that way stakeholder feedback can be taken into account before final reports are issued. Data reporting should also be careful to maintain confidentiality and cultural competency values of an ethical evaluation – historical or current socioeconomic disparities in the area should be presented in a way as to acknowledge these disparities occur, and offer upfront ways to mitigate any disparities that are discovered through empirical analysis.
Once each final report has been customized to its stakeholder group, the reports should be released and the dissemination of that information within each stakeholder group should also be facilitated by the evaluation team where possible or needed. For the most part, County and State organizations will have their own information dissemination channels, but local City and Community programs will most likely need the assistance of evaluation team staff to disseminate information. At the community level, the information should be the most streamlined and useful for the average non-expert in the area, and likely would consist of flyers, website updates, and community-based informational programs (e.g., town hall meetings) to share critical information with the community. In the case of drinking water, sanitation, food handling, and restaurant licensing, the community of Corvallis itself could benefit greatly from this information, meaning that proper communication of the evaluation result should be a critical goal of the evaluation team.
The four key standards for a “good” evaluation include Utility, Feasibility, Propriety, and Accuracy (Muriel, 2016). In this evaluation proposal, utility is probably the highest ranked standard employed here. Since environmental health has such a close linking to environmental science, transmitting that scientific information from the evaluation team to its stakeholders will secure the utility of this evaluation. In terms of stakeholder identification, since the environmental rules and regulations in the city of Corvallis (the most proximal stakeholder to this evaluation) are first governed by the Benton County Environmental Health Department, and then secondarily by multiple other County and State entities, identifying and including all of the relevant stakeholders as outlined in this proposal further contributes towards its utility. All these stakeholders have their part in some way and unifying and streamlining environmental regulations in the area could greatly expand the overall usefulness of environmental monitoring in the area. The timeliness of the evaluation is lengthy, but necessary to provide an adequate overview of the social health problem. In order to further ensure the greatest impact of the evaluation, the evaluation team itself should be highly credible and consist of experts familiar with the fields of water quality, food health, and environmental sanitation.
In terms of feasibility, the program goals outlined here are very achievable. Most of the evaluation design is centered on collecting data that is already being generated, the evaluation team is simply gathering and harboring a multitude of data in a singular warehouse for data analytics and reporting. One of the considerations for dealing with multiple levels of government authorities is the potential political aspects of the evaluation (e.g., budgeting changes, policy changes, etc.) that might result or influence policy changes within the state of Oregon. The evaluation team should take this into consideration and present reports to the concerned stakeholders in a manner that can be used to inform policy decisions objectively.
In terms of propriety, the ultimate stakeholder of any health program is the people that are subject to the health intervention. All of the government-sector agencies working in their respective environmental health departments are ultimately civil servants working for the people (their constituents – and ultimately, themselves), and the results of the program evaluation should be presented in way that both the needs of government agencies and the average community member are properly served in the evaluation. A list of line-item budget changes is not very useful to a community member dealing with a contaminated water well, and a list of proper care and management techniques for a septic tank is not very useful to a State-level stream water monitoring program. In other words, the level and scope of the evaluation, and to which stakeholders information is most relevant, should be carefully considered during the final reporting phase. It should also be noted that any type of information that is sensitive to a particular stakeholder, whether that is potentially private personal health information or proprietary information to a private organization, the evaluation team should be respectful of those privacy concerns at all times.
The final standard that the evaluation team should take into consideration while designing the proposal is accuracy. Since the repercussions for poor data collection or analytics for an environmental health study are so great, each stage of the program should be designed so that data management redundancies are in effect, in both information storage and analysis, to prevent mistakes from being reported to stakeholders. In other words, the information generated by the evaluation team should be both reliable (consistent) and valid (accurate). Finally, the results of the analysis should be analyzed in both the context of the stakeholders (ie. government vs. community) and in a greater context: How does the Benton County Environmental Health department and its programs compare to other counties in Oregon, and nationally? Where is there room for improvement, and where does the County do something very well? These contexts and others should be considered by the evaluation team during the final reporting phase.
We believe that all of these standards combined will yield a high quality, useful, relevant, and culturally competent program evaluation for the Benton County Environmental Health department and its constituents.
Benton County Environmental Health Department (BCEHD). Benton County Environmental Health. (2020, March 30). https://www.co.benton.or.us/health/page/environmental-health
City of Corvallis – Public Works (CCPW). (2020, March 30). Water Quality | Corvallis Oregon. https://www.corvallisoregon.gov/publicworks/page/water-quality
Department of Environmental Quality (DEQ). (2020, March 30). About Drinking Water Protection. https://www.oregon.gov/deq/wq/programs/Pages/dwp.aspx
Department of Health and Human Services. (2020, March 30). Environmental Health | Healthy People 2020. https://www.healthypeople.gov/2020/topics-objectives/topic/environmental-health
Harris, Muriel J. (2016). Evaluating Public and Community Health Programs. Wiley. Kindle Edition. Oregon Drinking Water Services (DWS). (2020, March 30).
Oregon Health Authority : Oregon Drinking Water Services : Drinking Water : State of Oregon. https://www.oregon.gov/oha/PH/HEALTHYENVIRONMENTS/DRINKINGWATER/Pages/index.aspx Oregon State University, College of Engineering (OSU CE). (2020, March 30).
Home | Well Water Program | Oregon State University. http://wellwater.oregonstate.edu/
Oregon Water Resources Department (OWRD). (2020, March 30). Water Resources Department : OWRD : State of Oregon. https://www.oregon.gov/OWRD/pages/index.aspx