Evaluation
The evaluation approach for the Public Health Councils (PHC) is formative and includes some outcome evaluation strategies to assess impact of the program objectives. The evaluation is ongoing, and the data collected thus far is preliminary. The data that has been analyzed shows that the CBOs have met all three programmatic objectives and offers evidence for the potential impact of the PHC program and insight into the major challenges many CBOs are facing. For example, several CBOs have been very successful in developing partnerships with employers to coordinate on-site vaccine education sessions followed by vaccine clinics. Some CBOs are facing major obstacles reaching workers while others are starting to see the tides turn in their ability to gain worker trust. For those CBOs that have gained worker trust, there has been a rise in complaint submission to Public Health and increase in the information about workplace conditions that workers share with certified CBO staff.
Objectives:
1) Establish PHCs at worksites in the focus sectors
2) Increase the number of complaints submitted to Public Health by workers in the focus sectors
3) Increase COVID-19 vaccinations among workers in the focus sectors
Public Health is utilizing a mixed-methods data collection approach to assess reach and impact of this initiative, modify the program as needed, understand challenges and lessons learned through implementation, and inform future initiatives.
Data sources
1) Quantitative – Public Health created two data tracking systems to track outcome data in real time.
a. CBO data tracker: CBOs input their data related to daily outreach numbers, complaints submitted to Environmental Health, and PHCs (e.g., the number of PHCs formed, monthly meetings, and people that attended each meeting).
b. Vaccination education and mobile clinic tracker: Public Health inputs data related to vaccine education events and mobile vaccine clinics (e.g., the number of vaccine education sessions, mobile clinics, people attended, and vaccines administered) they are organizing at job sites in partnership with CBOs.
2) Qualitative – Public Health is collecting qualitative data via monthly CBO reports and through interviews. At the time of this writing, interviews are still ongoing.
a. CBOs’ monthly reporting: CBOs are required to submit monthly narrative reporting to inform Public Health of their progress on program objectives, obstacles, lessons learned, and success stories from the field.
b. Interviews: Public Health is interviewing various program stakeholders to gain insight on their experiences in the program and to inform future program modifications if necessary.
Process and Outcome Measures
1) Process Measures
a. Sector Outreach
i. The number of 1-way worker outreach; this includes handing out fliers, leaving voice mail messages, or sending mass emails or texts
ii. The number of 2-way worker/employer outreach; this includes interactive conversations with workers/employers in person or by video conference, phone, email, or text
iii. The number of different businesses reached
b. Public Health certified worker education
i. The number of CBO staff trained to be certified workers
ii. The number of educational and training events that were provided to certified CBO staff
iii. The number of certified CBO staff attending educational/training meetings
iv. Qualitative narrative report data from LOSH/COSH on technical assistance they provided certified CBO staff
2) Outcome Measures
a. PHCs
i. The number of PHCs formed
ii. The number of Monthly PHC meetings
iii. The number of People attending monthly PHC meetings
b. Vaccination efforts
i. The number of vaccine education sessions held
ii. The number of workers attending vaccine education sessions
iii. The number of mobile vaccine clinics held
iv. The number of workers receiving vaccines at the mobile clinic
c. Complaints filed
i. The number of HOO violations filed to Public Health
ii. The number of anti-retaliation complaints filed to DCBA
Results
The data collected from various trackers was aggregated below to determine progress on process and outcome measures.
1) Sector Outreach
a. Restaurant
i. 1-way worker outreach – 1,398
ii. 2-way worker outreach – 901
iii. 2-way employer outreach – 399
iv. Total outreach – 2811
v. Businesses reached – 309
b. Warehouse
i. 1-way worker outreach – 1,305
ii. 2-way worker outreach – 394
iii. 2-way employer outreach – 44
iv. Total outreach – 1743
v. Businesses reached – 142
c. Garment Manufacturing
i. 1-way worker outreach – 1,660
ii. 2-way worker outreach – 756
iii. 2-way employer outreach – 57
iv. Total outreach – 2473
v. Businesses reached – 69
d. Food Manufacturing
i. 1-way worker outreach – 337
ii. 2-way worker outreach – 126
iii. 2-way employer outreach – 25
iv. Total outreach – 488
v. Businesses reached – 60
e. Grocery
i. 1-way worker outreach – 189
ii. 2-way worker outreach – 41
iii. 2-way employer outreach – 184
iv. Total outreach – 381
v. Businesses reached – 64
2. Public Health certified worker education
a. CBO staff trained - 51
b. Monthly certified worker education/training meetings – 10
c. Certified CBO staff attending educational and training meetings – 198
d. Qualitative narrative report data from LOSH/COSH on technical assistance they provided CBOs – Pending
3.PHCs
a.Total PHCs formed – 25
b.Total PHC meetings – 110
c.Total people attending monthly PHC meetings – 257
4. Vaccination efforts
a.Vaccine education sessions – 30
b.Workers attending vaccine education sessions – 616
c.Vaccine clinics – 18
d.Workers receiving vaccines – 390
5. Complaints filed
a. HOO violations filed to Public Health – 15
b. Anti-retaliation complaints filed to DCBA – 1
Analysis
As a pilot program, all initial data served as a baseline to inform program modifications. After the first six months of implementation, Public Health analyzed all collected data and aggregated it into individualized reports for each CBO. These reports helped inform strengths and weaknesses in the field across sectors and by CBO. The six-month data reports served as a baseline for each CBO. Each CBO is using their baseline data to develop year-long outcome goals and set target metrics for outreach and outcome measures. CBOs were instructed to create specific and measurable goals that could be completed within the span of a year. Additionally, CBOs are creating numerical target goals (e.g., establishing an average monthly target of 2-way conversations with workers).