Increasingly, policymakers are focusing on equity as a core principle. But if it’s a principle without metrics, how can we know if objectives are accomplished?
As the saying goes, you can’t manage — or improve — what you don’t measure.
Data come in all shapes and sizes. In the energy sector, research methods strongly favor quantitative data to evaluate public policy and program impacts. But when it comes to policies and programs being equitable, evaluation requires qualitative data on community engagement efforts, especially when engaging disadvantaged and low-income communities and communities of color.
Genuine community engagement is, after all, a key aspect of any effort to achieve the three tenets of energy justice: distributive, recognitional and procedural justice. And this engagement needs to happen in ALL phases of policy and program decision-making: ideation, program design, implementation and evaluation.
The Importance of Qualitative Data
From an equity perspective, numbers are only part of the picture. The narrative, the story behind the numbers, is just as important.
Qualitative research methods systematically elicit this narrative from people who represent a variety of perspectives. These perspectives can be compared, contrasted, categorized — and yes, counted — to understand how often and how important and for whom something matters.
Qualitative research helps decision-makers understand how and why desired program and policy outcomes occur – insights they can use to inform future actions.
Measuring How, Not Just How Many
Program administrators and researchers need to understand and measure the how of what they are doing, not just the what.
For example, say a program administrator reports they conducted X number of community meetings attended by Y number of people. These numbers do not indicate whether the meetings fulfilled community-identified needs or were meaningful and successful in informing community members.
Here are some questions to consider:
- Were the meetings conducted in the prominent language or languages of a community?
- Were the meetings conducted in a location accessible to community members and at a time that would facilitate their participation?
- Was the information discussed in a culturally appropriate manner that resonated with community members, including youth and elders?
- Were community members asked what their priorities are and how they want to accomplish them? Were they asked in a culturally sensitive manner? And did the researchers listen?
These questions speak to formative or process evaluation. They help determine if the quality of the output (e.g., people’s level of engagement in a meeting) and the delivery of the meetings were solid enough to result in the desired outcome (e.g., long-lasting change in people’s behavior).
Further questions to consider include:
- Did community members understand what was being presented?
- Did they feel they received the necessary information to spread the word about the proposed program to others in their community?
- Or was the “community meeting” a box to check without truly involving the community in decision-making?
These questions speak to impact, which is the desired outcome.
One indispensable way to get answers to these questions is to ask community leaders and community-based organizations (CBOs) for their feedback throughout the process. Researchers can also ground truth the extent of the impact by observing meetings and other community engagement activities and surveying and interviewing community members. To close the feedback loop, discuss the results from such studies and resulting decisions with the community.
Track and Measure Community Engagement
An example of this approach in action comes from CSE’s participation in the California Energy Efficiency Coordinating Committee (CAEECC) Equity Metrics Working Group. CAEECC brings together program administrators who run energy efficiency programs under the jurisdiction of the California Public Utilities Commission.
CSE highlighted the need for program administrators to include community engagement as an integral part of the design, tracking and evaluation of equity programs proposed in their energy efficiency portfolio applications and business plans for 2024-2031.
CSE also led in developing the rationale for how program administrators should, as a starting point, track and report the amounts and types of community engagement activities geared toward disadvantaged, hard-to-reach and underserved communities: 1) during program design and to identify community needs and solutions, 2) during program implementation and 3) during program assessment.
We further recommended community engagement and outreach activities adapted from the California Air Resources Board’s Community Inclusion Guidance for its Sustainable Transportation Equity Project (STEP), which include activities more qualitative in nature. Those activities include:
- Partner with CBOs and compensate them for their work
- Consult with advisory board/council/committee members (and include representatives from CBOs and disadvantaged and low-income communities of color)
- Conduct community-based participatory research and participant satisfaction surveys before and after interacting with a program
- Hold community and stakeholder meetings, webinars and calls
- Canvass door-to-door
- Host focus groups and community listening sessions
- Develop educational and promotional materials (e.g., flyers, mailers, social media posts)
Think Holistically
Treating community engagement as something that can be measured, and not solely as a principle, demonstrates that program administrators are intentional about operationalizing equity.
Over 30 years of evaluation research and practice (including Fetterman; Conner; O’Sullivan; Hood, Hopson, & Frierson; and Bledsoe & Donaldson) support the effectiveness of engaging people in the community in the design and evaluation of programs intended to support them.
Qualitative research methods should be grounded in a holistic approach where researchers keep an open mind and seek, with empathy, to understand community needs.
Few people wake up thinking about helping a policymaker or program manager achieve their goals. They’re thinking about their own lives. Disadvantaged and low-income community members may be managing additional quality-of-life and financial issues, such as how to pay for rent, childcare, medical costs, groceries and energy bills.
Program administrators, policymakers and researchers need to shift from their perspective about driving top-down program goals and objectives and instead embed the customer’s perspective and experience.
The result will be easier-to-navigate programs that meet community members’ needs while also addressing broader goals of creating a clean, equitable and sustainable environment for everyone.
