This is part two in a four-part series on program evaluation, dedicated to organizations and businesses that provide programs and services for women, girls, and communities of color (and for people with an interest in evaluation practice). Throughout this month, I will be discussing certain aspects of evaluation practice –from how I became interested in evaluation, myths about evaluation, knowing what type of evaluation to perform, and bringing your community together to support evaluation – with the intent on highlighting the importance of evaluation not just from a funding perspective, but from an accountability and empowerment perspective.
You know that feeling you get when you’re sitting across from your supervisor during your annual job performance review? You think you’re doing a great job, you’re engaging with your co-workers, your projects are completed on time, and you manage your time well. Your supervisor agrees with you and talks glowingly about your performance, but then proceeds to give recommendations on “ways to improve”.
And now you’re uncomfortable. We all believe that we can handle constructive criticism, but who wants to hear how they’re not doing well? And we already know what improvements need to be made! They’re supposed to make us a better worker. Your supervisor gives you this list of things you need to improve on, and tells you that she would like to check in with you to see how you’re doing. You walk out of her office, feeling frustrated. You see what isn’t going well, and are too self conscious to ask how to improve. With “ways to improve” come concerns that if you don’t measure up, you’re reprimanded, you don’t get your raise, you’re demoted, or you’re let go.
Or…you welcome the challenge. You still feel a little uncomfortable, because it’s human nature to want others to see us at our best. But you already knew which areas you needed to work on but weren’t sure how to go about it, and you’re glad that your supervisor is providing you with concrete ways to do so. You ask your supervisor to provide you with more resources, trainings, literature, and other tools that can help you out as well. She even offers to provide you with additional support by checking in on a monthly basis to see where you are in your improvements. You begin to feel more confident, and your quality of work improves.
This is an example in how shifting your mindset can bring about a better outcome. And our mindset is what Part Two is about.
Last week in Part One, I shared what I believe are common concerns that go through the minds of nonprofit, agency, and business staff when it comes to evaluating a program or service.
… You’re tasked to carry out an evaluation and you don’t know where to begin. You lack the staff capacity needed to carry out an evaluation, or you want to build the capacity and are leery of hiring an external evaluator or don’t have the money in your budget to hire an internal evaluation staff member. When the evaluation is finally completed, you’re disappointed because the results you receive aren’t what you were expecting, and now you have to report it to your stakeholders and your funders. You’re trying to meet the expectations of the people you’re serving and also the expectations of your stakeholders and funders, and you feel that you’re at the mercy of an entity that can end your organization’s work, especially if a good portion of your funding comes from a primary source.
It’s a lot to think about, which can make it very easy to approach program evaluation with a “Why do we need to do this again?” mindset.
And just like how you feel as you sit across from your supervisor, how we look at program evaluation determines how successful we’re going to be at monitoring and evaluating our own programs and services, or being successful at working with an external evaluator.
The reason why many nonprofits, agencies and businesses don’t welcome evaluation with open arms is because of the confusion and misconceptions surrounding it. While we understand that program evaluation can strengthen the quality of our programs and services and can lead to better outcomes for the populations we serve, all this other stuff just creeps up and it become unwelcoming.
This is even more apparent when all or some of your services are tailored to meet the needs of women, girls, and communities of color. You feel that you are under even more scrutiny to perform well, nail the performance outcomes you indicated in your grant proposal, and be a success.
In order to change our mindset about program evaluation, we need to get to the bottom of why program evaluation may intimidate us, or at the very least, makes us remain ambivalent. In no particular order, here are 7 reasons why many nonprofits, agencies, and businesses believe that program evaluation is intimidating (and my advice on figuring it all out):
Reason #1: “They’re out to get us”: The biggest myth about evaluation is that its goal is to seek out what’s not going right and find someone to blame for it. Evaluation is never about what you’re doing wrong, but all about knowing what is working and how to improve the scope of the work. Funders, stakeholders, community members, and your participants want to see your programs and services succeed. They wouldn’t invest their time, and resources if they only want a one-time program.
Reason #2: “It’s a waste of our time and resources”: Why put together an evaluation report when most reports are filled with unfamiliar language, are too long, no one reads them, and you don’t even try out any of the recommendations? No one shows up for your focus groups, and let’s not forget that how only half of your surveys are completed because your participants forgot to flip the page over to finish the other side. With all of these frustrations, it’s no wonder why program evaluation isn’t a top priority. While everything may not go smoothly, you do have control over certain outcomes. Instead of creating a cookie-cutter evaluation report, focus on who you’re presenting the findings to. Is it the Board of Directors, members of the community, staff, or to participants? Ask them how they want the information presented. If surveys weren’t fully completed the first time, what are some ways to make sure that you’re giving clearer instruction on completing the entire survey? Do you need to develop a shorter survey?
Reason #3: “For what? We already know we’re great because of XYZ”: A high number of participants, increasing interest, or renewed funding for your program or service is great, but they can sometimes be misleading. Are a high number of participants an indicator that you’re meeting your program objectives, or are participants attending because of some other factor? This is isn’t necessarily a bad thing, but an informed evaluation will let you know why they continue to come back. You’re getting a lot of community support, social media buzz and press? Wouldn’t it be great to share not only the important work you’re doing, but that it’s actually what your participants want? You need your funding to remain renewed (and to get additional money from other funders)? Funders aren’t given out money because you ask. They need to know that what you’re doing is effective and can be tweaked over time.
Reason #4: “It’s too complicated”: Qualitative data or quantitative data? Evaluate before or after the program ends or the service is completed? Ten survey questions or 20? Believe it or not, evaluation doesn’t have to be complicated. Organizations, agencies, and businesses conduct evaluation in a number of ways, depending on their capacity, the scope of their program or service, and the length of time they have to complete it. The process can take one month to several. You also don’t have to be an expert. What’s required is a willingness to know how well your program is doing and how it can become even better.
Reason #5: “We’re not equipped to do this”: Lack of staff getting you down? Don’t know how to analyze the data? Feel like you didn’t budget enough to put towards evaluation? When there’s a will, there’s a way! College and graduate students interested in data analysis and research are always on the lookout for ways to beef up their resumes. Bring them on board! It’s even better if you can compensate them for their time and effort, or at the very least, speak to their school about listing their services to you as work-study. If you want someone to teach you and your staff how to evaluate your program or service, you can always hire out. It’s also important to remember that certain evaluation activities are really just administration tasks that you’re probably already doing.
Reason #6: “We can do this! It’s only a one-time thing”: “One-time thing” thinking lends itself to the mindset of just being out to prove that the program or service is either good or bad. If you want to know how you’re really doing, you need continuous feedback for the duration of your program or service. You need to see how your program or service is progressing, how it can be sustainable, how easily it can adjusted to meet the needs or your participants, and how well you’re keeping on track.
Reason #7: “We have to get it right on the first try”: Not at all. As you become more comfortable with evaluation, you’ll start to see that you’ll always have to tailor it to meet the needs of the program or service. As mentioned in Reason #4, it can depend on your capacity, how expansive the program or service is, the amount of time given to complete the evaluation, and sometimes what your funders and stakeholders expect. Just remember to be flexible.
Get Ready for Part Three
Hopefully you’re starting to see what an investment that evaluation can be to your program or service and to the people you’re serving, that it doesn’t have to be expensive or complicated, and that no one is out to see you not do a good job. Just continue to make those necessary tweaks in your mindset about evaluation, and you’ll begin to see what great benefits evaluation has to offer to you and to your participants.
In Part Three, we’ll look at the most important aspect of evaluation practice. If what I’ve shared so far piques your interest, check out my consulting services page and contact me to see how we can work together to take your program or service to the next level.
