8 Aug, 2018

Try This: Curb Your Evaluation Anxiety

By Nicole Clark, LMSW|2021-08-19T20:15:34-04:00August 8th, 2018|Categories: Research & Evaluation|Tags: Capacity Building, program evaluation, Try This|0 Comments

With my program and evaluation clients, I offer the Evaluation Capacity Measure. It’s an assessment created by Ann Price of Community Evaluation Solutions , and I’ve revised it a bit to include some open-ended questions. The assessment assesses an organization’s current capacity to evaluate its programs, strategies and services and assesses its current support around evaluative thinking. Everyone from organizational leadership to board of director members are encouraged to take the assessment.

This assessment helps me to see what the current needs are, and it also gives clients a sense of what we can work on together in the evaluation capacity building process. I offer it to clients first as a baseline, then towards the end of our partnership together. Whether it’s a short-term or long-term project, clients like knowing their staff’s general views on evaluation, and how the process can be aligned with their respective roles and organizational mission and strategy. (At times, they may notice that what they expected to score lower on is higher than expected, and vice versa).

I recently administered the assessment for a new client, whose staff works remotely and is comprised of several departments that work individually and collectively on several programs and strategies. Leading up to sending the assessment to her colleagues, the staff member leading the project with me shared her nervousness about what the results may reveal about her staff, but is excited to see how the results can influence how they can proceed with their evaluation priorities.

The goal of program evaluation is to 1) see if your program is performing in the way it’s intended to and 2) facilitate organizational learning and improvement. In general, people tend to experience anxiety when they’re being evaluated. It’s like when you waited in a line with your classmates to see which kickball team captain would choose you for their team. You want the best players on your team, and the captains are sizing you up. Who is known to kick the farthest? Who has the capability to catch the ball mid-air? Who can run the fastest?

And when you’re last to be picked, it makes you feel a way. “Why wasn’t I one of the earlier picks?”, you asked. More often the not, the team captains probably chose their team based on who they’re friends with, but there’s a lot of emotional reactions that may come up.

You get into the real world of adulting, and the feeling is still there. This time, it surfaces when you’re meeting with your supervisor for you mid-year or annual review. You think you’re performing well in most areas, until you discover (through someone else’s perspective) that you’re not.

In “Strategies for Managing Evaluation Anxiety: Toward a Psychology of Program Evaluation” (American Journal of Evaluation, Vol. 23 Issue 3, 2002), Stewart Donaldson, Laura E. Googler, Michael Striven dubbed the term “excessive evaluation anxiety” (or XEA). Symptoms of XEA include:

Lack of access to important information and data
Compliance and cooperation problems
False reporting
Effects on bias and validity
Reduced utilization of evaluation findings.

This can lead to stakeholders behaving in ways that can destroy the credibility of evaluation findings (and of evaluators). When the findings of a program’s evaluation yield that it’s not performing in the way it was intended, the blame game can happen. Instead of falling victim to XEA, how can you and your staff become more open to viewing the process as a learning experience?

To give you a taste of the Evaluation Capacity Measure I give to my clients (and to help jumpstart the conversation with your staff around evaluation anxiety), here are some questions you can walk your staff through:

Here’s what you need:

Flip chart paper, chalkboards, or walls
Note cards
Pens or pencils

Designate five open areas around the room. Each area will have one question at the top of the flip chart paper, chalkboard, or wall.

Give staff enough time to answer the questions thoughtfully . When time is up, have staff place their note cards under the designated question.

Next, divide up your staff so that each question has several eyes on it. Have staff place the note cards into themes, grouping similar responses together.

Last, have staff do a “walk-around”, where they can view the themes for each question. To close out the exercise, discuss the themes with staff and create an action plan to address the themes.

The steps:

Have your staff answer the following questions alone on note cards:

(more…)

22 Feb, 2018

How to Weave Storytelling with Statistics

By Nicole Clark, LMSW|2023-09-12T16:44:09-04:00February 22nd, 2018|Categories: Research & Evaluation|Tags: program evaluation, Research, storytelling|0 Comments

Dr. Jennifer Aaker, marketing expert and professor at Stanford Graduate School of Business, once shared a story of a marketing researcher who asked students to each give a persuasive one-minute pitch to their classmates. While most students included statistics in their pitches (an average of 2.5 stats), only one student included a story in their pitch. Afterwards, the researcher asked the students to write down every idea they remembered from each pitch.

While five percent of the students remembered a statistic, 63 percent remembered the story.

The reason? Aaker offers three:

Stories are powerful tools that force people to slow down and listen.
Stories influence how people see you.
Stories move people from complacency to action.

Statistics may bring attention to a cause, but stories elevate their impact. In short, stories can give numbers more credibility.

Some people are hard numbers folks, and I get it. Especially if you’re someone that’s responsible for illustrating impact, such as a grant writer, funder, nonprofit manager. Social workers and others in the helping professional rely on hard numbers because it can lead to increases in funder for their programs and services.

It’s easier to pull numbers. Just create a survey and send it out.

But if the marketing researcher’s discovery is any indication, stories draw people in and have greater impact.

Take the “identifiable victim effect”, for instance. This refers to the human tendency to offer greater sympathy and aid when a specific person is observed under hardship, compared to a vaguely defined group with the same need. The identifiable victim effects puts a “face” to a problem, causing greater impact.

For example, last night, CNN aired a town hall featuring survivors of the mass shooting at Stoneman Douglas High School in Parkland, Florida, in which 17 people (including 14 students) were killed. Statistics from the Centers for Disease Control show that, on average, 96 people die by gun violence every day in the United States. An average of 96 people doesn’t sound like a lot, but hearing the voices of the survivors and family members during the town hall (and seeing the faces of the victims from the shooting) literally puts a face on the problem of gun violence in America more now than it ever has.

As a follow-up to my “Who Are The People Behind the Numbers?” blog post from 2014, I wrote a Try This exercise on using storytelling as a tool for Reproductive Justice. Sharing personal stories resonate with us and helps to build powerful connections with others while also helping to build compassion, especially when we’ve never had a particular experience.

Sometimes seeing a high percentage raises awareness; but numbers alone may not fully capture the entire picture. And there are even people who say that storytelling should replace numbers. Don’t throw out your spreadsheets and statistical software just yet, but don’t stress yourself out with figuring out how to tell the most compelling story without numbers to back it up. Instead, use storytelling to make your numbers stand out (and vice versa). Here are 5 ways to weave storytelling with statistics.

(more…)

14 Feb, 2018

Try This: Appreciative Inquiry

By Nicole Clark, LMSW|2021-08-19T20:11:01-04:00February 14th, 2018|Categories: Research & Evaluation|Tags: program evaluation, Try This|0 Comments

Have you ever noticed that when you go looking for problems, more problems tend to appear?

It’s like peeling back the layers of an onion and chopping it. It’s never-ending and your eyes water in the process.

The same goes for conducting community needs assessments. When designed to identify the pressing needs of a community, they often focus on deficits, which doesn’t do much for community morale. Continuous focus on the problem increases the likelihood of seeing the problem everywhere. This isn’t to say that communities should turn a blind eye to what’s happening, but there’s something to be said about raising awareness of this practice, as it can immobilize communities to create change.

A while back, I wrote a blog post on asset mapping as a tool for community organizing and engagement. One reason why asset mapping and similar strengths-based tools are growing in popularity is due to an increasingly mindset shift away from solely deficits-based to identifying community strengths. Whereas deficits-based practices are problem-focused, needs driven, and questions what’s missing, strengths-based practices are opportunity-focused, strengths driven, and identifies what is currently available that can be built upon.

Today, let’s look at another strengths- based practice, appreciative inquiry.

What’s appreciative inquiry?

Appreciative inquiry (AI) is strengths-based approach, developed by Dr. David Cooperrider in the 1980s. First used in organizational development and change, AI has helped institutions worldwide integrate the power of the strength-based approaches to multi-stakeholder innovation and collaborative design. It quickly gained ground in program evaluation following the 2006 release of Reframing Evaluation Through Appreciative Inquiry by Hallie Preskill and Tessie Catsambas.

AI focuses on identifying what is working well, analyzing why it is working well and then doing more of it. In other words, AI teaches us that an organization will grow in whichever direction that people in the organization focus their attention.

If this can be done in organizations, why not apply it to community change?

Here’s what you need:

(more…)

15 Nov, 2017

Are You Focusing on The Process or The Impact?

By Nicole Clark, LMSW|2021-08-19T20:00:56-04:00November 15th, 2017|Categories: Strategic Planning & Sustainability|Tags: program evaluation, Strategic Planning|0 Comments

Over the past year, I’ve become intentional in transitioning away from focusing solely on designing and implementing evaluations to working with my clients on building their capacity to do it all themselves.

And sometimes, I like to offer aspect of my services for free as a way of establishing a connection with a potential client. From a skills perspective, this helps me keep everything sharp. From a networking perspective, this puts me on the radar for future paid consulting work and referrals.

Recently, I offered to revised an evaluation tool for an organization providing small grants to community groups seeking to reduce abortion stigma. They want to develop an easy-to-understand evaluation tool that measures stigma pre and post grantee project interventions and measures overall project success.They are finding their current tool–a survey requiring grantees to provide open-ended answers–to be challenging for most of their grantees to understand, and want something that makes the process of giving feedback easier to obtain and to analyze.

In my work with past and current evaluation clients, some themes I’ve observed related to evaluation are A) a level of anxiety around evaluation as a whole given that evaluation tends to get a bad reputation, B) an interest in developing engaging ways to gather feedback that builds confidence in evaluation, and allows them to do the necessary follow-up with staff and the stakeholders, C) a question in how to implement feedback, or D) a mixture of A, B and C.

I see this more so with clients who are more grassroots or have a community organizing background as they ultimately go by direct community engagement for feedback, rather than on evaluations. Also, there is a sense of feeling protective of their programs, services, and campaigns, and no one wants to see that what they’re doing isn’t resonating with the communities they serve. So, finding a way to gather meaningful feedback and be objective of feedback that may be interpreted as negative is a balancing act.

For some background information, grantees receive small grants to develop a project/intervention that speaks to a level of abortion stigma: individual, community, institutional, media, and policy. Grantees have free reign to create what they want, with support from the organization, I made some preliminary edits to the organization’s evaluation tool, and provided some additional feedback:

First, before starting any process, understand what you want to evaluate. In this organization’s case, the current tool focuses more on the process the grantees went through in developing their project/intervention from conception to implementation. This is called a process evaluation.

An impact evaluation, on the other hand, would focus on the impact the grantees’ interventions had on their target audience(s). If a grantee expects A to occur as a result of their target audience(s)being exposed to their projects, did it really happen? If not, what factors may have contributed to this and how can they be addressed.

(more…)

24 May, 2017

Asset Mapping as a Community Organizing Tool

By Nicole Clark, LMSW|2021-08-19T19:34:14-04:00May 24th, 2017|Categories: Research & Evaluation|Tags: program evaluation, Research|0 Comments

Let’s say you are part of an advocacy group that promotes the health benefits of vegetarianism.

You attend a community town hall where residents speak their concerns over what’s occurring in the community and request resources that can benefit the community.

You stand up to the microphone, introduce yourself and your advocacy group, and make a suggestion to host a gathering for residents on vegetarianism. You share all the good statistics: Eating a plant-based diet high in fiber, folic acid, and a whole bunch of vitamins can reduce high cholesterol, lead to better weight management, lower blood pressure, and reduce the risk of heart disease. You make a suggestion of having your group facilitate a number of workshops in the community.

A resident raises her hand and stands up. Eating lots of fruits and vegetables sounds good and all, she says, but she and other residents have tried unsuccessfully to bring a farmers market to the community. There’s community support for a farmers market as it will aid in increasing food security in the community, but there are concerns that getting food at a farmers market will be more expensive compared to the community grocery store. Also, the community cannot decide which area would be best to have the farmers market. The community grocery store is located near the community’s most used subway station. And speaking of the grocery store–it’s not a very pleasant shopping experience. Not only does the meat smell rancid, the fruits and vegetables look questionable. Many of the items are past the expiration date, and the store’s electronic benefits transfer (EBT) system doesn’t work most of the time. Having a farmers market in the neighborhood will not only give the community another option to buy food, but they can also use their EBT cards to buy items at the farmers market.

Another resident stands up. He’s interested in creating a community garden because he likes the idea of growing his own food so he doesn’t have to deal with the community grocery store. There are plenty of vacant lots in the community, and he knows of a large lot of land near the community’s recreation center. There’s a”For Sale” sign but he doesn’t know the first thing in purchasing land and wants to know if other member are interested in buying the lot with him. In fact, he has no idea how to grow fruits and vegetables.

One of the most social-worky phrase you will ever hear is “meet the client where they’re at”. When you’re working with an individual (or a community, for this example), you may have your own agenda. Despite best intentions, if you can’t understand what matters to the community, their concerns and their successes, you will never find a way “in”. In order to build a connection between yourself and the community, you have to establish trust. Your advocacy group’s ultimate goal is helping communities eat healthier, and this community is interested doing that, but maybe not in the way you envision. So, what’s one way to marry your goal to the goals of this community?

Conduct a community asset map.

(more…)

Previous 123 Next

Try This: Curb Your Evaluation Anxiety

Here’s what you need:

The steps:

How to Weave Storytelling with Statistics

While five percent of the students remembered a statistic, 63 percent remembered the story.

Try This: Appreciative Inquiry

What’s appreciative inquiry?

Here’s what you need:

Are You Focusing on The Process or The Impact?

Asset Mapping as a Community Organizing Tool

This Is A Custom Widget