This is part four in a four-part series on program evaluation, dedicated to organizations and businesses that provide programs and services for women, girls, and communities of color (and for people with an interest in evaluation practice). Throughout this month, I will be discussing certain aspects of evaluation practice –from how I became interested in evaluation, myths about evaluation, knowing what type of evaluation to perform, and bringing your community together to support evaluation – with the intent on highlighting the importance of evaluation not just from a funding perspective, but from an accountability and empowerment perspective.

In Part One, I shared how I got started in evaluation practice. In Part Two, I shared some of the common myths about evaluation that can cause us to look at evaluation negatively. In Part Three, I shared how asking the right questions is the key to successfully evaluating your program or service.

Now that you’ve determined why you want to evaluate your program or service, and you’ve decided if you should evaluate as the program or service is in the development stages or at its conclusion (or both!), it’s time to determine what type of data you want to collect.

The Basics

When it comes to research and evaluation, there are two types of data: quantitative data and qualitative data.

With quantitative data, you’re collecting information that can be mathematically analyzed in a numerical fashion. You want to use quantitative data:

*To see what correlations exists between various factors

*To gather demographics (age, gender, race, grade level, etc.)

*To get the answers for “who”, “what”, and “how many” of an occurrence

*To draw a more generalized conclusion about a population

You can collect quantitative data through:

*Pre- and post-tests

*Surveys

*Questionnaires

*Brief telephone interviews or in-person interviews

In comparison, qualitative data is collected when you want to analyze a more narrative form of data that can’t be mathematically analyzed. You want to use qualitative data:

*To get more in-depth explanations between correlated factors

*To gain insights into behaviors and experiences of a population

*To get the answers for “why” and how” something is occurring

*To have a “voice” within a population rather than a generalization

You can collect qualitative data through:

*Observation

*Focus groups

*In-depth interviews (with stakeholders, program participants, or staff members for example)

*Case studies

Here Is an Example of Each

Let’s revisit our example from Part Three:

An evaluation of 11 high schools across the Lubbock, Texas school district looked at peer influence as a key component of delayed onset of sexual activity of the district’s mandated abstinence-based sex education curriculum. Let’s say that we expect the result of our program to be that students are heavily influenced by their peers in whether they are successful at delaying onset of sexual activity (vaginal, anal, and oral sex.)

An example of a quantitative question: Who are the students involved in the program? (Age, race, gender, sexual experience at time of program, etc.)

With this question, you’re collecting data on the participants’ demographics (the WHO) to determine who is participating in your program or service.

An example of a qualitative question: Do you feel that your peers play a role in whether or not you delay sexual activity?

With this question, you’re collecting more of a narrative as to why your participants feel they way that they do (the WHY). While this question can be answered as an open-ended question on a survey, by asking this question within a focus group or an in-depth interview setting, you’re able to get more detailed information.

Which is Best?

Comparing quantitative data and qualitative data is like comparing apples to oranges. They are both useful forms of data, like apples and oranges are delicious types of fruit. How will you know to use which form of data collection? It depends on how you want to answer your evaluation question.

Deciding between quantitative and qualitative methods is largely based on your evaluation questions as well as on the practicality of collecting your data. A clearer way to decide between quantitative and qualitative data collection is to decide how specific or how general you want to be. (more…)

This is part three in a four-part series on program evaluation, dedicated to organizations and businesses that provide programs and services for women, girls, and communities of color (and for people with an interest in evaluation practice). Throughout this month, I will be discussing certain aspects of evaluation practice –from how I became interested in evaluation, myths about evaluation, knowing what type of evaluation to perform, and bringing your community together to support evaluation – with the intent on highlighting the importance of evaluation not just from a funding perspective, but from an accountability and empowerment perspective.

So far, we’ve discussed some possible “WHYs” of evaluation practice (from the benefits of evaluating your programs and services, seeing if the objectives of your program or service is currently meeting the needs of your participants, to looking at the misconceptions of evaluation and how they can affect your work). Now, let’s switch gears and focus on WHAT you’re evaluating and WHEN to evaluate. This part of the series is trickier than the others, but I want to touch on the basics so that you have a working knowledge on this important part of evaluation. This is by no means complete list. If you have a question about anything in particular (logic models, strategic plans, etc.) or would like me to give more examples of this week’s topic, please let me know in the comments below and I can follow-up with additional blog posts outside of this series.

What Are You Evaluating?

In order to get to your destination, you need to know where you’re going. In order to do this, we need to develop a strategy that will guide you in how you will look at your data. This will help you determine if your producing the results you’re expecting. This is where evaluation questions come in. An evaluation question helps you look at your data to see if your program or service is producing its intended objectives.

There are two types of evaluation questions: a process evaluation question and a results-focused question. A process question wants to know how the program is functioning. How a program functions depends on a variety of factors, such as the length of the program, the number of participants, the activities being offered in the program, how the participants interpret ad interact to the activities, and so forth. In other words, the who, what, when, and how of the program’s implementation. Process questions are especially useful when you’re in the beginning stages of planning your program; however, they can be asked throughout the program so that you’re always thinking ahead and adjusting your program’s implementation.

A result-focused question, on the other hand, wants to know if the program is accomplishing the results you’re expecting. In other words, how effective is your program, and are your participants benefiting from the program in the way you’ve intended? Results-focused questions typically follow the completion of a program.

Now that we know more about the types of evaluation questions, let’s look at when each question comes into play. (more…)