This is part four in a four-part series on program evaluation, dedicated to organizations and businesses that provide programs and services for women, girls, and communities of color (and for people with an interest in evaluation practice). Throughout this month, I will be discussing certain aspects of evaluation practice –from how I became interested in evaluation, myths about evaluation, knowing what type of evaluation to perform, and bringing your community together to support evaluation – with the intent on highlighting the importance of evaluation not just from a funding perspective, but from an accountability and empowerment perspective.
In Part One, I shared how I got started in evaluation practice. In Part Two, I shared some of the common myths about evaluation that can cause us to look at evaluation negatively. In Part Three, I shared how asking the right questions is the key to successfully evaluating your program or service.
Now that you’ve determined why you want to evaluate your program or service, and you’ve decided if you should evaluate as the program or service is in the development stages or at its conclusion (or both!), it’s time to determine what type of data you want to collect.
The Basics
When it comes to research and evaluation, there are two types of data: quantitative data and qualitative data.
With quantitative data, you’re collecting information that can be mathematically analyzed in a numerical fashion. You want to use quantitative data:
*To see what correlations exists between various factors
*To gather demographics (age, gender, race, grade level, etc.)
*To get the answers for “who”, “what”, and “how many” of an occurrence
*To draw a more generalized conclusion about a population
You can collect quantitative data through:
*Pre- and post-tests
*Surveys
*Questionnaires
*Brief telephone interviews or in-person interviews
In comparison, qualitative data is collected when you want to analyze a more narrative form of data that can’t be mathematically analyzed. You want to use qualitative data:
*To get more in-depth explanations between correlated factors
*To gain insights into behaviors and experiences of a population
*To get the answers for “why” and how” something is occurring
*To have a “voice” within a population rather than a generalization
You can collect qualitative data through:
*Observation
*Focus groups
*In-depth interviews (with stakeholders, program participants, or staff members for example)
*Case studies
Here Is an Example of Each
Let’s revisit our example from Part Three:
An evaluation of 11 high schools across the Lubbock, Texas school district looked at peer influence as a key component of delayed onset of sexual activity of the district’s mandated abstinence-based sex education curriculum. Let’s say that we expect the result of our program to be that students are heavily influenced by their peers in whether they are successful at delaying onset of sexual activity (vaginal, anal, and oral sex.)
An example of a quantitative question: Who are the students involved in the program? (Age, race, gender, sexual experience at time of program, etc.)
With this question, you’re collecting data on the participants’ demographics (the WHO) to determine who is participating in your program or service.
An example of a qualitative question: Do you feel that your peers play a role in whether or not you delay sexual activity?
With this question, you’re collecting more of a narrative as to why your participants feel they way that they do (the WHY). While this question can be answered as an open-ended question on a survey, by asking this question within a focus group or an in-depth interview setting, you’re able to get more detailed information.
Which is Best?
Comparing quantitative data and qualitative data is like comparing apples to oranges. They are both useful forms of data, like apples and oranges are delicious types of fruit. How will you know to use which form of data collection? It depends on how you want to answer your evaluation question.
Deciding between quantitative and qualitative methods is largely based on your evaluation questions as well as on the practicality of collecting your data. A clearer way to decide between quantitative and qualitative data collection is to decide how specific or how general you want to be.
If you want something quick that is meant to draw more of a generalization, then quantitative data is the best route to take. If you want to know why a behavior is occurring and you want to go to a source for insight, then qualitative data is the way to go.
For more detail, here are some guidelines for choosing between quantitative data and qualitative data:
Use quantitative data when
You want a large number of opinions
You want to know how many times an answer is given
Numerical data is more important than non-numerical data
You’ve already conducted qualitative data, and you want to know what a large sample thinks about the program or service
You believe that your participants will provide thoughtful answers to your questions, and that qualitative data wouldn’t be necessary
You have an idea of what the key issues are
Your organization or agency has the capacity to enter and analyze the numerical data using statistical software (such as SPSS)
Use qualitative data when
You’ve already conducted quantitative data, and you want to know what a small representation thinks about the program or service
You’ve noticed some glaring discrepancies in your quantitative data
You want to hear people’s insights and feelings
You want to know if there are any cultural, gender, community, or historical implications for why a behavior is occurring
You have little to no idea what the key issues are
Your organization or agency has the capacity to record the qualitative data (via tape recorder, video, hand written, social media, etc.) and organize that data based on themes and common occurrences. An example of this is my What I Wish My Mom Would Have Told Me About Sex responses.
Use both
Your quantitative data leaves you with the feeling that something is amiss. Either the responses were too vague, too similar, or inconsistent. When this occurs, follow up with some in-depth interviews or a focus group. You can share the numerical data within this setting to get better insight on why the responses occurred.
Or maybe you’d rather create a survey instead to gather quantitative data. You can use a focus group to test out the questions you’ve already developed to see what works, what doesn’t, how each question is being conveyed, and whether the questions need to be closed-ended (Yes/No), open-ended (“Why do you feel…?”), or both.
It’s perfectly acceptable to do both. In fact, many evaluations adopt a combination of quantitative and qualitative data. This is ideal because, while you want to have statistically significant data to share, having explanations to back up the data makes the numbers come alive. Some things to also consider include current resources at your disposal and time restraints. These can often determine which route you should take.
Thank You!
Thanks for participating in the Program Evaluation for Women & Girls series. I hope you’ve found this series useful. I will definitely continue to share more informative blogs about various aspects of evaluation, as well as go more in depth with the topics presented in this series. There’s so much to evaluation practice, and when broken down, evaluation can become interesting, informative, and even a lot of fun. As a nonprofit, agency, or business that’s dedicated to raising the voices of women and girls of color, I have no doubt that you can learn evaluation practice quickly, tweak your current evaluation practices, and take your program or service to the next level.
Need more extensive training on evaluation? Check out my consulting services page and contact me to hire me as an evaluation consultant for your program or to work with you staff to build up your evaluation skills.
