Evaluating youth and education programs

Evaluation planning

Planning is one of the most critical steps of the entire evaluation. 

Create a plan in five easy steps and outline the plan with this evaluation template.

1. Program description : What is the essence of the program? Briefly describe the program or project in a couple of sentences. 

2. Purpose: Why are you conducting the evaluation? State why you are spending time and resources to do an evaluation. 

3. Goals: What are your program goals or outcomes? These are the changes that you hope your program will make. Program goals might be different for each group you work with. For example, you might have some goals for youth in your program and other goals for volunteers or staff.  

4. Approach: How are you going to conduct the evaluation? This is your methods section and there is no right or wrong approach. Pick what fits your program and goals. You could do interviews or focus groups. You also could send a survey out to participants. You might even have multiple methods.

5. Accountability: Assign people to each evaluation task. There may be other people involved in the project, but the evaluation needs a point person. Maybe one person will be involved in designing the evaluation and another might be involved in recruiting participants.

Evaluation outcomes

Set your evaluation up for success. 

Are your outcomes

• Practical - Can you actually achieve them in the amount of time you have with your target audience?
• About change - Are they focused on how program participants will be different because of the program?
• Measurable - Can you think of existing or design new ways to measure them?
• Clear - Can someone outside of your organization pick them up and understand them?
• Agreed upon - Is there agreement that you are focusing on the right things?

Practice evaluative thinking with these five strategies

• Know your community: Who are you currently serving and who might you serve in the future? Take the time to assess the strengths and needs of your community and how your program might build upon assets and provide support in needed areas.
• Start with the big picture: What is the vision of your program? What is the ultimate impact you hope to have on your community? How do you want the lives of youth, their families, and society to be different because of youth involvement in your program?
• Define your outcomes: What are your program goals or outcomes? These are the changes that you hope your program will make on youth, families and the community. What do you want youth to learn? How do you want them to behave differently because of what they learn? And how will this contribute to change in the larger environment?
• Reflect early and often: Ask for feedback from your participants throughout your program, not just at the end. Use responses and observations to make changes during program delivery that will keep you on track to achieving the outcomes you set for your program.
• Evaluate: At the end of the program circle back to the big picture and your outcomes.  Use feedback and measurement data to improve your program. 

Training

Take this short training for more details on building outcomes in your program.

Ideas for program planning

Listen to this Department of Youth Development podcast episode to get ideas on how to start new and creative programming. 

Avoid smiley face evaluation

Creative evaluation strategies

Want to collect evaluation data from youth but don't want to do another survey?

• Watch this short video to learn some creative evaluation strategies to use with youth.

Five steps to improve youth evaluation

1. Know your audience: Find out more details about the group including academic level, attention span and how they’ve been involved in evaluation in the past. Youth are tested all the time, so you need to tell them why the evaluation matters. 
2. Get approval: Consent from parents; assent from youth. Let parents know you’re doing the evaluation. Youth under 18 cannot legally consent for themselves, but give youth the power to choose to be part of your evaluation. Their involvement shouldn’t be tied to their continued participation in the program. 
3. Think about reading level: Reading level isn’t always tied to grade. If you’re working with a group of typical 4^th graders, you can expect that half are reading below grade level. Make your questions as easy as possible so that everyone can understand them. Use a readability calculator to test the reading level. If youth struggle, read it for them.
4. Be ready to explain: Evaluation isn’t like standardized test taking. Take time to ensure youth understand and allow them to ask questions throughout. If your survey contains scaled questions, make sure they understand the rating scale. 
5. Test it out: Test the questions ahead of time. Do youth understand? What doesn’t make sense? Tell them: “Think about kids your age and let me know when you get to a question or spot that wouldn’t make sense to someone your age.” This way they don’t have to say that they are confused, they can answer for a friend.

What is a focus group?

Originally called "focused interviews" or "group depth interviews," a focus group is a planned discussion led by a moderator who guides a small group of participants through a set of carefully sequenced (focused) questions in a permissive and non-threatening conversation. The goal is not to reach agreement but to gain participants insights on the topic of discussion.

A focus group is typically five to eight people who have been carefully recruited. These participants are selected because they have certain characteristics in common that relate to the topic of the focus group. The group discussion may be conducted several times with similar types of participants to identify trends and patterns in perceptions. Careful and systematic analysis of the discussions can provide clues and insights as to how a program, product, service, or opportunity is perceived by the group.

While focus groups are an excellent methodology to meet many research and evaluation objectives, there are times to use them and not to use them.

Listen to this Department of Youth Development podcast episode to learn how to do a focus group in an online setting.

Five keys to effective focus group design

1. Decide if focus groups are appropriate: Determine if focus groups are an appropriate and efficient data collection method for your need. Refer to the “when to and not to use focus groups” sections on the website for ideas.
2. Clarify the purpose of the study: Decide what types of information you wish to obtain through the study and what you will do with it. Having a clear purpose makes planning, conducting the groups, analyzing and reporting simpler.
3. Pick your participants carefully and keep your group reasonable: Focus groups use a homogenous, purposeful sample composed of “information-rich” participants. As you begin planning, determine what types of people should be invited to your session. People should only be invited because they have something in common that is related to the goal(s) of your study. While group size can vary, the number of people in a focus group should never be more than 12.
4. Keep your questions focused: The question used in a focus group interview should be carefully sequenced so that they focus more and more specifically on the topic of the study. In other words, the questions should progressively direct the participants into discussing the topic in greater detail and more depth. Write your questions so they are conversational and easy to understand.
5. Find a skillful moderator and assistant moderator: The effectiveness of the focus group moderator is the key to a successful focus group. The moderator’s job is to keep the group “focused” and to generate a lively and productive discussion. As well as being able to plan the group, the moderator needs to have effective leadership and communication skills as s/he will need to recognize how to obtain a balanced input from a diverse group of people. It is important that you also have an assistant moderator, who can help by taking care of details and ensuring the quality of the analysis by taking careful notes, summarizing the discussion at the end and acting as another set of eyes and ears for analysis.

When to use focus groups

• To dive deeper into issues identified through a larger, quantitative study. Focus groups are helpful when the goal is to generate an explanation. If an issue or problem is identified in a quantitative study and you want additional information on why it is an issue or problem, a focus group offers an avenue to gather this information at an affordable cost.
• To pilot testing things, such as ideas, campaigns, surveys, or products. Focus groups can be used to get reactions to plans before big amounts of money are spent in implementation.
• To evaluate programs or products. Focus groups are useful in uncovering if programs and products are working and how they might be improved.

When to not use focus groups

• To make major decisions. While focus groups can provide a wealth of ideas and feedback on perceptions and opinions, qualitative data lacks statistical precision. Focus groups work well in line with a quantitative study, but they should not be used in place of a quantitative study when there is a lot at stake.
• If you need to generalize results to a large population or statistical data is required. The participants of a focus group are often representative of the population, but they are not necessarily a statistically representative sample of the population. The sample sizes are usually too small to draw statistical conclusions about a large population.
• To save time and money in the data collection process. A common myth about focus groups is that they are a quick and cheap way to collect data. Focus groups, like other methodologies, require a great deal of planning and effort in order to be effective. While the meeting itself may last only 1 or 2 hours, it takes time to create an effective set of questions, locate the appropriate participants, and make sense of the data they provide. Recruitment and analysis are especially likely to be expensive and time-consuming, unless the participants are already at hand and the project goals are very limited and direct.
• If a group discussion is not an appropriate forum. The basic goal in conducting focus groups is to hear from the participants about on the topics of interest to the evaluator. This means that focus groups are not a viable option unless we can compose and conduct groups in ways that allow participants to voice their views.
• If the topic is not appropriate for the participants. The match between the evaluators' topics of interest and the participants' ability to discuss those topics is essential for successful focus groups. To assess this match during the planning stages, ask the basic question, “How easy will it be to generate a free-flowing and productive conversation on this topic?”

Create better surveys

Should you send a survey?

• Does a survey method fit your questions? There might be better ways to collect your data. Don't be afraid to be creative.
• Will your audience respond by survey? Think about obvious things like if they can read. Do they trust you enough to give honest feedback? Are they suffering from survey fatigue?
• Can you get a survey into the hands of your audience? Surveys can be given online or in person. What works best for your program?
• How are you planning to use the data? Don't send the survey until you have thought through data use.

Key considerations for developing effective surveys

• Define your purpose: What is the purpose for the survey?  What information is needed in order to meet this purpose?  How does the mode of administration support the nature of the needed information?
• Know your audience: Who is the intended population? What special characteristics do they have that will need to be considered during survey design?
• Find the right words: Language should be simple and free of jargon and technical terms.  Negatively worded and inconsistent options should be avoided as should   items that suggest a response by leading or loading.
• Ask personal questions last: Questions should be ordered to allow them to increase in complexity while leaving sensitive questions and demographic information until the end.  Use logical progression in the survey layout, organizing by topics and using section headers to guide respondents. Items with similar response categories can be grouped together to create a battery of questions.
• Connect to your audience: Establish trust with participants by being clear about the survey’s purpose and outcomes. Help ease the burden of completing the survey by ensuring confidentiality and limiting costs to participants (such as minimizing the time needed to complete the survey and increasing accessibility (i.e. multiple modes of delivery including: (online, by phone, mail, etc.)).

Maximize staff expertise to implement evaluation

Build skills across your organization with these five tips to build evaluation capacity.

• Build the team: Know your team members and build upon their strengths. Who on your team is naturally observant or who on your team has strong opinions? Who is thoughtful about how they design and plan for activities? Tell your group to bring their strengths into the evaluation process.
• Ask the right questions: Your team will be tempted to jump right into what type of survey you will do and what questions you will ask. When you jump straight into methods, you miss out on some of the most important steps in the evaluation process. Slow down and think about: Why are you doing an evaluation? What do you want to learn? How will you use the data? Who else cares about the data?
• Connect to design: Build evaluation into your program from the design stage. Document your program outcomes and how you plan to evaluate them. Program staff will be more engaged if they can see the connection between the evaluation and improvements in the program.
• Call in an expert : Your team might be able to support all your evaluation needs. If you are stuck or need an outside perspective, seek an evaluator for input. An evaluator can give critical advice and notice problems early in the process.
• Involve participants: Get energy from the participants in your programs - both youth and adults. Three big benefits of involving participants are: Increased investment of program staff, grounded evaluation questions and measures, and better use of evaluation findings.

Clean up your charts

• Make a better title.
• Use color to highlight.
• Remove unnecessary clutter.
• Format data labels.
• Increase text size.

Watch this short tutorial  and read this Youth Development Insight blog post about how to clean up your charts in five easy ways.

Data visualization

Listen to this Department of Youth Development podcast episode about why youth workers should care about data viz.

Read this Youth Development Insight blog post about how bad data viz can have bad consequences.