Episode 23: A community approach to evaluating programs

Christy Kallevig: Welcome to Vital Connections On Air, a podcast brought to you by the University of Minnesota Extension Center for Community Vitality that explores the trends and topics important to communities and leaders throughout Minnesota. My name is Christy Kallevig and I'm an Extension educator with the Center for Community Vitality. Today I'm joined by Scott Chazdon, evaluation specialist for University of Minnesota Extension Center for Community Vitality, as well as Pam Bishop, who is the vice president of economic development with the Southern Minnesota Initiative Foundation. Welcome to the podcast.

Scott Chazdon: Thanks Christy. Glad to be here.

Pam Bishop: Good morning, Christy.

Christy Kallevig: Thank you so much for joining us today. During our last podcast we had the opportunity to get to know Pam quite well. But Scott, we haven't gotten to meet you before. So would you please take a minute to introduce yourself and tell us about your role in Extension?

Scott Chazdon: Sure. I'm an evaluation specialist with our Center for Community Vitality. I've been with Extension for about 12 ½ half years, and my job is twofold. First, it's to help our Extension educators understand the strengths of our programming, so they can improve the programming. So we have a lot of different efforts to evaluate the outcomes and impacts of all of our Extension programs. And I also work with educators who are interested in different rural community development topics to design research projects and actually help them implement research projects, often working with graduate students here at the University.

Christy Kallevig: Today, it's great to have you both with us so that we can dig into the idea of Ripple Effects Mapping, which Pam shared in our last podcast was a process that rural entrepreneurship ventures used to evaluate their program. Can you give us an explanation of this process, Scott?

Scott Chazdon: Sure. So Ripple Effects Mapping is a group and very participatory method for looking at both the intended and the unintended impacts or results of a program, which makes a lot of sense when you're trying to evaluate community development work. It's never pre-canned kinds of programs where typical evaluation methods work very well. And it's also the kind of work where you want the evaluation to feed that energy of the work that's happening in the community. And ripple mapping really does that. You're gathering people together, they're learning from each other and they're learning about the things that have happened in the community as a result of some program or collaboration.

Christy Kallevig: Yeah, I have had the opportunity to be involved in some ripple mapping with you. And it's always just amazing the energy that comes into the room as people start talking about their shared experiences.

Scott Chazdon: I agree. Yeah, it is. I have a colleague, Kathy Allen, who's a leadership educator and she has a book in which she said that change flows along the lines of relationship. So just by virtue of having a group of people come together to reflect together about some programming in their community is promoting relationships and also promoting people to tell stories about relationships, because it's usually through those relationships that good things happen. When you do a traditional evaluation, let's say it's a survey that goes to one person, there's no kind of relationship happening. You're just kind of collecting the data and it can be very useful to have the data, but you're not actually adding to the intervention or the program by doing the evaluation. Except maybe a report will be read, hopefully.

Christy Kallevig: The thing that I think is always a lot of fun, because I have have gotten those mail surveys, right, where you fill them out and you give your view. But with ripple mapping, it's really interesting to hear another person's experience and that jogs something in your memory, and you are then able to contribute that. So essentially it allows you as an evaluator to get even more data than you would through just a traditional survey. Right?

Scott Chazdon: It definitely does. Yeah. You get, you get a greater depth of information. You have these people, people having “aha moments” with each other, people really actually listening to each other, which often doesn't happen. People taking the time also to reflect together, [which] often doesn't happen. So there's just a lot of benefits to doing it this way, in terms of evaluation, and one of the other benefits is that it's not super expensive or time consuming, even as compared to doing a survey where it takes a lot of time to analyze the data. Often you have to send people multiple reminders to fill out the survey. And with ripple mapping there's some work involved for sure. But it's pretty efficient in terms of the amount of evaluator time, and community member time that goes into it. You're making the most out of people's time.

Christy Kallevig: And that's really important. Today we're all so short of time. You were asked to become involved in the rural entrepreneurship ventures program to do ripple effects mapping. What was kind of the background as to why they felt this strategy might be really good for this new program?

Scott Chazdon: Yes, it was really exciting because, you know, this wasn't originally an Extension program. It comes from the Center for Rural Entrepreneurship. It's been implemented by the Southern Minnesota Initiative Foundation in Minnesota. And so it was a real honor that they wanted to ask us to help evaluate this program and be involved with the coaching of the communities. And what they were looking for, because they already have metrics and things they want to measure about the program, things that they intend to happen in these communities. But they understood that that ripple mapping allows you to surface these unintended things that happen in communities as a result of this entrepreneurship work. And also they like the idea of doing this at a couple of points in time. So in this case, we're doing the ripple mapping one year into the intervention. So after these communities have been working on this program for about a year, we did the ripple mapping and then we're going to do it again after two years, which will be coming up in this coming winter. So I think that there was just a lot of interest in trying out this process because it also feeds the community development activity, in addition to collecting really useful information.

Christy Kallevig: Pam, it sounds as Scott describes this process that it was kind of made to be a part of the REV program. What was your experience or your insights with the ripple effects mapping process?

Pam Bishop: The ripple mapping has been extremely helpful and that's all above all. Thanks to the University of Minnesota, and Scott and Jennifer Hawkins and their work in leading that. I will just tell you that I think anytime you introduce a new concept, there's going to be a place where everyone is extremely excited because it's new, right? And then you take this journey where it kind of starts feeling, well—hard, but this ripple mapping allowed every community to see what was actually happening in their town. And the conversations that were starting to occur, because I believe, because REV was part of the conversation all along.

Christy Kallevig: That is a very powerful testimonial to the process of a ripple effects map. But I know that the information that you heard in the stories that you gathered really came from the questions that are developed. And that is a key part of this process, isn't that correct, Scott?

Scott Chazdon: Yeah, that's a great part of the process where you have these appreciative inquiry questions, which you work on beforehand with the community organization that's running the program. So then you pair people up at the beginning of the process to interview each other. And there's usually three or four questions they ask each other and you ask them to write down what they're hearing on the back of the agenda sheet. Then after they've taken turns interviewing each other, then you gather people up in the front of the room. Looking at it the way I do it, looking at a screen where there's a projector set up projecting onto the screen and people start reporting out maybe two or three things they heard in their interviews and that gets recorded on this mind map.

And at first everything is disconnected and just loose pieces of information. But over the roughly two hour period, you start to organize that loose information into themes and then into stories or chains of effects. What led to what led to what. And that's one of the nice things about this mind mapping software that allows you to show these chains of effects, like right there on the screen. And I think people find it very interesting and they enjoy seeing their words up there and they enjoy seeing how their experience connects to the experiences of others and to the themes that get generated from everyone's experiences.

Christy Kallevig: I think that it's very empowering to them to actually see that their comments are getting put into a document right away. And that just goes back to that very first thing I said. It just feeds the energy in the room.

Scott Chazdon: Yeah, it does. And, without fail, that's, that's what people have to say after a ripple mapping session. That it was a really good experience. That it energized them, that they really appreciated the time. And I have to say that my colleagues, like Deborah Hansen and Mary Emery, have a slightly different way of doing the ripple mapping than I do, but we all agree that we're all talking about the same general process. My colleague in particular, Mary Emery, really likes to have three [pieces of] big butcher paper on the wall [in] the three concentric circles, where the first circle is the immediate impacts on individuals. Then the second circle is the impacts on individuals who know those individuals. And then the third circle is impacts in the broader community. So that really is a very, very specific imagery of circular ripples going out from a central point.

Christy Kallevig: Do you, in the process of  mapping and going through the interviews to gather this information, hear negative comments as well, or does the process really only draw out the positive?

Scott Chazdon: Well, the process starts with appreciative inquiry, which is very much focusing on the positive, but during every ripple mapping session, I like to have a discussion towards the end about what I call the negatives or the challenges that have occurred. In this case, what challenges have occurred as the community participated in rural entrepreneurial venture?

Christy Kallevig: So now that we understand the process of ripple effects mapping a bit better, tell us about the findings that you had through the process with the communities involved in rural entrepreneurship ventures.

Pam Bishop: So I guess just general themes because what's interesting too is that we'll do another round. So this was basically just to get kind of the landscape of what is actually happening.  What's coming out of the ripple mapping is that we're noticing that there's a lot of strength in the relationships that are starting to be built among the businesses themselves. So we know that based on the survey work that is happening, the resources that are being targeted help support these businesses, these relationships are now starting to multiply among the businesses themselves. The community support is building. While we know that volunteerism is strong in all these communities, they are starting to recognize that the work they're doing by volunteering in a multitude of ways actually connecting themselves together, it's helping them build a stronger ecosystem because now they're all working collectively in ways that they didn't even recognize. The ripple mapping is indicating that.

We also noticed more collaboration between the public and private partnerships. There's more of an intentional way that they are starting to build ways to incorporate business development in the work they're doing. The attitudes are shifting. People are now saying this is something that we need to pay attention to and they're recognizing that they need to be thinking about those resources and building the tunnel pool to support the businesses’ needs. There's this whole new energized environment now this happening because now it's starting to elevate their work and we're starting to see more youth engagement, younger students, more with some of the community's being very intentional about building youth entrepreneurship and incorporating that in a way that will allow younger people in the community to realize that the community does care about their future, and if they want to be an entrepreneur, that they're building a culture to support and embrace that.

Christy Kallevig: And I believe that you didn't just map the information that you got through this REM process according to the mind map. You also did some additional mapping with it. Correct, Scott?

Scott Chazdon: What we did, which was interesting as we coded all of these ripple maps according to something called the community capital's framework, which is a framework. You’ve heard people have mostly heard about social capital or human capital. And in rural sociology there's all these capitals combined, seven of them. So political capital, cultural capital, natural capital built capital, and financial capital of course. And so we looked at the data in each community according to the these capitals, but we looked at both the positives and the negatives and coated the information. So we did learn quite a bit about some of the challenges. Some of the communities are facing real shortages of available space for local business. Some communities, pretty much all of them, talked about how hard it is to mobilize and energize people in the community. They also talked about a lack of financial resources to support entrepreneurship. So we had a whole range of different challenges in the communities and it was important that that all got documented in the ripple mapping as well.

Christy Kallevig: What are some key things that a community organization should keep in mind if they want to do a ripple effects mapping?

Scott Chazdon: You Bet. So if a community wants to do ripple mapping, it's best to not evaluate a program you ran yourself. So you do need to find someone who has some outsider status to get trained to do the ripple mapping. It could be someone who is an evaluator or someone who just a really good facilitator of meetings because those are the skills that are the most important for running a ripple mapping. You do need some qualitative, in my opinion research experience or some comfort generating themes from raw data. So when people have comments in the ripple mapping session, you have to be comfortable generating themes or combining things that people have said in meaningful ways.

Christy Kallevig: That's a great point, Scott. And one that I really hadn't given much thought to. Are there other things that people should keep in mind when planning?

Scott Chazdon: The thing you want to evaluate needs to be something that everyone in the room is going to understand. Like everyone in the room has to know this was a program or a project of some organization or some group of organizations. Sometimes I've been involved in ripple mapping where the focus of the ripple mapping session wasn't very clear, and so the conversation went in too many different directions. So it's sort of like you have to know what it is you're evaluating and when you do the ripple map, it’s what shows up in the middle of the ripple map. That's what you're evaluating. And so that's important. Number two, you've got to invite and work hard to recruit people to this thing. It's hard to explain to people what they're being invited to. It's super important to try to explain what ripple mapping is and how people will benefit from it.

It's also helpful to feed people. So we always combine the ripple mapping with a meal. I've learned over time what the best time for the meal is. It's about halfway through the process after people have reported out. Then you break for a meal and while people are having their meal, I can sit there and start to organize the ripple map into some themes, which I can then throw out to the group after they've eaten, and see what they think of those things. But it's nice to have a little break to do that. But we want to recruit a really good group of people, people who are close in to the work that has happened and people who are a little further out. I think of them as your strategic partners, but people who might not have been involved in the day-to-day activities of the program, but who are in a position to make things happen for the program moving forward.

So those could be local media, local government officials, local clergy, movers and shakers or influence leaders in a community are important to invite to a ripple mapping session.

Christy Kallevig: And also probably some elected officials if you're trying to get funding or potential donors.

Scott Chazdon: Yes, definitely. They need to know enough about the program that's being evaluated so they can't be complete[ly] like, “I've never heard of this before.” But you know, they don't have to know all the details because sometimes they'll know that there's some connections that have happened in the community that people who are really close into the work might not have known about.

Christy Kallevig: How do you see people using the ripple effects map?

Scott Chazdon: Once the mapping is done, I've seen different strategies. Some communities have been able to use the map, like the whole big map, printed on a poster to share at different public meetings.

That can really be impressive to some stakeholders or funders to support further work. I've also seen people take the ripple map and maybe write it up as a shorter report to share as an appendix to grant proposal to show examples of the work that's already going on and next community that has created the capacity to do whatever this grant is asking to do. So people have used the ripple maps to really support funding proposals. It's also useful to just share information with the broader community about some activity that's going on in the community that not everyone may know about. So it can become a really useful communication tool to help bring more people in the community on board to some sort of process that's happening.

Christy Kallevig: Absolutely. And I think that that's also the thing that is just very unique about a ripple effects map as well, is that it belongs to the community or the organization that you do it with. This is in data that just comes back to the university or to a specific organization. But it's their information.

Scott Chazdon: It is, and you know, it's really interesting when you do it at multiple points in time, also to document progress towards, towards goals that are important for the community.

Christy Kallevig: I would like to think Scott and Pam for joining us for this podcast and sharing information about Ripple Effects Mapping and also Pam, how it's impacted the communities in Southeastern Minnesota. I look forward to hearing from you both again after the second round has been completed.

Christy Kallevig: Thank you to Scott Chazdon with the UMN Extension Center for Community Vitality and Pam Bishop with the Southern Minnesota Initiative Foundation for joining us for today's episode. To learn more about the rural entrepreneurial ventures program (REV), visit www.smifoundation.org/rev. Visit the University of Minnesota Extension Center for community Vitality webpage at www.extension.umn.edu/community-development  where you will find more resources on supporting entrepreneurs and your community. Make sure to follow us on Facebook and Twitter to stay up-to-date on your research and resources for communities and those who lead them. We hope that you will join us again for a future episode of Vital Connections On Air.