Six Questions To Ask & Answer Before Conducting An Impact Evaluation

Six Questions To Ask & Answer Before Conducting An Impact Evaluation

Rotary District 5960 Impact Evaluation Summary

By Kathy Stutzman, M.A., M.A.

Rotary District 5960 is developing a relevant impact evaluation; including why and when impact evaluation is important and what we hope to accomplish through an impact evaluation.  In this particular initiative, Rotary District 5960 was conducting a pilot project, developing new methodologies and conducting new research about delivering services in the world in different ways. This executive summary reviews six questions that are critical to ask and answer before conducting an impact evaluation.

Impact Evaluation Summary

When conducting an “evaluation” the groups involved need to be clear about the following questions prior to beginning:

• What are we evaluating?
• Why are we evaluating?
• When are we evaluating (timeframe)?
• Who is conducting the evaluation, and who is being surveyed?
• What instruments are being used? Why? Standardized? Ease of collation, accessibility?
•  What are we going to do with the data?

These questions MUST be answered before you begin developing any type of evaluation. Clarity and understanding of what you hope to accomplish is critical to the fidelity of the evaluation.

In any research or pilot project there are many levels of evaluation used, as knowledge and information evolves and as lessons learned require modification of direction or process. As part of this effort we studied a presentation by Jacqueline Novogratz of the Acumen Fund about new techniques that Acumen has been exploring in order to evaluate social impact. This is a brief summary:

  

1.      Pioneer Capital Evaluation is measuring lessons learned, inspiration and replicability. This evaluation is a longitudinal study.  This is being done through reviewing original goals, objectives and hypothesis and charting them against lessons learned and the replicability of the overall Initiative. Funding, volunteer engagement and resources leveraged are part of the data being analyzed. The Simple Poverty Scorecard for Nicaragua developed by Microfinance Risk Management, L.L.C. is one of the measurement tools utilized in this evaluation.

2.      Growth Capital Evaluation is measuring increased capacity, leadership and project planning.

a.       The first evaluation instrument used was a “Community Assessment Survey” administered by the community itself and the local NGO, to the entire subject community.  This survey was a valuable resource in determining demonstrated readiness and viability of project planning as well as indicators of community leadership and increased capacity. In the future it is recommended  the survey questions become more standardized, the baseline questions from the Poverty Index be included and the data is analyzed using a statistical computing program.  However, for purposes of this evaluation, the information that was extrapolated from the survey was useful in determining impact.

b.      A Process and Relationship Assessment” was conducted in person amongst a delegation from D-5960, the local NGO, and community members. The survey consists of a questionnaire which uses standardized questions to measure relationships, leadership and capacity. Also included were questions to gather antidotal information in order to inspire improvements and opportunities for growth.

c.       Relationships are an instrumental part of the Initiative, and there needs to be effort taken to include relationship questions during the pioneer capital stage of the development.

d.      Additional measurements of impact of growth capital include the increased social ties and leveraged resources and funding for identified plans – demonstrating the fidelity and increased capacity, leadership and project planning.

3.      Project or Impact Evaluation: Projects will have their own inherent evaluations and those evaluations may be funder driven, goal and objective driven or be determined by community demographics. To measure the effectiveness of the impact of the Initiative on the overall economic well-being of a community, the measurements will come from funder, or project driven evaluations, baseline data collected in the “Community Assessment” and a pre-and post-test of the Simple Poverty Scorecard for Nicaragua. An additional matrix has been developed to collate relevant pieces of data across the spectrum of project related evaluations.

Before any group begins dedicating resources to “impact evaluation” please answer the first six questions listed above. Once you are clear about the how, why and whens, carefully consider who will be spearheading the evaluation process and how much of your resources you are going to dedicate to the evaluation process. Begin your evaluation conversation informed and by sharing a common language about what you hope to accomplish in your impact evaluation. It is through those conversations that each organization can answer the “impact evaluation” question in a manner that works best for your own group.

Respectfully Submitted, Kathy Stutzman, M.A., M.A. kastutzm@smig.net (507) 219-0912 http://www.linkedin.com/in/kathystutzman or www.KathyStutzman.blogspot.com

This initiative and resulting evaluations are the collective work of countless people.  This executive summary is an excerpt from a larger body of work that an evaluation committee is developing to measure impact within the Rotary District 5960’s poverty eradication initiative in Nicaragua. We are currently in the early stages of disseminating information from evaluations and progress made over the past seven years. This summary is submitted solely by the author, who is a member of the Initiative’s Steering Committee and the evaluation subcommittee.  To learn more about Rotary District 5960’s work on this impact evaluation, you may contact the author.

This article was written as a reply to Quentin Wodon's question posed on Linkedin and on Rotarian Economist asking if impact evaluations are important in Rotary click on this link to read that article: Impact Evaluations: Do We Need Them in Rotary

Kathy Stutzman has been a Rotarian in the Austin Rotary Club for 24 years and has been a member of Rotary District 5960’s Fast For Hope Committee since 2007 when the group began exploring new ways to serve as Rotarians in the world. As a result of her experience within Rotary and the Fast For Hope Committee specifically, she was asked to speak at TEDxHoracePark in March of 2014 http://tedxtalks.ted.com/video/The-Power-Responsibility-of-One  and her work with a team of women in Ghana was featured in The Rotarian Magazine in August 2014, http://therotarianmagazine.com/out-of-chaos  As a consultant she has been conducting evaluations for businesses and organizations since 1999. An author and facilitator, Ms. Stutzman loves creating connections and then “watching the magic happen.”

Investing for Impact - Start by Building Capacity

Developing relationships is a critical component
in capacity building and poverty alleviation

 This week’s Social Impact Lab created a great deal of spirited discussion prior, during and after our class meeting.  We have been working on this new method of working in the world in a more sustainable, and impacting manner for seven years and our lab group is made up of people who have varying perspectives, experiences and length of time within the think tank – so we are well-rounded and all have opinions. As well, we are willing to listen to what is being said and what is not being said because despite our variation in perspectives, we are all grounded in the vision.

The vision has shifted and changed as we have grown, learned and experienced over the years, adding partnerships, knowledge and resources. This week’s lab class gave us a framework around which to articulate some of the shifts as we shared our ideas, insights, and frustrations with charting all of the impact and outcomes that we have seen and captured. And interestingly enough it was the model case of d.light.com that brought much of the discussion to a head, and got me to my ah-ha moment.

This reflection is my own, although I have shared it with the group for feedback and clarity, and I think I am onto a critical understanding of how to articulate our greatest struggle – what are we really trying to do – what is our real impact?

It is so easy to start calculating the social impact of our investments in the roads and bridges, schools and books, water delivery systems and health centers and that is exactly what we did not want to do – invest in a top-down, project-driven, imposition of our own strategic priorities. Yet, every time we look at what is happening in the community in which we are actively engaged, we start problem solving about using those successes (outputs) to tell our story.

The leadership of El Corozo prepares for a community meeting

So backing up from the roads and bridges is the real story – how did the community get to the point where THEY were empowered to organize their community; developing their own strategic plans, building capacity to train others, learning about choices and options so that they could identify and select those choices which would most closely align with their vision. And probably most importantly develop enough confidence in their competence that they could collectively as an organized community say no to aid offered which did not fit their strategic plan (despite the good it could provide.)

In a recent response about developing capacity, there was a question related to “people development” and Acumen’s response alluded to people development not being a strategy to alleviate poverty and I had a physical reaction as did several others in the think tank with whom I am working. That response stirred me to finally be able to articulate exactly why capacity development is the most critical component to sustainable poverty alleviation strategies.

Without empowered leadership within the organization, community or group with whom you are trying to alleviate poverty, the projects (regardless of the financial return on investment) will be an imposition of someone else’s strategic priorities. Projects don’t eradicate poverty. In a recent TEDx, TEDxHoracePark, I outlined this very problem and some solutions. You can view my opinions about this by clicking here:  Kathy Stutzman - TEDxHoracePark

 Creating the environment where a community is organized, empowered and has confidence in their own competence – that is capacity building – that creates opportunities for choices and options to be identified and selected – that is where we will see a more efficient and greater return on our investment. That part of social impact investing is critical, expensive and time-consuming and that is what we are working to develop, evaluate and replicate. For purposes of this study, I am going to call this a Tier 1 Impact.

Here is my synopsis of my ah-ha moment from Lab 2 of the +Acumen Course of Investing for Social Impact:

The Assumption:

We (District 5960 Rotarians and friends) will explore and develop a process through which we could engage in creating sustainable choices and options for economic well being in a poverty-stricken community in Nicaragua, with whom we had no prior existing relationship.

The Tier 1 Impact:

Empowered, focused leadership within the community will be active and instrumental in organizing, and directing the community to discover and explore new opportunities and options and work with partners to articulate the community’s choices, not simply accepting someone else’s strategic priorities imposed upon them. (Demonstrated Readiness)

As a Result:

The community will efficiently and sustainably incorporate, and implement appropriate economic initiatives and will actively participate and be vested in the investments chosen. (Higher and more efficient returns on investments)

How we are accomplishing this is another chapter, however, as a demonstration of the impact of building capacity please view this video which was recently produced by our local NGO partners and features the leaders of the community of El Corozo. Capacity Building and Leadership in El Corozo Nicaragua

For more information about the work of our think tank, please feel free to contact me, or any of the members of our group.

Getting Our Arms Around Social Impact Investing

Introduction

I have been a member of a volunteer Think Tank since 2007 that is wrestling with a lot of complex issues related to exploring new methods of delivering service through Rotary in a meaningful and sustainable manner with the ultimate goal of eradicating poverty. I recently spoke at Kathy Stutzman at TEDxHoracePark about some of my experiences related to our work.

A statement which I wrote at the beginning of my involvement in this was “Think Big, Stay Focused” since we were dreaming of changing/impacting the world. Since then several sayings that have been helpful to me are to remember also include: “You eat an elephant one bite at a time”, and “Pick the low hanging fruit.”

I am currently in the process of writing a “Lessons Learned” book which will describe many of the experiences and lessons that we encountered during this process, and there are many articles and presentations written about the work we are doing that I will reference as we go along.

The purpose of this particular series of articles is to process and interconnect the work of the Think Tank and a class that some of us from the group are taking from Acumen Acumen called “Making Sense of Social Impact: Acumen’s Building Blocks for Impact Analysis.” There are nine of us taking the class as a cohort, each week; we get our lessons and study materials from Acumen, work on our own to prepare for the class doing homework and then we meet weekly for four weeks. At the end of each lab or cohort class, we write a report and submit that report to +Acumen so that we can receive our certificate at the end of the class.

As I was writing my report I realized that many of the concepts and thoughts I was reflecting on and writing about relate directly to the work that I am doing in my professional life and so will post my reports and thoughts about Social Impact here. Enjoy, ask questions, and send feedback. I invite vibrant discussion!

Making Sense of Social Impact Report from Lab 1

Our group is a volunteer, committed, think tank that has been working on new models of delivering service through Rotary in a meaningful and sustainable manner following emerging concepts and ideas resulting from the introduction of “The End of Poverty, the UN Millennium Project and then subsequent study, activities and research.

We know very clearly what we do not want, after seeing years of our strategic priorities imposed upon others, “doing” project to others, and not engaging the community on which we were focusing in the planning, implementation and evaluation. We have seen millions of dollars thrown at projects that made us feel good when we created them, but ultimately, were not grounded in true sustainability. What we don’t want is often used as a baseline to describe what we are doing although we have created a vision and goals; they are ever evolving as we learn more and engage new partnerships complete with new sets of lessons learned.

1.       One of the first lessons learned is to develop a common set of definitions to describe our visions, dreams and ideas and that is one of the goals for this class: our group will work to develop some common definitions to better describe the breadth, depth and focus of the work in which we are engaged and the concepts we will define include:

·         Capacity Building

·         Sustainable

And then – what are we evaluating (evaluation measures) when looking at:

·         Breadth of Impact;

·         Scope of Impact and

·         Focus of Impact

Several aha’s from the class this week included:

2.       Each “partner/group” needs to come to the table with their own set of goals and visions and then work together to explore and define the intersection of those visions on which they will move forward together. Developing a common set of goals where within the intersection does not mean letting go of each individual set of goals, but enhances and strengthens the partners’ goals.

3.       Each individual brings to the table their own perspective and frame about how we are measuring success and all are important and need to be acknowledged and captured; understanding that some successes support the individual partners’ goals and some demonstrate successes of the intersection.

4.       Capacity building is the foundation of sustainability. We need to get very clear about the components of capacity building that are driving the sustainability of the project because both concepts are often mis-represented and mis-used.

5.       Several of the key components of capacity building include:

·         Being detached to outcome;

·         Supporting resources, tools and skill development for empowerment of the focus community;

·         Patience to give time and opportunities for the focus community to learn, develop and practice new skills for empowerment and sustainability;

·         Creating choices and opportunities in a manner that supports dignity, and does not undermine established empowerment;

·         Developing relationships by being present;

·         Coming to agreement as a group about timelines and evaluation measures;

·         Delivering, engaging in and creating resources, skills and tools that can be left behind and continue after the interventionists are gone.

6.       This is complex and difficult work and there are no easy fixes; even listing the components implies simplicity, which would not be accurate – each component is a study within itself.

7.       We are on the right track, there are huge successes and extremely important lessons learned that we have captured, now the task is sifting through the seven years of work and sharing the stories in ways that are meaningful, replicable and sustainable.

8.       There is much left to be done. Just because we are seven years down the road and have achieved a high level of success, this is our time to continue to press forward and this class provides a great foundation to do just that.

As a group we came to consensus about tabling what to “name” ourselves. The initiative with which we have been engaged needs a more clear identity and it is hoped that through the course of this study we will be able to determine a name that best describes our work.

No big conclusion here, lots of work left to be done and yet, there are many committed, talented people prepared and ready to do so – can’t wait for next week.

Useful Group Exercise

After our lab class watched the video of Sasha Dichter http://youtu.be/KzdrRqXswas about Acumen’s investment model, our group was asked to share a little about what social impact meant to us and when we had experienced/learned about it. We are a diverse group with a broad perspective of life and educational experiences and although we are all focused on the topic that drew us together, we each described our first experiences of observing social impact. Interestingly, our perspectives covered the gamut of the 3 parts of social impact; breadth, depth and focus.

From storytelling, school-based learning and the beginning stages of volunteer engagement – all of us shared what social impact meant to us and how we learned about it. I can’t wait to plot each of the experiences and stories into a chart of the 3 parts and start looking at that (I am a visual learner).

For organizations and groups that are beginning to talk about social impact; the lives reached, jobs created, the impact on the lives being served, and increases in well-being – a great “get-to-know-you” exercise is to watch the Acumen video http://youtu.be/KzdrRqXswas , reflect and then share your experiences. This exercise helped me understand the framework from which each participant drew upon. Of course, the best thing you could do is to register for the class the next time it is offered, but until then – enjoy the lessons learned of those of us who are currently participating.

For me this contributed to the greatest ah-ha moment: an understanding about why it is so hard to capture the impact in one or two sentences – there are so many measures (successes) within each of the components of social impact, that coming to a common understanding and definition of the components of the framework and then charting those successes within the framework – will create a broader impact understanding; first among our group, and then the greater community.

Respectfully submitted – Kathy Stutzman, M.A., M.A.