Six Questions To Ask & Answer Before Conducting An Impact Evaluation

Six Questions To Ask & Answer Before Conducting An Impact Evaluation

Rotary District 5960 Impact Evaluation Summary

By Kathy Stutzman, M.A., M.A.

Rotary District 5960 is developing a relevant impact evaluation; including why and when impact evaluation is important and what we hope to accomplish through an impact evaluation.  In this particular initiative, Rotary District 5960 was conducting a pilot project, developing new methodologies and conducting new research about delivering services in the world in different ways. This executive summary reviews six questions that are critical to ask and answer before conducting an impact evaluation.

Impact Evaluation Summary

When conducting an “evaluation” the groups involved need to be clear about the following questions prior to beginning:

• What are we evaluating?
• Why are we evaluating?
• When are we evaluating (timeframe)?
• Who is conducting the evaluation, and who is being surveyed?
• What instruments are being used? Why? Standardized? Ease of collation, accessibility?
•  What are we going to do with the data?

These questions MUST be answered before you begin developing any type of evaluation. Clarity and understanding of what you hope to accomplish is critical to the fidelity of the evaluation.

In any research or pilot project there are many levels of evaluation used, as knowledge and information evolves and as lessons learned require modification of direction or process. As part of this effort we studied a presentation by Jacqueline Novogratz of the Acumen Fund about new techniques that Acumen has been exploring in order to evaluate social impact. This is a brief summary:

  

1.      Pioneer Capital Evaluation is measuring lessons learned, inspiration and replicability. This evaluation is a longitudinal study.  This is being done through reviewing original goals, objectives and hypothesis and charting them against lessons learned and the replicability of the overall Initiative. Funding, volunteer engagement and resources leveraged are part of the data being analyzed. The Simple Poverty Scorecard for Nicaragua developed by Microfinance Risk Management, L.L.C. is one of the measurement tools utilized in this evaluation.

2.      Growth Capital Evaluation is measuring increased capacity, leadership and project planning.

a.       The first evaluation instrument used was a “Community Assessment Survey” administered by the community itself and the local NGO, to the entire subject community.  This survey was a valuable resource in determining demonstrated readiness and viability of project planning as well as indicators of community leadership and increased capacity. In the future it is recommended  the survey questions become more standardized, the baseline questions from the Poverty Index be included and the data is analyzed using a statistical computing program.  However, for purposes of this evaluation, the information that was extrapolated from the survey was useful in determining impact.

b.      A Process and Relationship Assessment” was conducted in person amongst a delegation from D-5960, the local NGO, and community members. The survey consists of a questionnaire which uses standardized questions to measure relationships, leadership and capacity. Also included were questions to gather antidotal information in order to inspire improvements and opportunities for growth.

c.       Relationships are an instrumental part of the Initiative, and there needs to be effort taken to include relationship questions during the pioneer capital stage of the development.

d.      Additional measurements of impact of growth capital include the increased social ties and leveraged resources and funding for identified plans – demonstrating the fidelity and increased capacity, leadership and project planning.

3.      Project or Impact Evaluation: Projects will have their own inherent evaluations and those evaluations may be funder driven, goal and objective driven or be determined by community demographics. To measure the effectiveness of the impact of the Initiative on the overall economic well-being of a community, the measurements will come from funder, or project driven evaluations, baseline data collected in the “Community Assessment” and a pre-and post-test of the Simple Poverty Scorecard for Nicaragua. An additional matrix has been developed to collate relevant pieces of data across the spectrum of project related evaluations.

Before any group begins dedicating resources to “impact evaluation” please answer the first six questions listed above. Once you are clear about the how, why and whens, carefully consider who will be spearheading the evaluation process and how much of your resources you are going to dedicate to the evaluation process. Begin your evaluation conversation informed and by sharing a common language about what you hope to accomplish in your impact evaluation. It is through those conversations that each organization can answer the “impact evaluation” question in a manner that works best for your own group.

Respectfully Submitted, Kathy Stutzman, M.A., M.A. kastutzm@smig.net (507) 219-0912 http://www.linkedin.com/in/kathystutzman or www.KathyStutzman.blogspot.com

This initiative and resulting evaluations are the collective work of countless people.  This executive summary is an excerpt from a larger body of work that an evaluation committee is developing to measure impact within the Rotary District 5960’s poverty eradication initiative in Nicaragua. We are currently in the early stages of disseminating information from evaluations and progress made over the past seven years. This summary is submitted solely by the author, who is a member of the Initiative’s Steering Committee and the evaluation subcommittee.  To learn more about Rotary District 5960’s work on this impact evaluation, you may contact the author.

This article was written as a reply to Quentin Wodon's question posed on Linkedin and on Rotarian Economist asking if impact evaluations are important in Rotary click on this link to read that article: Impact Evaluations: Do We Need Them in Rotary

Kathy Stutzman has been a Rotarian in the Austin Rotary Club for 24 years and has been a member of Rotary District 5960’s Fast For Hope Committee since 2007 when the group began exploring new ways to serve as Rotarians in the world. As a result of her experience within Rotary and the Fast For Hope Committee specifically, she was asked to speak at TEDxHoracePark in March of 2014 http://tedxtalks.ted.com/video/The-Power-Responsibility-of-One  and her work with a team of women in Ghana was featured in The Rotarian Magazine in August 2014, http://therotarianmagazine.com/out-of-chaos  As a consultant she has been conducting evaluations for businesses and organizations since 1999. An author and facilitator, Ms. Stutzman loves creating connections and then “watching the magic happen.”

Get Focused and Make an Impact

This relevant, timely and thought-provoking class offered me an opportunity to relate social impact investing to much of the work I am engaged in. My classmates were innovative and passionate about alleviating poverty and our project focused on articulating the model of poverty alleviation we have developed and measuring the impact of our work.

No small task, but this group is up to the challenge. Seven years later, it is important for us to stay focused and select the variables for the matrix carefully and intentionally otherwise we will overwhelm and be overwhelmed with data and outcomes. I am pretty excited at being this close to being able to tell the story and begin replicating the model.


Want to know more? Contact me via e-mail at kastutzm@smig.net, via Twitter @KathyStutzman or visit me on Facebook or LinkedIn and we will set up a time to connect. I am available to come to your organization or conference and present or conduct workshops. Get focused today and see what kind of impact you can be making in the world.

Getting Our Arms Around Social Impact Investing

Introduction

I have been a member of a volunteer Think Tank since 2007 that is wrestling with a lot of complex issues related to exploring new methods of delivering service through Rotary in a meaningful and sustainable manner with the ultimate goal of eradicating poverty. I recently spoke at Kathy Stutzman at TEDxHoracePark about some of my experiences related to our work.

A statement which I wrote at the beginning of my involvement in this was “Think Big, Stay Focused” since we were dreaming of changing/impacting the world. Since then several sayings that have been helpful to me are to remember also include: “You eat an elephant one bite at a time”, and “Pick the low hanging fruit.”

I am currently in the process of writing a “Lessons Learned” book which will describe many of the experiences and lessons that we encountered during this process, and there are many articles and presentations written about the work we are doing that I will reference as we go along.

The purpose of this particular series of articles is to process and interconnect the work of the Think Tank and a class that some of us from the group are taking from Acumen Acumen called “Making Sense of Social Impact: Acumen’s Building Blocks for Impact Analysis.” There are nine of us taking the class as a cohort, each week; we get our lessons and study materials from Acumen, work on our own to prepare for the class doing homework and then we meet weekly for four weeks. At the end of each lab or cohort class, we write a report and submit that report to +Acumen so that we can receive our certificate at the end of the class.

As I was writing my report I realized that many of the concepts and thoughts I was reflecting on and writing about relate directly to the work that I am doing in my professional life and so will post my reports and thoughts about Social Impact here. Enjoy, ask questions, and send feedback. I invite vibrant discussion!

Making Sense of Social Impact Report from Lab 1

Our group is a volunteer, committed, think tank that has been working on new models of delivering service through Rotary in a meaningful and sustainable manner following emerging concepts and ideas resulting from the introduction of “The End of Poverty, the UN Millennium Project and then subsequent study, activities and research.

We know very clearly what we do not want, after seeing years of our strategic priorities imposed upon others, “doing” project to others, and not engaging the community on which we were focusing in the planning, implementation and evaluation. We have seen millions of dollars thrown at projects that made us feel good when we created them, but ultimately, were not grounded in true sustainability. What we don’t want is often used as a baseline to describe what we are doing although we have created a vision and goals; they are ever evolving as we learn more and engage new partnerships complete with new sets of lessons learned.

1.       One of the first lessons learned is to develop a common set of definitions to describe our visions, dreams and ideas and that is one of the goals for this class: our group will work to develop some common definitions to better describe the breadth, depth and focus of the work in which we are engaged and the concepts we will define include:

·         Capacity Building

·         Sustainable

And then – what are we evaluating (evaluation measures) when looking at:

·         Breadth of Impact;

·         Scope of Impact and

·         Focus of Impact

Several aha’s from the class this week included:

2.       Each “partner/group” needs to come to the table with their own set of goals and visions and then work together to explore and define the intersection of those visions on which they will move forward together. Developing a common set of goals where within the intersection does not mean letting go of each individual set of goals, but enhances and strengthens the partners’ goals.

3.       Each individual brings to the table their own perspective and frame about how we are measuring success and all are important and need to be acknowledged and captured; understanding that some successes support the individual partners’ goals and some demonstrate successes of the intersection.

4.       Capacity building is the foundation of sustainability. We need to get very clear about the components of capacity building that are driving the sustainability of the project because both concepts are often mis-represented and mis-used.

5.       Several of the key components of capacity building include:

·         Being detached to outcome;

·         Supporting resources, tools and skill development for empowerment of the focus community;

·         Patience to give time and opportunities for the focus community to learn, develop and practice new skills for empowerment and sustainability;

·         Creating choices and opportunities in a manner that supports dignity, and does not undermine established empowerment;

·         Developing relationships by being present;

·         Coming to agreement as a group about timelines and evaluation measures;

·         Delivering, engaging in and creating resources, skills and tools that can be left behind and continue after the interventionists are gone.

6.       This is complex and difficult work and there are no easy fixes; even listing the components implies simplicity, which would not be accurate – each component is a study within itself.

7.       We are on the right track, there are huge successes and extremely important lessons learned that we have captured, now the task is sifting through the seven years of work and sharing the stories in ways that are meaningful, replicable and sustainable.

8.       There is much left to be done. Just because we are seven years down the road and have achieved a high level of success, this is our time to continue to press forward and this class provides a great foundation to do just that.

As a group we came to consensus about tabling what to “name” ourselves. The initiative with which we have been engaged needs a more clear identity and it is hoped that through the course of this study we will be able to determine a name that best describes our work.

No big conclusion here, lots of work left to be done and yet, there are many committed, talented people prepared and ready to do so – can’t wait for next week.

Useful Group Exercise

After our lab class watched the video of Sasha Dichter http://youtu.be/KzdrRqXswas about Acumen’s investment model, our group was asked to share a little about what social impact meant to us and when we had experienced/learned about it. We are a diverse group with a broad perspective of life and educational experiences and although we are all focused on the topic that drew us together, we each described our first experiences of observing social impact. Interestingly, our perspectives covered the gamut of the 3 parts of social impact; breadth, depth and focus.

From storytelling, school-based learning and the beginning stages of volunteer engagement – all of us shared what social impact meant to us and how we learned about it. I can’t wait to plot each of the experiences and stories into a chart of the 3 parts and start looking at that (I am a visual learner).

For organizations and groups that are beginning to talk about social impact; the lives reached, jobs created, the impact on the lives being served, and increases in well-being – a great “get-to-know-you” exercise is to watch the Acumen video http://youtu.be/KzdrRqXswas , reflect and then share your experiences. This exercise helped me understand the framework from which each participant drew upon. Of course, the best thing you could do is to register for the class the next time it is offered, but until then – enjoy the lessons learned of those of us who are currently participating.

For me this contributed to the greatest ah-ha moment: an understanding about why it is so hard to capture the impact in one or two sentences – there are so many measures (successes) within each of the components of social impact, that coming to a common understanding and definition of the components of the framework and then charting those successes within the framework – will create a broader impact understanding; first among our group, and then the greater community.

Respectfully submitted – Kathy Stutzman, M.A., M.A.