Six Questions To Ask & Answer Before Conducting An Impact Evaluation

Six Questions To Ask & Answer Before Conducting An Impact Evaluation

Rotary District 5960 Impact Evaluation Summary

By Kathy Stutzman, M.A., M.A.

Rotary District 5960 is developing a relevant impact evaluation; including why and when impact evaluation is important and what we hope to accomplish through an impact evaluation.  In this particular initiative, Rotary District 5960 was conducting a pilot project, developing new methodologies and conducting new research about delivering services in the world in different ways. This executive summary reviews six questions that are critical to ask and answer before conducting an impact evaluation.

Impact Evaluation Summary

When conducting an “evaluation” the groups involved need to be clear about the following questions prior to beginning:

• What are we evaluating?
• Why are we evaluating?
• When are we evaluating (timeframe)?
• Who is conducting the evaluation, and who is being surveyed?
• What instruments are being used? Why? Standardized? Ease of collation, accessibility?
•  What are we going to do with the data?

These questions MUST be answered before you begin developing any type of evaluation. Clarity and understanding of what you hope to accomplish is critical to the fidelity of the evaluation.

In any research or pilot project there are many levels of evaluation used, as knowledge and information evolves and as lessons learned require modification of direction or process. As part of this effort we studied a presentation by Jacqueline Novogratz of the Acumen Fund about new techniques that Acumen has been exploring in order to evaluate social impact. This is a brief summary:

  

1.      Pioneer Capital Evaluation is measuring lessons learned, inspiration and replicability. This evaluation is a longitudinal study.  This is being done through reviewing original goals, objectives and hypothesis and charting them against lessons learned and the replicability of the overall Initiative. Funding, volunteer engagement and resources leveraged are part of the data being analyzed. The Simple Poverty Scorecard for Nicaragua developed by Microfinance Risk Management, L.L.C. is one of the measurement tools utilized in this evaluation.

2.      Growth Capital Evaluation is measuring increased capacity, leadership and project planning.

a.       The first evaluation instrument used was a “Community Assessment Survey” administered by the community itself and the local NGO, to the entire subject community.  This survey was a valuable resource in determining demonstrated readiness and viability of project planning as well as indicators of community leadership and increased capacity. In the future it is recommended  the survey questions become more standardized, the baseline questions from the Poverty Index be included and the data is analyzed using a statistical computing program.  However, for purposes of this evaluation, the information that was extrapolated from the survey was useful in determining impact.

b.      A Process and Relationship Assessment” was conducted in person amongst a delegation from D-5960, the local NGO, and community members. The survey consists of a questionnaire which uses standardized questions to measure relationships, leadership and capacity. Also included were questions to gather antidotal information in order to inspire improvements and opportunities for growth.

c.       Relationships are an instrumental part of the Initiative, and there needs to be effort taken to include relationship questions during the pioneer capital stage of the development.

d.      Additional measurements of impact of growth capital include the increased social ties and leveraged resources and funding for identified plans – demonstrating the fidelity and increased capacity, leadership and project planning.

3.      Project or Impact Evaluation: Projects will have their own inherent evaluations and those evaluations may be funder driven, goal and objective driven or be determined by community demographics. To measure the effectiveness of the impact of the Initiative on the overall economic well-being of a community, the measurements will come from funder, or project driven evaluations, baseline data collected in the “Community Assessment” and a pre-and post-test of the Simple Poverty Scorecard for Nicaragua. An additional matrix has been developed to collate relevant pieces of data across the spectrum of project related evaluations.

Before any group begins dedicating resources to “impact evaluation” please answer the first six questions listed above. Once you are clear about the how, why and whens, carefully consider who will be spearheading the evaluation process and how much of your resources you are going to dedicate to the evaluation process. Begin your evaluation conversation informed and by sharing a common language about what you hope to accomplish in your impact evaluation. It is through those conversations that each organization can answer the “impact evaluation” question in a manner that works best for your own group.

Respectfully Submitted, Kathy Stutzman, M.A., M.A. kastutzm@smig.net (507) 219-0912 http://www.linkedin.com/in/kathystutzman or www.KathyStutzman.blogspot.com

This initiative and resulting evaluations are the collective work of countless people.  This executive summary is an excerpt from a larger body of work that an evaluation committee is developing to measure impact within the Rotary District 5960’s poverty eradication initiative in Nicaragua. We are currently in the early stages of disseminating information from evaluations and progress made over the past seven years. This summary is submitted solely by the author, who is a member of the Initiative’s Steering Committee and the evaluation subcommittee.  To learn more about Rotary District 5960’s work on this impact evaluation, you may contact the author.

This article was written as a reply to Quentin Wodon's question posed on Linkedin and on Rotarian Economist asking if impact evaluations are important in Rotary click on this link to read that article: Impact Evaluations: Do We Need Them in Rotary

Kathy Stutzman has been a Rotarian in the Austin Rotary Club for 24 years and has been a member of Rotary District 5960’s Fast For Hope Committee since 2007 when the group began exploring new ways to serve as Rotarians in the world. As a result of her experience within Rotary and the Fast For Hope Committee specifically, she was asked to speak at TEDxHoracePark in March of 2014 http://tedxtalks.ted.com/video/The-Power-Responsibility-of-One  and her work with a team of women in Ghana was featured in The Rotarian Magazine in August 2014, http://therotarianmagazine.com/out-of-chaos  As a consultant she has been conducting evaluations for businesses and organizations since 1999. An author and facilitator, Ms. Stutzman loves creating connections and then “watching the magic happen.”