Six Questions To Ask & Answer Before Conducting An Impact Evaluation
Rotary District 5960 Impact Evaluation Summary
By Kathy Stutzman, M.A., M.A.
Rotary District 5960 is developing a relevant impact
evaluation; including why and when impact evaluation is important and what we
hope to accomplish through an impact evaluation. In this particular initiative, Rotary District
5960 was conducting a pilot project, developing new methodologies and
conducting new research about delivering services in the world in different
ways. This executive summary reviews six questions that are critical to ask and
answer before conducting an impact evaluation.
Impact Evaluation Summary
When conducting an “evaluation” the groups involved need to
be clear about the following questions prior to beginning:
- What
are we evaluating?
- Why
are we evaluating?
- When
are we evaluating (timeframe)?
- Who
is conducting the evaluation, and who is being surveyed?
- What
instruments are being used? Why? Standardized? Ease of collation,
accessibility?
- What
are we going to do with the data?
These questions MUST be answered before you begin developing
any type of evaluation. Clarity and understanding of what you hope to
accomplish is critical to the fidelity of the evaluation.
In any research or pilot project there are many levels of
evaluation used, as knowledge and information evolves and as lessons learned
require modification of direction or process. As part of this effort we studied
a presentation by Jacqueline Novogratz of the Acumen Fund about new techniques that
Acumen has been exploring in order to evaluate social impact. This is a brief
summary:
1. Pioneer
Capital Evaluation is measuring lessons learned, inspiration and replicability.
This evaluation is a longitudinal study.
This is being done through reviewing original goals, objectives and
hypothesis and charting them against lessons learned and the replicability of
the overall Initiative. Funding, volunteer engagement and resources leveraged
are part of the data being analyzed. The Simple Poverty Scorecard for Nicaragua
developed by Microfinance Risk Management, L.L.C. is one of the measurement
tools utilized in this evaluation.
2. Growth
Capital Evaluation is measuring increased capacity, leadership and project
planning.
a.
The first evaluation instrument used was a
“Community Assessment Survey” administered by the community itself and the
local NGO, to the entire subject community. This survey was a valuable resource in
determining demonstrated readiness and viability of project planning as well as
indicators of community leadership and increased capacity. In the future it is
recommended the survey questions become
more standardized, the baseline questions from the Poverty Index be included
and the data is analyzed using a statistical computing program. However, for purposes of this evaluation, the
information that was extrapolated from the survey was useful in determining
impact.
b.
A Process and Relationship Assessment” was conducted
in person amongst a delegation from D-5960, the local NGO, and community
members. The survey consists of a questionnaire which uses standardized
questions to measure relationships, leadership and capacity. Also included were
questions to gather antidotal information in order to inspire improvements and
opportunities for growth.
c.
Relationships are an instrumental part of the
Initiative, and there needs to be effort taken to include relationship questions
during the pioneer capital stage of the development.
d.
Additional measurements of impact of growth
capital include the increased social ties and leveraged resources and funding
for identified plans – demonstrating the fidelity and increased capacity,
leadership and project planning.
3. Project
or Impact Evaluation: Projects will have their own inherent evaluations and
those evaluations may be funder driven, goal and objective driven or be
determined by community demographics. To measure the effectiveness of the
impact of the Initiative on the overall economic well-being of a community, the
measurements will come from funder, or project driven evaluations, baseline
data collected in the “Community Assessment” and a pre-and post-test of the
Simple Poverty Scorecard for Nicaragua. An additional matrix has been developed
to collate relevant pieces of data across the spectrum of project related
evaluations.
Before any group begins dedicating resources to “impact evaluation”
please answer the first six questions listed above. Once you are clear about
the how, why and whens, carefully consider who will be spearheading the
evaluation process and how much of your resources you are going to dedicate to
the evaluation process. Begin your evaluation conversation informed and by sharing
a common language about what you hope to accomplish in your impact evaluation.
It is through those conversations that each organization can answer the “impact
evaluation” question in a manner that works best for your own group.
This initiative and resulting evaluations are the
collective work of countless people. This
executive summary is an excerpt from a larger body of work that an evaluation committee
is developing to measure impact within the Rotary District 5960’s poverty
eradication initiative in Nicaragua. We are currently in the early stages of
disseminating information from evaluations and progress made over the past
seven years. This summary is submitted solely by the author, who is a member of
the Initiative’s Steering Committee and the evaluation subcommittee. To learn more about Rotary District 5960’s
work on this impact evaluation, you may contact the author.
Kathy Stutzman has been a Rotarian in the Austin Rotary
Club for 24 years and has been a member of Rotary District 5960’s Fast For Hope
Committee since 2007 when the group began exploring new ways to serve as
Rotarians in the world. As a result of her experience within Rotary and the
Fast For Hope Committee specifically, she was asked to speak at TEDxHoracePark
in March of 2014
http://tedxtalks.ted.com/video/The-Power-Responsibility-of-One and her work with
a team of women in Ghana was featured in The Rotarian Magazine in August 2014,
http://therotarianmagazine.com/out-of-chaos
As a consultant she has been conducting
evaluations for businesses and organizations since 1999. An author and
facilitator, Ms. Stutzman loves creating connections and then “watching the
magic happen.”
