Evaluating Success: No Laughing Matter
Most Not-for-Profits can tell you how busy they are. How many people attended. How many sessions were delivered. How many calls were answered. How many volunteers participated. How much funding was distributed. These numbers matter. But they do not necessarily tell an organisation whether it is succeeding. The harder question is what changed because the organisation was there.
For Australian NFPs facing increasing expectations from government, philanthropy, Boards and communities to demonstrate outcomes, that distinction is becoming increasingly important.
A recent learning paper from Comic Relief's Change Makers Programme provides a useful reminder. Drawing on the experience of 20 partnerships involving 45 organisations, the report found that effective monitoring, evaluation and learning works best when it is focused, proportionate and adaptable. It also found that trying to measure too much can produce exactly the opposite result: overloaded staff, disengaged participants and large volumes of inconsistent data that provide little useful insight.
The lesson for Australian NFPs is straightforward. Evaluation should not begin with the question, "What can we measure?" It should begin with, "What do we need to know?"
Measurement should help you make decisions. Monitoring and evaluation can easily become a compliance exercise. A funding agreement requires six monthly reporting, so data is collected. A Board wants a dashboard, so another set of indicators is created. A strategic plan contains KPIs, so someone starts counting them. Before long, an organisation can have substantial amounts of data without necessarily knowing much more about whether its work is effective.
The Change Makers experience found that evaluation became more useful when organisations shifted from thinking primarily about reporting to thinking about learning. Instead of collecting information because it might be useful later, organisations identified the decisions they needed to make and worked backwards to determine what evidence would help them make those decisions.
That is particularly relevant in Australia.
The Australian Government's Not-for-profit Sector Development Blueprint identifies outcomes measurement as an increasingly important part of government and philanthropic funding and investment decisions. At the same time, it acknowledges that the diversity of the sector makes a single approach inappropriate and that measurement can impose significant costs, particularly in prevention, early intervention, advocacy and systems change work.
The Blueprint goes further, calling for better sector capability, more appropriate tools and frameworks, and approaches that produce information organisations can actually use to improve strategy, practice and operations. That should matter to smaller NFPs in particular.
A community organisation does not need the monitoring framework of a national service provider. A volunteer run association should not be attempting to replicate an academic impact study. Nor should a $2 million organisation devote scarce delivery resources to collecting dozens of indicators simply because they look impressive in a spreadsheet.Good evaluation is proportionate. A useful test is simple: if nobody can explain what decision will be influenced by a particular piece of data, ask why the organisation is collecting it.
Not everything that matters can be neatly attributed. One of the more important lessons from the Change Makers work concerns the difference between direct service delivery and broader systems change. Where an organisation provides a direct service, measurement may be relatively straightforward. A training program can measure changes in knowledge or confidence. A counselling service might examine participant wellbeing over time. An employment program may track whether participants obtain or retain work.
Even here, good evaluation generally requires more than counting activity. The Change Makers organisations combined quantitative information with participant feedback, practitioner observations and qualitative evidence to build a fuller picture of what was happening. They also embedded feedback during delivery rather than waiting until the end of a program.
But many Australian NFPs do more than provide direct services.Thus, there may be no clean line between one activity and one outcome. Change may take years. Several organisations may contribute. Government policy, economic conditions and community attitudes may also change along the way. Trying to prove that one organisation "caused" the final outcome can therefore create misleading evaluation.The better concept is often contribution rather than attribution.
What evidence suggests that our work contributed to the change? What intermediate changes can we see? Did knowledge improve? Did another organisation adopt a practice? Did a government agency engage with the issue? Did relationships strengthen? Did the issue move onto a policy agenda?
The Change Makers paper found that systems change evaluation often needs a stronger theory of change, intermediate indicators, qualitative evidence and an acceptance that meaningful outcomes may emerge over longer periods.
This is consistent with longstanding Australian thinking. The Productivity Commission has distinguished between inputs, outputs, outcomes and longer term impacts and noted that there is no single "gold standard" measurement approach suitable for every NFP or every purpose. The Australian Centre for Evaluation similarly emphasises evaluation as a tool for continuous improvement and decision making and asks a particularly useful question: what works, why, and for whom? For an NFP Board, that is a much more useful conversation than simply receiving a table of KPIs. Do not make the people you serve carry the evaluation burden
There is another issue NFPs should consider carefully. More data is not always better data.
The Change Makers organisations found that excessive surveys and forms could reduce participation, particularly where people were already dealing with complex systems or vulnerable circumstances. Evaluation could begin to feel extractive rather than useful. Participants could reasonably wonder why they were repeatedly being asked for information and what difference providing it actually made.
This deserves particular attention in Australian community services.
People accessing disability, housing, family violence, mental health, refugee, homelessness and other support services may already be repeatedly asked to tell their story to government agencies, service providers and funders.
+An NFP should have a good reason before becoming one more organisation asking the same questions.
+Good evaluation therefore asks not only whether information would be useful, but whether collecting it is justified.
+Can existing information answer the question?
+Can the survey be shorter?
+Can feedback be built into an existing conversation?
+Can people choose how they participate?
+Are questions culturally appropriate and accessible?
+Will participants ever hear what the organisation learned from their contribution?
These are not simply methodological issues. They are questions about respect, trust and organisational values.
The NFP Sector Development Blueprint similarly recognises that outcomes measurement can create unnecessary burden and that approaches need to be fit for purpose, minimise transaction costs and avoid harm.
What should Boards and CEOs ask?
The Board does not need to design the organisation's evaluation framework. It should, however, know whether the organisation can demonstrate that its activities are contributing to its purpose. This is fundamentally a governance question.
Australian charities are required to work towards their charitable purpose, while good governance requires Boards to understand whether organisational activities are effective and appropriately managed.
The Australian Institute of Company Directors has similarly argued that impact measurement should move Boards beyond purely financial and operational reporting towards understanding the outcomes being created for clients, communities and systems. Its guidance recommends focusing on impact, measuring only what matters, keeping approaches simple and embedding findings into organisational decision making.
For many NFP Boards, five questions would substantially improve the conversation:
+What change are we actually trying to create?
+What evidence would reasonably tell us whether that change is occurring?
+Are we measuring outcomes, or mainly counting activities?
+What have we learned from the information we collect, and what have we changed because of it?
+Are we asking staff, partners and the people we serve to provide more information than we genuinely need.
Evaluation should not be an exercise performed at the end of a grant so that a report can be submitted. At its best, it is part of how an organisation learns.
It helps a Board understand whether strategy is working. It helps management decide where resources should go. It helps practitioners adapt services. It helps funders understand what is realistic. And, importantly, it gives NFPs the confidence to stop doing things that consume resources without producing sufficient benefit.
That last point can be uncomfortable.
Not every initiative works. Not every strategic assumption proves correct. Not every program produces the intended outcome. A mature evaluation culture does not treat that information as failure. It treats it as evidence. For NFPs operating in an environment of constrained funding, increasing demand and growing expectations to demonstrate impact, that may be the most important reason to evaluate at all. Success is not how much information an organisation can collect.
It is whether the organisation is prepared to learn from what the information tells it.
NFP Success assists Australian Not-for-Profits and charities with strategic planning, theory of change, governance, organisational reviews and practical approaches to measuring outcomes and organisational performance.
