Lesson 3: The eight principles of measuring and valuing impact

The following principles are found in the ‘Measuring and Valuing Social Impact’ Handbook.

Principle 1: Measure social impact in terms of wellbeing

Measuring social impact in terms of wellbeing takes a comprehensive, whole-of-life approach that understands the holistic needs of individuals. Historically, figures like Aristotle have emphasised the significance of a fulfilling life as the ultimate end; this is resonated in many Indigenous cultures' holistic views that include connections to community, place, and generations. It’s also a perspective increasingly supported by organisations like the OECD and the UNDP.

Measuring social impact in terms of wellbeing not only focuses on specific targeted outcomes of your program, but simultaneously captures the capabilities, opportunities, and broader needs of communities. So you can understand where your program aligns to needs, and where there is opportunity to better support needs either through your own programs or in collaboration with others. You can also see where you might have unintended consequences, despite successfully achieving your target outcomes, for example your job placement program might see increases in outcomes relating to income, skills and confidence, but decreases in quality time with family.

Principle 2: Use subjective wellbeing as the overall measure of impact

Historically, many frameworks have tried to encapsulate wellbeing with a one-size-fits-all approach, often overlooking the unique determinants that matter to individuals or cultures. As discussed in module one, by adopting a subjective wellbeing measure, we respect individuals' or community’s sovereign understanding of their own lives. Emphasising the individual doesn't diminish the collective; the cumulative data from individual subjective wellbeing provides a more authentic picture of community wellbeing.

Imagine you are at a conference. You are introduced to a new face. They appear well dressed and well spoken. Their job title indicates they have a tertiary education. You start talking with them and find out they have a spouse and two seemingly delightful children. From everything you can see, this person is succeeding at life. Yet, we all know from living life ourselves that our public facing persona isn’t necessarily an accurate picture of how we feel about our lives. This is the danger of using wellbeing indexes based purely on objective data point such as income, education, relationship status and home ownership. We need to ask – how satisfied are you with your own life? Are you living a life that YOU value.

We acknowledge that measuring subjective wellbeing typically increases the effort and costs of measurement (when compared to collecting objective data), however in most cases the benefits of capturing real impact, improving initiatives based on feedback, and ensuring the absence of unintended consequences far outweigh these considerations.

Principle 3: Measure the lived experience

Measuring the lived experience means understanding the holistic outcomes as they are experienced by individuals given the context of their lives. It is information that must come directly from the perspective of those impacted by an intervention.

In the same way that attempts to infer overall wellbeing from objective proxies such as income and education are fraught with biases and implicit assumptions, so it is with lived experience. Different people subject to the same initiative can have very different experiences, which means that the lived experience cannot be inferred from activities or events but must be measured directly with those impacted. When we make assumptions that activity or output A results in improved outcome B, at best we risk wasting resources on interventions that are unsuccessful. At worst we continue to scale initiatives that are causing harm and distress to people.

This principle is especially important in relation to underrepresented or marginalised groups who tend to have their problems and solutions defined and imposed by others. It’s here that many well-intentioned initiatives have failed to have the expected impact because they do not understand the needs of those affected. Measuring the lived experience provides a direct voice to these communities.

Principle 4: Practice cultural safety

Practicing cultural safety in social impact measurement means evaluating impacted individuals in their own context rather than through an external cultural lens. Often, established norms or the cultural identity of the evaluator shape the measurement framework, which can misrepresent and even perpetuate social issues. For instance, a program designed to improve women's empowerment through employment might negatively impact women who greatly value their caregiver and homemaker roles, by imposing a Western idea of empowerment.

To overcome these concerns, a participatory approach that centres on the values of the impacted groups is vital. Such an approach ranges from having impacted groups ‘inform’ the measurement/research to completely empowering the community to lead the study. Practitioners need to be aware of and challenge power dynamics, especially when engaging with vulnerable groups. Emphasising cultural safety involves more than just stakeholder involvement; it requires ongoing self-reflection and self-awareness by those measuring impact. This ensures the processes are culturally respectful, accurate, and that the tools used are linguistically suitable and safe. The success in creating a culturally safe environment is determined by the impacted group.

Principle 5: Measurement results should be actionable

Social impact measurement should produce data that directly informs actionable decisions, both to understand the effectiveness of an initiative or to identify how impact may be improved. Timing is crucial, as data collection should align with key decision-making points within a program or initative. You will find that many existing approaches simply categorise data without delivering actionable insights, however, an effective measurement system requires quantitative data backed by a solid impact thesis or theory of change. This ensures that the data collected is not just abundant, but actually meaningful in making informed decisions and guiding strategic action.

If the intent to act is not present, then the measurement itself may raise ethical questions – it is likely that any inconvenience or potential harm to participants would outweigh the upsides of measurement where the data won’t be used to improve or inform decisions or activities relating to community wellbeing.

Principle 6: Measurement results should be comparable

Social impact measurements should be designed to allow results to be comparable over time, across various contexts, and between different initiatives. This comparability is crucial for directing resources where they will have the most significant social impact and enables ongoing improvement and adaptation. Traditional output-based metrics are often not suitable for this because they are usually specific to a single initiative. Instead, focusing on outcome-based metrics can provide broader comparability. By measuring outcomes, organisations can better allocate resources among different programs or initiatives, improving the social impact overall.

Principle 7: Measurement should be assurable

Social impact measurement should be conducted with rigour and verifiability akin to the financial system. This ensures minimised bias and enhances credibility. Ideally, the party conducting the measurement and analysis should be separate and independent from the organisation creating impact. Or where that is not possible, some level of independent assurance should be conducted through ad hoc reviews or an audit.

The methodology used for measurement should be well-defined and consistent to ensure repeatability and verifiability, as a measurement approach that creates different results depending on who applies it is questionable. As social impact measurement is often a new endeavour for organisations, building internal capabilities in this area is critical.

Verification can be done at two levels: 1) procedural, confirming the methodology was correctly applied; and 2) substantive, evaluating the quality and validity of the conclusions reached.

 Principle 8: Social impact measurement is a scientific pursuit

Social impact measurement should adhere to scientific principles, promoting transparency and reproducibility. A scientific approach is grounded in evidence-based, repeatable methods, enabling decision-makers to understand the relevance, limitations, and confidence level of the data as they make decisions.

Measurement teams need to be transparent across all areas of the measurement process including the design, data collection, analytical methods, and statistical tests, as well as disclosing any potential bias or conflicts of interest. Reproducibility helps in standardising the approach across different initiatives and time periods, making the data more universally applicable and trustworthy.

It is important to acknowledge that social impact occurs in complex, real-world settings rather than controlled environments like a laboratory. As such it is not always possible to achieve a ‘gold star’ scientific standard, such as using randomised control trials. Hence the importance of recognising the limitations of our measurements. Measurement findings must come with communication about the level of confidence and any assumptions or limitations that are essential for accurate interpretation and application.