Friday, December 21, 2007

A diversity of measures

(Reposted from Alliance Magazine)

What do public benefit, performance metrics and evolution all have in common? No, it is not just the tendency of some to dismiss them all as ‘merely theories’. It is that they all rely on diversity to thrive. Whether you are looking at the provision of social services in Britain, the arts community in Berlin, or affordable housing in Boston, you will find a mix of non-profit organizations, government agencies and private companies providing the benefits. With regard to performance metrics and measures of impact, we have seen an explosion of indices for corporate social responsibility, social return on investment, and double bottom line accounting. And, of course, evolution is inherently linked to biological diversity, which is itself a lynchpin of a healthy region, ecosystem, and, ultimately, planet.

It is ironic, then, that one of the most consistent challenges to developing meaningful outcome or performance measures in the social sector is the old canard that ‘there are too many definitions of success’. In other words, success is too diverse to be measured by any common standards. Critics of measurement efforts will argue that some of the social sector is focused on school choice programmes, while others are trying to expand public investment in public schools. These entities are working at cross-purposes – how can there be one definition of success?

The answer is simple: the search for measures, indices, metrics and standards need not be monolithic. They will be useful when they are as diverse as the sector(s) that they measure. This is why the Alliance/Keystone survey and analysis of the state of impact measures in civil society is so important – we must know what we already know in order to improve.

And the survey findings are illustrative if not scientifically testable or comprehensive. Donors and civil society organizations agree on the value of collecting data across several domains, from activity level, participant reach and productivity to direct and indirect change. There is widespread agreement that evaluation is under-resourced.

There is, not surprisingly, some difference of opinion – for example, about the utility of various evaluation processes, and the degree to which evaluative processes are under-funded – and there are differences in how each side views the other in terms of follow-up. What does all this tell us about impact evaluation in civil society?

There are several findings that could advance the sector as a whole, if they are applied with an eye towards improvement and not obstruction. First, we have some way to go in developing broadly useful practices and distributing them widely. Second, when we shift our gaze from individual organizations to the sectors as a whole, we find workable test groups, each developing suites or clusters of measures and processes that make sense in certain situations. Third, it is this level of analysis that may hold the most promise for developing comparable, widely applicable measures.

The Keystone/Alliance survey doesn’t answer our question – how do we measure success, but it is an important step forward in categorizing the stakeholders involved, and their perceptions of what works and what doesn’t, and identifying future directions for action. It is true, we may not be able to boil everything down to a single measure that allows for universal comparison of food security programmes in Ghana with cultural preservation efforts in Grenada or immigrant rights programmes in Greenwich. But wait a minute – we don’t have a single standard in any other sector, either.

This kind of diversity of measures also characterizes commercial activity and politics. We pretend that there are standard measures that cut universally across commercial activity or political action, such as market share, stock price or electoral victories. But this is not really the case. Some stock analysts rely on price/earnings ratios, others pay them no mind. Some investors develop super computer-powered formulae. Others, Warren Buffett famously among them, only buy companies that make products that the investor actually understands and uses.

Similarly, while electoral victories are one measure of success in democratic politics, there are many others. Some put more weight on the long-term development of ideological strongholds, others pursue political ends through the court system, and others put their focus on local issues over national politics, based on their beliefs about control and the avenues of change. In both markets and politics there are many measures of success, and many applications thereof. It is likely that such is true in the civil sector as well.

We have confused the need to measure our impact with the need to develop single common metrics. The Alliance/Keystone survey is valuable for several reasons, including its contribution to debunking this myth. The universe of organizations that contribute to public benefit and social good is expanding and morphing and hybridizing. The types and interests and wishes of donors are doing the same. We need to measure how we are doing, but we need to do so in ways that are useful and flexible, not simply available and comparable. I have written at length about the costs of settling for less – for using measures simply because they exist and can be gathered, even when they don’t actually tell us what we want to know – see It is time to move beyond metrics of convenience to those that can help us make better investments, provide better programmes, and make a difference that matters.

This survey gives us hope that we can move towards complex and useful metrics by bringing forward new ways of looking at this age-old problem. We need to look at areas of work, not necessarily organizational structure. We need to look beyond that which is easy to count to that which may be meaningful. And we can get further than we’ve ever been before if we are willing to consider the diversity in the sector – its structures, actions and outcomes – as a potential source for the answer rather than an insurmountable obstacle.

No comments: