Monday, September 07, 2009

Metastasizing Metrics: Panel summary from SoCap09

I had the honor of facilitating a session on metrics at last week's Social Capital Markets conference. I also had the duty of apologizing for the name of the session, which was Metastasizing Metrics. I didn't name it, I just moderated it.

I've done a lot of moderating. This session called for every trick in the book. We had five incredible panelists (Kevin Starr, Mulago Foundation; Brad Presner, Acumen Fund; Margot Brandenburg, Rockefeller Foundation and IRIS; Laura Callanan, McKinsey & Co.; Steve Wright, SalesForce.com Foundation), each of whom has spent years working on their piece of this complicated puzzle. So of course, in typical panel fashion, we were going to ask them to describe their work in 2 minutes or less. And then we were going to take questions from the 100+ people who'd come to learn about their work, try to get good discussions going, highlight the very important tensions in this work, and point people to more information. All in 60 minutes. Ready. Go.*

So that is what we did. Here are two independent reviews of the session - one from Stephanie McAuliffe of the Packard Foundation and the other from Jack Samuelson of GIIN. (If you know of other reviews posted on the web please link to them in the comments. Thanks.)

My thoughts on all this:
  • The panel represented measuring efforts from several vantage points - a single foundation (Mulago), a portfolio of projects (Acumen, Pulse), social investors' reporting needs (IRIS), across a sector (McKinsey's database) and within a context that might allow benchmarking (SalesForce.com and others). How these pieces could or will or won't fit together something we didn't get to really talk about.
  • Two inquiries from the room - one about the current state of starvation of established enterprises and another from a new entrepreneur reflecting the abstract terror that the metrics discussion was fomenting in him - were really important, and no real discussion of how they relate to each other or the full range of panel expertise was possible.
  • What is the "gold standard" in evaluation? Is there such a thing? An impassioned disagreement about random control trials got started in Room 370 of Fort Mason, but we didn't have the chance to hear from more than 2 people in the room and certainly didn't push the state of understanding on this issue much further.
  • I do, in fact, think metrics are the carbon of the social impact ecosystem and that the policy environment is the oxygen. We better get the balance right. Thanks to all who picked up on this new metaphor I've been trying out.
As unfortunate as the title of the session was, it did cause each of the panelists and me independently to look up the meaning of the word metastasizing. It means to spread or grow by changing position or form. The negative connotations come because the word is almost exclusively used in reference to cancer. However, spreading or growing by changing is not, inherently, a bad thing. I do hope those in the session, or readers who click through to read the reviews of the session above, note that we have reached a critical point in the discussion about metrics. Time was, metrics were optional, elusive, or unknown. Now they are plentiful, pervasive, and dynamic. That calls for a different discussion than we've had in the past. I hope we started one at #SoCap09. And that we will find many ways to continue it.



*If it is not obvious from my tone, I think this whole panel structure thing is a particularly ineffective way to learn and find it amazing that we - collectively - just keep falling back to it. Then again, I don't organize conferences, I just attend, speak, and moderate at them.

7 comments:

Kevin Jones said...

great post lucy. did you stay around for thursday, the day the community built their own sessions and took as long as they needed to?

Lucy Bernholz said...

Kevin
I was only able to stay for the beginning of Thursday and I should say that the design of the day seemed like it had great potential. I hope people made the most of it, and followed through on the issues raised at the metrics session and elsewhere.
Lucy

Kevin Jones said...

next year I'd love to involve you more. i am trying to figure out how the for profit and the non profit capital markets intersect. it's kind of confusing to me at this point.

Unknown said...

Lucy, I was there. One of the 100.

I should have said it then because I said it often after: You did a brilliant job. You're a terrific moderator. A rare capacity.

Another point raised that might rise in importance was the need for leading as well as lagging + current indicators. Forecasting is unrealistic, but what are we doing in metrics that helps us choose among future options?

Jeremy Heigh said...

Lucy, I was there. One of the 100.

I should have said it then because I said it often after: You did a brilliant job. You're a terrific moderator. A rare capacity.

Another point raised that might rise in importance was the need for leading as well as lagging + current indicators. Forecasting is unrealistic, but what are we doing in metrics that helps us choose among future options?

Unknown said...

Jeremy
Thanks for the moderating shout out - much appreciated. Yes - thanks for reminding me of this key question (asked by Rhesa Jenkins, I think) - what (if any) examples of leading indicators of change are out there? This is an important frame - it may be a way of making some sense of the difference between performance measures and outcome measures; it may be a way to tie together what are currently parallel conversations about operating expenses, capacity, revenue sources, outcomes and metrics.

Lucy

Jeremy Heigh said...

Absolutely (and sorry for the double post).

My biggest interest in "impact investment" is that it creates an appetite for the future. Seeking impact is a future oriented intention. And it's generally longer-term than pure profit.

I agree with you. "Outcomes" and "metrics" are not necessarily the same. Metrics seem, generally, to focus on the past and present. Outcomes (or impact) would be tailored to present and future.

Fun.