Sunday, December 04, 2011

The data ecosystem

Open up the data!

I feel like I should tattoo this on my forehead. This is what I think about, write about, speak about, and blog about. My forecast for next year (and the years beyond) includes a whole section on big and open data as a resource for social good.

But I don't think data are going to change individuals' behavior directly. Most of use a lot more than data when we make decisions. Daniel Kahneman, a psychologist, won the Nobel prize in Economics for proving that we're not as rational (read: data driven) as we think we are. We factor in lots of other interests and issues besides data when we make choices.

This is especially true in philanthropy. We give for reasons of the heart, personal connections, feeling good, looking good, and doing good. Efforts to shift our giving toward more rational, data driven, informed practices know this - they aim to shift the margins (which are quite big) in order to eventually, perhaps shift some of the middle.

So why am I so focused on opening up data when individuals may not use them? Because data are the most basic organic matter for the ecosystem of social good. They provide fodder for how we identify "the problem," which then plays a huge role in "the solutions" that we build.

I believe that opening up philanthropic data will help enough innovators to think differently about how we change the world. Their efforts will yield new opportunities for the existing system and for all of us.

Stop for a second and think about how data now subtly guide or inform your choices in all sorts of realms - air fares, music, book and movie recommendations, jobs, directions, bank rates, cupcake shops, hairdressers - we all have the option for using more data than ever before when we make these daily decisions. And it's not just the easy stuff, like price. It's the tough stuff, like opinions and reviews, that are now available anywhere, everywhere on seemingly everything. When Angie's list started advertising that you could compare plumbers and doctors I knew we'd turned a corner.

I didn't personally seek out these data. Entrepreneurs saw the value of data as raw materials from which they could put new tools into my hands. Those tools help me do the things I like to do faster, easier, and with better results. And when enough of us start expecting this information to be available it spills over onto how the whole system works.

In philanthropy, we're just moving beyond the most basic information. Basic data on operational overhead is widely available and is beginning to power some new tools that let you contrast nonprofits. But that approach still assumes that you and I care about that administrative ratio comparisons first and foremost when we make a gift. Some of us do, but most of us don't.

This approach also sees data as an end-point in the decision making process, not as an input to thinking about solutions.

What we need is an approach to data that goes beyond the basic quantitative comparisons and gets to the level of how we solve problems. If we shared information on where private and public money flows in a community, or where needy people spend their days, or how much food gets wasted and where, or we could hear directly from our elderly neighbros, or help artists connect with each other we might imagine whole new approaches to our shared problems.

What if we could match something like RelayRides (neighborhood car swaps) with TaskRabbit (small job doers) with volunteers for the elderly so that we could help our neighbors keep their doctor appointments and avoid the ER? Or use data on car sharing services to reroute busses so they serve the areas that really need them? Or use the Twitter patterns of food trucks to help identify cohorts of professionals who might be willing to volunteer? Used the opt-in text messages of young people to engage them in community or public service?

What can we learn from giving patterns? Of individuals, corporations and foundations? We don't really know because we don't really have these data in the form that would allow us to know. We also don't have the ability to mix giving data with shopping data, political giving, voting patterns, faith traditions, or other potentially useful information. Charitable giving is an enormous part of our communities, yet we haven't cracked a way to use the aggregate information on money flow, causes, organizations, or donors so we can see this pervasive activity in any kind of meaningful context.

I see philanthropic data as a "nutrient" in a healthy, diverse ecosystem of social solutions. The open government movement has accomplished some of this with public data - and we have better 311 systems, better public transit, and quicker response times for public works departments as just some of the results.

Early, proprietary efforts at mashing up philanthropic data are being used to develop strategy maps and plot grants by location. These are great first steps.

When we get to the point when you can track funding from all investors (impact investors, philanthropy, government) or monitor particular organizations or enterprises that interest you, then we can influence the flow of capital.

When we can see entire  ecosystems of organizations and funding by issue, geography and population, we can engage communities, guide public policy, and fund accordingly.

And when we can map and mine patterns of success and supply, we will inspire the next era of change makers to expand what works and build what's missing.

Open data are the fuel to make all of this happen.


Andrea Schneider said...

I couldn't agree more about the need for good data, not data for data's sake. The suggestions you make are excellent for what kind of data might actually make a difference in how and what we fund.
In addition, I want to suggest gathering data in both the public and private sectors on funding. There is no doubt we fund the same idea over and over again, the various sectors usually have no idea what the other one is doing, let alone any results.
In a time of economic challenges this is a big problem.
Underneath this is building relationships between the sectors, sharing initiatives, using entirely new strategies and processes which place the end user firmly in the center.
I am encouraged by the work and thinking you are doing. Thanks.

Todd S. said...

This is right on target, Lucy. I have said before that when head and heart find each other, the best of human potential is revealed. Giving from emotion is not contradictory to giving thoughtfully. I do want to emphasize that this way of thinking is just as important on the service delivery side as it is on the philanthropy side. I'd like to see nonprofit executives, managers and workers measuring -- or at least thinking about -- results, impact and outcomes. Not to take the heart out (as some measurement schemes invariably do but as the subtle guide you allude to. As an adviser to philanthropists and the leader of an organization that manages nonprofit performance, I appreciate your insigts. Todd J. Sukol,

Bradford Smith said...

Nice post Lucy with lots of good metaphors. I was struck, though, about how the metaphors take us down different paths. If data is "organic matter" it reproduces itself. If it is "fuel" a significant amount of effort is required to turn it into something than can produce energy. Some data is easily made open and free and the cost for doing do, subsidized by governments or advertisers. For the data that is more like fuel (data on giving from private sources)who pays(or should pay)for the the extraction and refinement?

Lucy Bernholz said...

Thanks Andrea and Todd for your comments. Yes - the power of data is that it can come from many sectors and sources. My hope is that the data will help us work cross-sector

Lucy Bernholz said...


Metaphor madness. Yes, they do lead us in odd directions. While some fuels certainly need a lot of processing and refining, and centralized processors to do that work, others don't. Think of micro-wind turbines, solar powered houses, and even Edison's original model for the power plant (every company would have had their own - the consolidation came later).

I agree with the content behind your comment - raw data are often really dirty data. Data often need lots of work - expensive work - to be made useful. This will continue to be the case - however, it's possible (isn't it) that the work of preparing the data will be decentralized over time? No simple or short term time frame for that, but if we think about the input interfaces for the world's biggest data companies (FB, Google, LinkedIn, etc) - they've standardized the front end so we the users can be blind to the back end.

Another read on your comment - the data cleaners (such as FC) become ever more important - they do more than gather and clean, they interpret, analyze, sort, filter, view, etc. All things FC is already doing.

Many thanks for writing in


Jeff Stanger said...

Great piece Lucy. I need your help in clarifying my thinking about data... you know where I come from -- I think data, broadly defined, play another role and that's in our public dialogue on the issues that foundations work on and care about. It's different from the flavor of "operational" data that is the primary focus of your post. What should we call this other class of data/information? The studies, the papers, the journal articles -- all intended to be public as a way to inform our public conversation, but largely not optimized for modern digital consumption. How is it distinguished from the "operational" data you are primarily talking about here?

Victoria Vrana said...

Hi Jeff - just a quick thought - if we're thinking of thes same kind of studies, papers, etc., to me those are "knowledge" not data/information. It's the analyses of the information that begins to make it actionable. It's an essential part of the ecosystem as well. Lucy, what do you think?

Lucy Bernholz said...

Thanks for the question, Jeff, and the reply, Victoria. I'm thinking hard about all this right now and not sure I have an answer. I do think that the strategy papers/evaluation/ commissioned research of foundations is a critical piece of the ecosystem. They are of a fundamentally different nature than other data. They are a certain kind of knowledge - internally generated off the very data that I'd like to see made more public. If the other data were more public we'd shift some of the balance of how this secondary knowledge were generated. It's cyclical (food chain metaphor?) but, given dynamics of foundations the sticky question for me "is what's the basic food?" - putting out the secondary knowledge might precede putting out raw data...

Would LOVE to work collectively on this. Wiki anyone?


Victoria Vrana said...

One last (for the moment) thought - does the sequencing come from a culture that uses data/information punitively and does not have a tolerance for failure and/or support for continuous improvement?

Lucy Bernholz said...


This is one of the critical factors here - the cultural uses/expectations of data access, use, and purpose. I think we can postulate all kinds of rational theories of how opening certain kinds of info would improve all kinds of things but data and systems aren't as rational as might like (reading @benkler's Penguin and Leviathan - brilliant). We need to think about data access and use within the real "culture" in which they get used even as we're hoping, thinking, beleiveing that the release of those data will change those cultures.

Architecture of collaboration - @benkler's term - is key here.

All of this is part of what #ReCodeGood hopes to inform.


Victoria said...

Not sure my other comment came through - just that I was struck by the sequencing issue -- that in our field, the secondary knowledge (from foundations, consultants, nonprofits and intermediaries) often proceeds the raw data (much of which doesn't exist of is released ever). Drawbacks of this of course are that the analyzed data/information is inherently biased by the point of view of the analyst.

Is this only a challenge for our sector?

Your point about changing the culture while we are working out the more technical data issues is hugely important.

Lucy Bernholz said...


I was thinking the same thing - we share "finished" knowledge products more easily than raw information/data . I've always credited this (tongue in cheek) to creation of pdfs.

I think it speaks a lot to the culture issue. Analyzed, finished, published less scary than raw, iterative, generative, open. Not unique to philanthropy surely - how do we use it as an opportunity, not a threat?


Jeff Stanger said...

I'll try to formalize this, but until then, a napkin sketch:

Victoria Vrana said...


I love the napkin and wish I could pick up a digital ink pen and interact with it. Two thoughts:

-- On Purposes, do you include "continuous improvement" to begin to change that culture to the POV that the underlying purpose is in getting better and creating more social good?

-- I would suggest a layer between data and interfaces about protocols and standards that ensures when the data is accessed through the interfaces or platforms, there is a commonality that allows for analysis across programs/orgs/grants/funders? Jeff,

I love the napkin and wish I could pick up a digital ink pen and interact with it. Two thoughts:

-- On Purposes, do you include "continuous improvement" to begin to change that culture to the POV that the underlying purpose is in getting better and creating more social good?

-- Is there a layer between data and interfaces about protocols and standards that ensures when the data is accessed through the interfaces or platforms, there is a commonality that allows for analysis across programs/orgs/grants/funders? That takes the New column to a whole different level.