The Big Gap in the Big Data Ecosystem

According to the 2011 IDC Digital Universe Study (sponsored by EMC,) in 2011 we will create and replicate 1.8 ZettaBytes (1.8 Trillion GigaBytes) of information. More importantly, we are growing fast (it has grown by a factor of 9 in just five years). So, yes, the Big Data revolution is here. But, I see that we have a big gap in the big data ecosystem. While the data explosion is happening at a breathtaking pace, the technologies to make meaning out of this data are still not there yet.  Unless you cannot get the data to the people who can make use of it in a beneficial/profitable way, a lot of the value is simply locked up. If you look at the ecosystem of data for a moment, then the reality is:

  • The new age data producers (the online services) don't have effective ways to syndicate and share that data and at best use it for internal purposes only.
  • The incumbent data producers & collectors (the market researchers and the government) are stuck in old fashioned methods, providing access to their data in ways that make it hard for people to reuse.
  • The journalists and bloggers are still writing most of their articles in text form, knowing but not using the ancient adage that "a picture is a thousand words"
  • The consumers (aka “the rest of us”) are still in despair searching for data that we can easily reuse for our purposes

The paradox: Everyone acknowledges that Data is exploding, but no one seems to have a handle on finding relevant data and making meaning from it. I agree that the ecosystem of data is still in a very infant stage. The big gap I referred to above is a data publishing and distribution platform that makes it simple to take the data from the source to where it can be utilized most effectively (think of the oil industry). The two core ingredients are: the raw data that needs to be distilled into something newsworthy and the chart that carries that news to whomever it may concern. While very much like the chicken and the egg, these two, however, are very different worlds, the data scientist world and the data visualization world. The tools and skills don't necessarily align easily. Much like having the artist and the nerd share a dorm. At iCharts, we believe that you need to solve both in tandem and in an integrated way. Our approach if you think about it is simple - make the chart as the container and make the data as it’s ingredient. The chart is a fungible, rich object.  We process the data in a way that it can be quickly injected into a chart. The chart is also designed to be social, so it can travel where it needs to be - in front or behind pay and firewalls. It can be syndicated or straight from the source. I am not saying that we have built something that will entirely fill the “big gap” completely. But, we surely have begun the journey towards making something meaningful in that direction. This excitement is what I am looking forward to share at the Strata Conference. If you are going to be there, I look forward to meeting you. Here is the link: Strata Launch Pad (Sep 20-21, 2011, NYC)