Sorry Folks, So Far Big Data Has Crashed and Burned
The sound you heard was Big Data hitting the ground
It was almost a decade ago when technology leaders were gushing about how Big Data would dramatically change the world.
Let’s look back at a few of those bold claims:
“The use of big data will underpin new waves of productivity growth and consumer surplus.”
“The technology will identify business opportunities all by itself.”
“Harvesting more data will automatically generate more value.”
Big Data’s epicenter
It all started in May 2011 with a research report entitled ‘Big data: The next frontier for innovation, competition, and productivity’ created by a management consulting firm named McKinsey. They assessed that Big Data would be the keystone to all future competition, innovation, productivity, and consumer activity.
Their reasoning is based on the fact that our world is literally being swamping with tidbits of information from every corner imaginable. And it only stands to reason that the front runners in all business sectors will have to grapple with the data about their market. Actually, these assumptions are pretty solid in retrospect — you can’t argue with their assessment.
But as with most things in our society, assessments are generally pretty accurate. It’s the planning and, more importantly, the solution that falls way short. And Big Data is Exhibit One of the corporate tendency to fail when it comes to finding actual solutions to real problems — despite having planning documents that reach the sky.
Evaluating the real progress of Big Data
The year 2021 will be the official tenth anniversary of Big Data and its big promise. Check out a few statistics from the NewVantage Partners 2020 Big Data and Executive Survey in regards to the industry’s data initiatives:
- Just 26.8% of companies have established a data-culture
- Just 37.8% of companies report that they were now data-driven
- Just 45.1% of companies claim to be competing based on data analytics.
- Companies spend $50 billion on artificial intelligence annually with little interaction with big data.
Cold hard facts about Big Data
The hardest pill to swallow for most organizations eager to embrace Big Data is that it doesn’t happen overnight. In addition to the installation and validation of the massive hardware and software required, it pales in comparison to the cultural changes that are necessary for success.
There are going to be years of missteps and failures. Too many businesses today have put such a premium on successes that no one is willing to fail — they see it as a death knell to their career. These are real and legitimate concerns for personnel involved in these data projects.
There must be a tolerance for those who are artful and intuitive in the corporate world of procedures and instructions because assessing data can be very much an art form — at least in the early going. The fact is that some people will be more skilled at analyzing data than others.
Yes, to operate as a data-driven business would be the utopia of the corporate world, but not all data is useful — in fact, sometimes it misleads you.
Common sense dictates that all these factors be included in the future of Big Data. Try leaving all the big promises by the wayside.
[1]: Joab Jackson. (March 3, 2012). The Big Promise of Big Data. https://www.cio.com/article/2398317/the-big-promise-of-big-data.html.
[2]: Eric Almquist, John Senior and Tom Springer. (April 8, 2015). Three Promises and Perils of Big Data. https://www.bain.com/insights/three-promises-and-perils-of-big-data/.
