Gartner Research's Hype Cycle diagram (Photo credit: Wikipedia)
Last week, a Gartner research director proclaimed that big data is entering the trough of disillusionment. How tiresome!
What the hype cycle describes is the bored tech news media. Big data was last year's celebrity, now it's the whipping boy for oversold Hadoop solutions, and next year it will fade into being an accepted part of tech news life.
It's important to understand the hype cycle for what it is. At best, the hype cycle is common sense dressed in a dark suit. To focus on the froth of tech news is a distraction from making appropriate choices for your business. You get value when you treat big data as a business problem, not an IT one. In a world where IT must be integrated with business leadership, the tech analyst industry sits strangely apart, taking a vendor and product-focused view of the world.
So where is big data? I've written elsewhere on the issues businesses will tackle in 2013, but in short the general sentiment is that it's early days. No surprise. We're taking baby steps.
The value in big data that has everyone excited is about smart use of data: using data for better resource management, improve your product or to create new businesses. For most of us, who don't have the luxury of a green field, that means the first thing to do is getting at that data in the first place.
This is a hard problem, involving tying together different systems and liberating data from silos. The technical side of the project can prove less difficult than the logistical and political side. Only when you've collected the data can you start to get value from it. And then it gets hard again, as you need to ask the right questions.
If you bought Hadoop because you thought some magic would happen to bring all your data together, then here's some advice: stop reading the news media and pay more attention to the goals of the business and flow of data in your organization.
Every time I speak with people who've had successful big data projects, the advice is always the same: start with understanding the problem you're trying to solve. The way you solve that problem won't just be about the tools, it will also be about business process, employee perceptions and incentives, and organizational change.
It's entertaining to watch the ebb and flow of the tech news media, but we shouldn't be led by it.
