Last week the Gartner Group released the results of a survey that suggests adoption of Hadoop is not a priority for most enterprises, and the blog-o-sphere erupted with articles decrying Hadoop’s acceptance and adoption by businesses. Many people reading these intentionally provocative headlines might think Hadoop is an unsuccessful experiment in changing the way business does analytics. Here are three reasons Hadoop adoption will continue despite what Gartner says in its latest survey results.
Gartner is famous for its Hype Cycle, a graphical representation of the maturity and acceptance of various technologies. The Hype Cycle illustrates the effect of media on technology adoption and attempts to map it to the reality of implementing production solutions for early adopters.
Gartner believes Hadoop is in the so-called “Trough of Disillusionment” — a period when technology has gained considerable adoption by organizations before they are ready to seriously make use of it. They do this because the media hype leads them to believe they can’t stay competitive without it.
Not many years ago, Gartner was posting articles and survey results about Data Warehouse and Business Intelligence that sounded very similar to what I read in its release about Hadoop. Articles like Gartner Reveals Nine Fatal Flaws in Business Intelligence Implementations or Gartner Says More Than 50% of Data Warehouse Projects Will Have Limited Acceptance or Will Be Failures through 2007 were similarly disruptive in their time, and those of us in the data management space often spent significant time with our customers convincing them their Data Warehouse or Business Intelligence efforts really could be successful.
Fast forward a few years, and most organizations now cannot live without their DWBI solution. In fact, in many cases these solutions have gone beyond what was originally intended simply as a historical reporting system to become mission-critical for day-to-day business operations.
When I read through these reports from Gartner and think back to all of the similar conversations we’ve had about Data Warehouse and Business Intelligence I can’t help but think, “Why would you adopt a technology that you have no concrete plan for?”
Data analysis of one form or another is core to any business. Understanding your data is essential to understanding your customer, your performance and your competition. You must have a strategy supported by techniques, technologies and tools. Only once you’ve defined your strategy is it appropriate to be choosing the techniques, technologies and tools you will use to implement it. If you fail to plan your strategy then it is likely your first attempt at building any solution will be unsuccessful.
The “Trough of Disillusionment” is a bit disingenuous because the disillusionment is driven by these early adopters who did not have a considerable strategy. By taking a step back and thinking through what you hope to accomplish and implementing an appropriate plan before you leap into a new paradigm you will be far more likely to succeed.
Three Reasons Big Data Adoption Will Continue
The trend of big data is unstoppable. Here are three reasons big data adoption will continue.
Businesses Want a 360° View of Their Customer
Traditional Business Intelligence and Data Warehouse solutions contain only information from operational systems such as sales, inventory and finance. The systems are great for giving you a backward looking picture of what happened in the business from the available perspectives. DWBI solutions have been very successful at showing organizations their performance to plan and year-over-year growth, but have not been able to identify what a customer thinks or wants.
Businesses now want to understand how their customers feel about their products, what they tell their friends about a product and what other products they might buy. None of these answers lie in operational systems.
Big data technologies give us the opportunity to consume and analyze information we have never had access to before.
The Technology Is Catching up at Lightning Speed
One of the typical objections to Hadoop for enterprise data analysis is responsiveness. Naysayers often point to the Map-Reduce model as an antiquated, batch oriented method of analyzing data not suitable for interactive analytics. The fact is the Hadoop community is moving far faster than we have ever seen. Relational databases were first proposed by E.F. Codd in 1969, and they have had a 40 odd year head start to become the de facto standard for storing and processing data. However, the relational model really hasn’t changed in all that time.
Hadoop was born out of work built on two research papers from Google on Map-Reduce and the Big Table storage mechanism in 2006. Nine years later and Hadoop is the core data processing platform for many very large organizations such as Yahoo!, Netflix and LinkedIn.
Adoption at scale has been made possible by the extremely rapid development done by the Hadoop community and organizations such as Hortonworks, Cloudera and Microsoft. Where in the past we have seen maybe one release every two years from the major relational database vendors, with Hadoop we are seeing multiple releases per year. This has led to advancements such as Hadoop integration with Apache Spark and the Stinger Initiative.
With the rapid development cycles in Hadoop community, Hadoop and Hadoop tools development will soon be on par with anything that we have had available with relational database management systems.
The Tools Aren’t As Hard As You Have Been Led to Believe
Another common objection to Hadoop is it is difficult to learn, and there simply aren’t enough skilled practitioners to do the work. While it is true in some respects, the technologies are still very much developer-focused, and in some cases such as writing Map-Reduce code, require programming background, with proper architecture and design, a Hadoop implementation can provide similar experiences for business users to what they have been used to with traditional DWBI solutions with the added advantage that advanced users and data scientists can perform explorations and analyses that have never been possible before.
How Not to Fail
If your business is asking to use new kinds of data that simply don’t fit into your existing data analysis paradigm, you may be considering big data. If you are, you should prepare by formulating a strategy that defines your short and long-term goals. Your strategy should consider data systems and sources both internal and external to your organization and the needs of all of your business users, not just one group of analysts. Don’t believe you have to throw everything away that you have already built. Consider that big data can work together with your existing data warehouse or analytics system.
Once you’ve defined your strategy, don’t think it is necessary to attack it all in one go. The most successful implementations will take a “think big and start small” approach. Your strategy should envision goals from one to three years out but your implementation should deliver business value in short meaningful increments.
If you’re interested in discussing a big data strategy, we can help! Get in touch to learn how we can help you get the most from your big data solutions.