Don’t Jump From the BI-on-Hadoop Ship Just Yet - DZone Big Data

After just five short years in the spotlight, tech experts are calling for the death of Hadoop[1]. For many, it seems like Hadoop is already going the way of legacy applications — it’s not going anytime soon, but many business leaders are seeing it as more of a necessary evil than the miracle it once was.

The frustration is justified. The Hadoop distributed computing platform has been oversold as a solution for your entire business. But in reality, Hadoop is designed for data scientists who can code in MapReduce — not business analysts who need interactive business intelligence functionality.

Sometimes, though, the grass isn’t greener on the other side. If you’re thinking of abandoning Hadoop for BI, you may want to reconsider.

It’s True: Hadoop Isn’t Meant for Business Intelligence

Despite what you may have thought when you dove headfirst into Hadoop, it isn’t built for visualization software that promises near real-time business intelligence.

Consider three main points that summarize why Hadoop isn’t made for business intelligence[2] (and is “failing” you as a result):

  1. Data lakes aren’t meant for interactive queries. The lack of guaranteed response times makes latency too much of a challenge in these situations.
  2. Hadoop is best-suited for ETL batch workloads and machine learning because it offers a cheap storage repository.
  3. Data scientists can master Hadoop while critics say business users would have to learn Hive, Pig, or Spark to actually make it work — which obviously isn’t going to happen anytime soon.

These “failures” of Hadoop lead to many workaround solutions to make BI work on the distributed computing platform, which is where even greater frustration often sets in.

Workaround Challenges for BI on Hadoop

Latency issues in Hadoop point to a larger problem: Big data might just be too big for business intelligence in Hadoop. Business users need insights in near real-time and Hadoop and BI tools won’t integrate seamlessly to make this a reality.

Your instinct might be to abandon the Hadoop ship and find a new distributed computing platform to meet the needs of both data scientists and business users — but finding an actual solution won’t work.

DataTorrent Co-Founder Phu Hoang summed up the problem best[3]:

“Hadoop is painful. But [business leaders] don’t see another solution. Until there may be other distributed computing platforms out there, our focus will be on making that one as easy to use as possible.”

This means finding workarounds to make BI work in Hadoop[4]. But if you’ve ever tried the following workarounds, you know they aren’t without their own challenges:

Implement a Generic, Out-of-the-Box BI Solution

Trying to force a standard BI solution into Hadoop won’t work. These solutions are often slow when connected to Hadoop, might not fit into your specific use case, or might fail flex to the querying needs of your users. However, this additional software layer is the only way to create a BI dashboard for low latency insights.

Make Big Data Smaller With Extracts

BI tools like Tableau[5], Qlik[6], and MicroStrategy[7] are excellent for visualizing ingested data from Excel spreadsheets. But when you start working with three billion rows of big data, you’ll just freeze/crash typical solutions. Some companies extract smaller sets of data to work with in standard BI solutions[8]. However, this negates the benefits of granular big data analytics when you’re working with massive datasets. You end up with data silos across the organization and increasingly frustrated users.

Adopting Any of the SQL-on-Hadoop Solutions

When you choose a SQL-on-Hadoop solution like Hive[9], it doesn’t work the way you expect. While they are a step up from Hadoop’s lack of inherent SQL support, these solutions aren’t enough to meet your high-performance needs. On their own, data warehousing and massively parallel processing (MPP) solutions for SQL-on-Hadoop will only fuel the perceived downward trend of Hadoop for BI.

Hadoop Isn’t Going Anywhere, but BI on Hadoop Must Get Easier

Will Hadoop always be the primary (or only) distributed computing platform? Probably not. But regardless, so many companies have billions of rows of data in Hadoop and migrating it all won’t happen in the short term.

Instead, we have to start thinking about ways to satisfy business users as well as data scientists within Hadoop-powered organizations. The best way to do this — and to end the calls for Hadoop’s death — is to take advantage of a query acceleration engine. You can enjoy the cost efficiency of Hadoop without sacrificing high-performance business intelligence.

Comments

Post a Comment

Popular posts from this blog

New Article Posted: Seven Things You Should Do in Your AMS

Given how advanced and thorough the modern association management software (AMS) system has become, there are many opportunities for your organization to leverage it. Here are seven ways I believe every association should be using their AMS. Click here to read the rest of the article . Of course, if you’re already on my  announcements list , you would have received notice of this new article right in your email box. Not signed up yet?  Click here to sign up . Go to the article
Read more

10 Common Mistakes Biblical Counselors Sometimes Make, Part 8

Image
A Word from Bob: You’re reading Part 8 of a 10-part blog series on 10 Common Mistakes Biblical Counselors Sometimes Make . Mistake #1: We Elevate Data Collection Above Soul Connection . Mistake #2: We Share God’s Eternal Story Before We Listen Well and Wisely to Our Friend’s Earthly Story . Mistake #3: We Talk at Counselees Rather Than Exploring Scriptures with Counselees . Mistake #4: We Practice Half-Biblical Counseling If We Address Sin but Neglect Suffering . Mistake #5: We Fail to Follow the Trinity’s Model of Comforting Care . Mistake #6: We Tend to View People One-Dimensionally . Mistake #7: We Devalue Emotions Rather Than Seeing Emotions as God’s Idea . Mistake #8: We Minimize the Complexity of the Soul-Body Interconnection Because we are soul physicians and soul care-givers, our tendency is to focus on matters of the soul. Most biblical counselors are not trained in the medical field, so we wisely avoid offering medical guidance. Yet, that appropr...
Read more