What are the Biggest Challenges to Getting Returns from big data?

Big Data Study - What are the Biggest Challenges to Getting Returns from big data?

The media has written extensively about the challenges companies face in collecting, processing, analyzing and using Big Data in their businesses. Much public discussion has focused on the three ‘V’s’ – handling the volume, variety and velocity of the data. Another frequently mentioned challenge is finding people who know how to analyze the data – data scientist has become a hot profession. Driving business decision-makers to actually use the data and abandon making decisions on intuition is also a challenging task.

But which of these challenges are the greatest? To gain greater insights on this, we asked respondents to rate 16 challenges that we found mentioned most frequently in press articles, public speaking presentations, and our client work. We provided a scale of 1 to 5 (1 = not at all a challenge, 2 = minor challenge, 3 = moderate challenge, 4 = high challenge, and 5= very high challenge). The results can be seen in Exhibit II-17..

Exhibit II-17: Key Challenges of Big Data Across Regions of World
Q23: Mean Rating of 16 Challenges in Getting Returns from Big Data (Scale of 1-5)

Exhibit II-17: Key Challenges of Big Data Across Regions of World

We have several observations regarding our findings on this question, combined from the four regions around the world:

  • None of the 16 challenges stands far above the others. All but one of the 16 challenges received mean ratings between 3.0 and 3.4. In other words, across all 643 companies, they were slightly more than ‘moderate’ challenges. That surprised us; we expected to see a few rated higher than 4.0.
  • The highest-rated challenge is not a technological issue. It is an organizational or cultural issue: getting business units in a company to share information across the organizational silos (divisions, business functions, etc.). This issue has plagued companies for decades, long before the phrase ’Big Data’ was coined. Business functions become protective of their data and often don’t have any incentive to share it internally. However, many business decisions can be dramatically improved when decision makers have access to the bigger picture: what’s happening with certain customers, problematic products, and service issues. For example, Xerox Corp. sees a ’big play’ with Big Data when it can combine data on how its 1.2 million machines are operating in the field with claims information. The devices themselves report data on their operating condition back to Xerox every day. But the company wants to aggregate this data with claims and other data kept by a company division that provides business process outsourcing services.1
  • Finishing a close second was a technological issue: dealing with the three V’s of Big Data.
  • Companies see getting managers to make decisions based on Big Data rather than on intuition – relative to the other 15 challenges – to be a smaller challenge. It ranked 12th on the list, just higher than figuring out which technologies to use.

We explore some of the top-rated challenges in getting returns from big data:

Getting Business Units to Share Information Across Organizational Silos

Getting business units and functions to share data is a huge cultural challenge at most companies, where functional or divisional measures and performance incentives trump organization-wide metrics and rewards. In our consulting work, this issue is vastly underappreciated amidst all the hype about Big Data. It has prevented many companies from tapping the greater potential of Big Data.

A senior executive at a large company who spoke to our research team said employees in his company are very protective of data because of privacy and security issues. (Customer data privacy is a major concern in this company’s industry.) He said most data sources in the company are locked up. “The easiest part of Big Data is the technology part. The hard part is practice standards, legal ramifications, regulatory issues, all these kind of others things.”

Handling the Three V’s 

The three V’s of Big Data – volume, variety and velocity continue to be a key challenge. The telecommunications company we spoke with said handling the velocity of the data they get is of paramount importance, especially in the firm’s wireless communications unit. Customer turnover is higher there than in other businesses, and consumers get new phones every couple of years. “I think speed is probably the important thing for us because if we don’t quickly get that information, it’s not that useful to us,” he said.

Determining What Data to Use: Building Trust Between Data Scientists and Functional Managers

In our interviews, we often heard that if functional managers don’t trust the data scientists who bring fact-based recommendations for improvements, those managers are not likely to accept the advice. This is especially the case if those managers are insecure about their current strategy.

An executive who runs marketing analytics at one large insurance company said it is important to get functional managers involved upfront in an initiative. “One of the key challenges is people who question the data. They don’t like the results, so they’ll find problems with the data. … You get people to believe it’s good data by getting them in the design of the project at the beginning. We work with folks and say, “Here’s what we want to do and measure, and here’s why we think it’s going to work.” If we get them involved upfront, they’re less likely to have a problem with it.” The executive said that this approach has resulted in building trust across business functions. “When I first came on board several years ago, you had to beg to get a seat at the table. We had to push our ideas toward them. Now it’s shifted to more of a pull. They know what we can do with data, and now they are pulling information from us – contacting us rather than us contacting them.”

An analytics manager at one large company said one of the keys to building strong relationships with functional managers is having analysts with ’storytelling skills’. They must be able to translate analytics into something that busy executives understand and care about. If they can’t, the likelihood of having executives do things differently is small. “If you have generated lots of data, the first key to success is being able to be persuasive with it,” this manager said. The company provides training to its analysts on how to make data more visual and give succinct answers to pointed questions.

We heard many times that, at the end of the day, executives must be willing to listen to the often-contrary conclusions of what the Big Data analysts are telling them. This was nicely summarized by Bank of America’s head of technology and operations, Catherine Bessant, in a conversation with The Wall Street Journal: Big Data pays off when “the manufacturing of brilliant data and making brilliant use of it [is accompanied by] a drive for abject purity in listening.”2 .

Hiring Data Scientists

Data scientists who are experts in quantitative skills and highly effective at communicating their findings in language that functional managers understand are very rare. This does not bode well for companies, given that the need for professionals who can process and analyze Big Data is large and growing quickly.

Ford Motor’s analytics chief John Ginder indicated to a reporter that the $134 billion automaker doesn’t have nearly enough highly skilled people who know how to use Big Data tools for managing huge sets of data (like Hadoop) and analysis. “We have our own specialists who are working with the tools and developing some of their own in some cases, and applying them to specific problems,” he told ZD Net.3 “But there is this future state where we’d like to be where all that data would be exposed – where data specialists (not computer scientists) could go in and interrogate it and look for correlations that they might have not been able to look at before. That’s a beautiful future state, but we’re not there yet.”

A 2011 McKinsey Global Institute study predicted that by 2018, the demand for analytics and Big Data people in the U.S. alone will exceed supply by as much as 190,000.4 Many companies are already seeing a shortage. A July 2012 survey of 108 business technology professionals by the trade publication InformationWeek found only 17% believe they’ll ‘easily’ fill Big Data jobs. More than half (53%) said the skill set may be hard to find. And 23% said the salary demands of many data scientists may be more than they can afford.

A senior executive at a major insurance company said he sees many more resumes than real candidates for the analytics jobs in his company. “In addition to the job skills, our people have to be able to communicate and build relationships with [functional and business unit] managers. It can be very, very hard to get all of those things in one person.”

He recruits Big Data analysts largely through personal connections – friends of friends. “The good news is that there are tons of people out there looking for jobs. The bad part is that there are tons of people out there who I would never think of hiring,” he told us. “That cuts down the population big time. Applicants have to have credibility to tell me what we need to do. Then they need the ability to do it and communicate and build relationships.”

Optimally Organizing Big Data Activities

Should a company’s Big Data professionals operate outside the business functions so they might offer more unbiased information? But won’t that mean they don’t
understand the business functions they are asked to help as deeply as they should? And if they operate within a business function, how do they avoid finding data and making recommendations that help confirm the functional head’s long-held beliefs and strategy?

And if analytics is centralized, should it be part of a central IT group so that it can work more closely with the technologists who maintain many of the firm’s core data sources? Or should analytics be centralized but report somewhere else?

Questions like these were on the minds of the executives who spoke to our research team. A telecommunications company executive said his firm’s three business units had their own analytics groups focused on their own customer segments. But within each business unit, functions (such as marketing) also had their own analytics professionals. Another large company’s analytics function (with more than 75 employees) resides within IT. Nonetheless, other analysts are embedded within business units across the organization.

A health care company has pushed down its Big Data activities into business units and functions. In recent years, the IT organization launched an internal advisory group that collects best practices and tries to standardize usage and licensing of analytics tools. However, this has been difficult to do because of the company’s decentralized approach to Big Data.

These and the other eight challenges vary, sometimes significantly, by industry and by business function, as we’ll explain in the corresponding sections of the report. Nonetheless, combined across all regions, industries and business functions, none of the challenges stands head and shoulders above the others.

Big Data Study Findings: Regional
Read more topics in this section:

Home | Download Report | Big Data Services | Contact a Consultant Today


  1. Xerox vice president and global chief information officer Carol Zierhoffer mentioned this to The Wall Street Journal recently. []
  2. From an article by Michael Hickins, Banks Using Big Data to Discover ‘New Silks Roads’, The Wall Street Journal, Feb. 6, 2013.  Note: Bessant has also said she dislikes the term “Big Data.” “It implies something monumental out of something that should be fundamental, and should be basic, which is the creation of accurate, timely data on a reliable basis,” she explained to CIO magazine in January 2013 []
  3. Read Zdnet article  []
  4. McKinsey report []

If you like this story, please share it

Leave a Reply

Your email address will not be published. Required fields are marked *