Big Data Skills – The best way to organize Big Data activities in organizations

Big Data SkillsThe history of how companies have adopted information technology shows time and again how they had to first centralize a new technology and the skills to master it. Big Data skills – particularly data-driven analytics abilities found in the best data scientists – are in acute shortage today.

Over time, however, as a new technology becomes easier to use and people learn how to use it, the technology and the people who use it are shifted from the corporate IT function to business functions.

Yet Big Data is fundamentally different than other technologies. Companies today need a single view of each customer – not multiple versions (one held by sales, a slightly contradictory one held by marketing, a third one by customer service, and a fourth by the accounting department). It won’t be easy to break down the organizational silos that prevent a company from using all its information about customers (or suppliers, for that matter).

We see two separate but related aspects to the ’organizational silos’ issue:

  • How to get divisions and business functions to share data that they often fiercely protect. This is an issue that likely requires a mandate from the top of a company, as well as a CTO, CIO or chief analytics officer with the political skills to carry it out.
  • How to knit the data together technologically since it will be in different kinds of database management systems and be beset by other technical disparities that can make it a nightmare to integrate. There has been a shortage of technologies that can extract and merge big volumes, varieties and velocities of Big Data from different sources. However, this is changing. For example, Sears has broken down its organizational data silos in an Analytics Center of Excellence and is using the Hadoop open source data process platform to house huge volumes of data. It plans to sell analytics services to other companies1.

While a company must be able to tap and knit together silo-ed data to get a more comprehensive picture of its business problems, it still may not know the answer to the question of how to organize the people who analyze that information. Should they reside in the business functions that need the insights? Should the analysts operate in the IT function, where much of the data often exists? Should they live in a separate analytics department – a “center of excellence” of the type that GE, a large Internet company, and other firms have launched?

The companies that have created large and well-funded analytics centers of excellence gave us of several reasons for doing so. One was preserving the data scientists’ independence — their ability to provide unbiased advice to functional managers about how to run their businesses. The manager of 70 analysts at a large Internet company told us that the key to helping the company increase revenue by hundreds of millions of dollars through numerous (and ongoing) tweaks of its website was extracting the analysts from the company’s product units and centralizing them. “There was a heavy bias back then for analytics to confirm what the product units were doing,” he said. Taking Big Data analysts out of the product units has led to better insights – ones that product unit managers might not have wanted to hear from their underlings.

Centralizing a company’s Big Data analysts has another big benefit: giving them an attractive home within a big company. That was one reason why General Electric set up its $1 billion analytics center of excellence. The analytics manager from the Internet company said the same thing: having a center of excellence provides a more attractive career path. Even more important, he said, was to get analysts sharing methods far more freely than they did in the past. “Centralization got us to talk together,” he said. “That was the special sauce. It didn’t happen overnight. But within a year, we really had improved our analytics skills.”

Our survey findings of leaders and laggards points to the need for centralizing at least some of the analytics staff. Some 37% of leaders put their analysts in a separate Big Data group compared with 23% of laggards. And 37% of leaders put people who process the data in a central Big Data group vs. 19% of laggards. (See Exhibit VII-11.)

Exhibit VII-11: How Leaders and Laggards Organize Their Big Data Staff

Exhibit VII-11: How Leaders and Laggards Organize Their Big Data Staff

From our interviews with executives, we find that the optimal organizational structure must accomplish the following:

  • It enables data scientists/analysts across a company to collaborate closely, share and improve one another’s methods and technologies
  • It provides them with a clear internal career path (thus, not forcing them to move into a business function to move up in the company)
  • It allows them to provide objective insights on decisions that business unit and functional managers need to make – insights that are not watered down or censored by the business before they are delivered
  • It helps analysts learn enough about the business operations that increasingly need data-driven insights without becoming trapped by traditional thinking
  • It helps data scientists/analysts build trusting relationships with business unit and functional managers to increase the chances that they act on Big Data

We believe the optimal structure for a company’s Big Data analysts is to create a center of excellence — and have other analysts operating within divisions and business functions. The task of the central analytics group would be to build the deepest, most rigorous data science capabilities – skills that enable the company to get to the truth of what’s really happening in its business and what it should do about it. The analysts in the business functions may not need such deep quantitative skills. But they need to be experienced enough to know how to put in place the recommendations that the central group makes for their functions.

Our research suggests that the optimal way to accomplish this is to create a center of excellence for a good portion of a company’s Big Data analytics. The center must report to or near the top of the company (to be objective and not be influenced by a particular business unit or function). However, to help data scientists better understand the business functions that need their insights, regular interactions with functional managers and tours of duty in those functions or units will do much to strengthen relationships — even if the reporting lines are to the analytics center.

Big Data Study Implications & Recommendations
Read more topics in this section:

Home | Download Report | Big Data Services | Contact a Consultant Today

  1. From an April 2012 article in the Chicago Tribune []

If you like this story, please share it

Leave a Reply

Your email address will not be published. Required fields are marked *