We’ve all heard the success stories. Netflix’s recommendation engine, powered by big data, has saved the company an estimated $1 billion in customer retention since it went live in 2009. Even lumbering UPS has adopted data analytics to enhance its logistics network, aiming to save up to $300 million annually. Research by PwC has found over half (54%) of business executives saw productivity improvements from adopting AI solutions.

Yet despite all the chatter to the contrary, data initiatives don’t always work out. More than a few enterprises devote massive resources to gathering and storing data, implementing analytics, and building vast data lakes, but not a lot changes. By some estimates, 85% of big data projects fail. Organizations find themselves data rich yet insight poor.

Why is this the case? Partly because the data craze has created myths that need to be toppled. Google search data shows interest in the term "big data" has eased slightly while interest in "data scientists" and "artificial intelligence" is on the rise. This suggests businesses are looking for ways to make sense of the data hype. There’s no doubt most organizations need a data strategy, but now more than ever that strategy should be based on principles grounded firmly in reality. We asked leading data experts, inside and outside of ThoughtWorks, what those principles should be. We used their insights to develop questions to help organizations evaluate whether they’re on the right track or still in the land of data fantasy.



The best information results in insights, teaching the recipient new things and serving as a basis for action. But as businesses typically produce and store avalanches of data as a result of the most mundane day-to-day operations, finding/generating the information requires knowing where to look, and what data to prioritize, gather, and process. That can’t be done without a good grasp of the questions the business needs to answer, or the problem you’re trying to solve. 

The need to link data projects to concrete business goals is a clear argument for the involvement of senior business leaders at the earliest stages—and means such projects should never be strictly "technological" ventures.

Principle 2: More isn’t always better (or, startups could teach you a thing or two)

If bigger data really is better, many established companies are in pole position as they typically have more historical data at their disposal than a startup. It’s true that bigger firms excel in terms of volume but this often comes at the expense of clarity. As Rachael Hadaway, SVP of Product at Kroger-owned 84.51°, pointed out at ThoughtWorks’s executive event ParadigmShift, "Having lots of data is not an asset; it’s a liability. It’s cumbersome and expensive. The asset is knowing what to do with the data."

Parsons said, "Most enterprises have got much more data at their disposal just because of how long they’ve been around. But that data is locked up in systems that are not necessarily well integrated, particularly given all of the M&A activity. You’ve got data coming from different organizational homes that has to be reconciled, and so the scope of the problems you have to deal with, from an implementation perspective, is completely different."

Similarly, established enterprises may have people with the potential to push new approaches to data but Mark Brand, Independent Consultant, believes they tend to be overlooked. "[They are] lost in a sea of other general roles and people who don’t draw a line around data-driven decision making," he said. "And therefore, they can never break out and do it."

A hungry young startup, by contrast, can embed data in the culture, and adopt a much nimbler architecture, straight out of the gates. At such companies, virtually everyone is "expressing things in terms of metrics and spreadsheets, writing their own algorithms or rules and asking the tech guys to deploy them," Brand said. "Even the people handling functions like branding are data-driven. They’re always collecting data, and they all talk in numbers towards enterprise outcomes. That’s what it looks like to be a data-driven business—not necessarily using AI or having a cutting-edge product."

"What successful data-driven startups really have in common is a data-driven mindset throughout the whole organization," agrees Windheuser. "When they design a process, it’s data-driven by definition. Data governance is part of the culture and not something the company has to work on or define in a cumbersome process. All the IT systems and architecture are built with the free flow of data as a major requirement. In traditional big companies, that’s not the case. They have to change their culture as well as their IT systems."



“It's common for people to say 'I want a data lake,' or a 'modern data architecture,' but they have very little idea what for,” Brand said. “It’s common because those people often have two characteristics—they own data and data architecture, but they don't own any decisions or any outcomes. The most common result is people attempting to build data lakes like you build a data warehouse. At first they try to put everything in there and tag it, and then they give up because that's a bit tough. So they try to put everything in there from one of their monolithic critical systems and capture all states; so if the quality of a data point changes every 60 seconds, they try to somehow capture and tag that, before they've even worked out why they need to hold that information."  

Similarly, while many enterprises are keen to adopt machine learning, Ken Collier, Author of Agile Analytics, notes "not every problem needs a machine learning solution. Some are best suited to just writing some standard software or some other code."

"AI by itself is a huge subfield in computer science that has a whole lot of different disciplines," Collier said. "I get bothered when I hear business leaders using machine learning as some sort of buzzword, when what they really mean is 'We want to do things in more leading-edge ways.'"



The immediate risks of businesses applying data-driven solutions for their own sake are clear enough, including non-performing projects and wasted resources, but Collier believes the impact could be even broader: "What I fear is that we're headed toward the next AI winter. After all of this hype wears off, and only some things have stuck or have proven value, all of a sudden AI is going to get a bad rap for not having lived up to some unrealistic expectations."

When faced with failure, the tendency is to blame the tool at hand. "Most data projects do not fail because of technology, but because of human factors," Windheuser said. "There may be a new [method of] organization necessary, or resistance to change, or no data processes defined." Which brings us to our next principle.



… and don’t ever stop

Rather than algorithms or cutting-edge analytics platforms, it is this discipline of constantly building and testing, as well as collecting, analyzing and acting on results that characterizes the data-driven enterprise.



By incorporating insights from one iteration to the next, products or solutions are in a state of perpetual improvement. In other words, continuous intelligence includes continuous delivery principles to minimize the data-information-action cycle. Agile has taken on all kinds of meanings in recent years but is best defined as the ability to complete this cycle quickly and seamlessly. This is an area where many enterprises still struggle.

"This circle of getting data, extracting information, getting insight and knowledge, then transforming that into decisions and actions in a lot of organizations takes months or years," Brand said. "That’s not agile. In many cases this is down to an isolated data science team working in an isolated environment on the data and the data models. They’re starting from scratch to develop a scalable and productive user application which is given to operations to integrate and run in the IT environment. Continuous delivery allows you to reduce the whole cycle to days, so it’s more or less automated."

Data’s role in that process (and in agility) should be recognized and celebrated. Data and the fast-developing technologies that harness it are closer to building blocks than a commodity worth hoarding.

By Daniel Pallozzi