Beyond the algorithm: Overcoming hidden barriers to data and AI success

Data and AI are dominating conversations. We constantly hear from clients aiming to harness these technologies, driven by fear of missing out and the promise of radical transformation. Like anything on the Gartner Hype Cycle or Thoughtworks Technology Radar, everyone wants to be doing it, but few understand how to be successful.

There are now many tools with vendors promising great results. Yet, beneath the surface lies a challenging reality as many organizations struggle to translate AI investment into tangible value.

Why are so many initiatives failing and what can be done to make them successful? My contention is that while tools are important, we first need to understand what they are trying to do and why. This directs funding towards genuine potential and builds solid foundations for adoption. 

Based on our work, we see recurring barriers derail promising efforts. Here are the most common, with practical suggestions on how to overcome them.

1. Strategy vacuum

The problem: Launching AI initiatives without a clear connection to business objectives is like setting sail without a destination. Often, the motivation is reactive — a desire to "do AI" because competitors are, or because it’s the current hot trend. This leads to scattered experiments, difficulty measuring return on investment and projects that, even if technically successful, fail to move the needle on core business priorities. Without strategic direction, teams lack focus, resources are diluted and executives remain unsure of what value has actually been delivered.

The solution: Develop an AI strategy that’s explicitly tailored to your organization's unique context and ambitions. Ensure this is sponsored by both technology and business leadership. Resist the urge to pursue AI for its own sake but identify where AI could genuinely differentiate your business. Identify potential use cases across the value chain, prioritizing based on potential business value and technical and organizational feasibility. This focus will ensure that investments are directed towards initiatives promising the greatest strategic impact and are achievable within your organization.

2. Defective data foundations

The problem: AI algorithms are fundamentally dependent on data. Issues like poor data quality, lack of representative datasets for training models, unclear data lineage and inconsistent data management practices hinder AI models from the start. This leads to inaccurate results, biased outcomes and a lack of trust in the outputs.

The solution: Before scaling AI ambitions, assess your data landscape. Is your data fit for the purpose you intend? Does it accurately represent the populations you're modeling? Can you trust its quality and consistency? Does your data platform have AI capabilities? Do you have people with the right skills and capabilities? 

If the answer to any of these is 'no', prioritize building these foundational data capabilities before starting to work with AI. This might involve data quality improvement, establishing clear data ownership, implementing robust data governance and ensuring data practices align with security, privacy and ethical standards. 

3. Portfolio blind spots

The problem: In many large organizations, initiatives emerge organically across different departments, often without central coordination or visibility. This leads to duplicated efforts, conflicting projects, wasted resources and an inability to make informed decisions about where to allocate capital and talent. Without a clear view of the entire portfolio of Data and AI work, organizations cannot effectively manage their investment or track value realization.

The solution: Establish a clear, prioritized, actively managed portfolio of all significant Data and AI initiatives. This should capture key information about: 

• Strategic objectives. 

• Expected outcomes.

• Required investment.

• Dependencies. 

It’s also important to implement a regular review cadence where leadership assesses the portfolio's alignment with the overall strategy. Here, they can monitor progress towards value creation and make more intentional and informed decisions. A well-managed portfolio brings discipline, transparency and ensures resources flow to the work that’s delivering the most value.

4. Organizational friction

The problem: Traditional organizational structures can create significant roadblocks for effective data management and AI implementation without clearly defined roles and responsibilities for owning and processing data.

The solution: Design organizational structures and roles that explicitly support effective data ownership and management. Promote clear accountability for data assets, ideally placing ownership of them inside the business domains that understand the data best. Balance this distributed ownership with central enablement functions that provide consistent tooling, support, standards and guardrails. It’s also important to define clear roles and responsibilities to minimize ambiguity and organizational friction and empower teams closest to the data while also ensuring consistency and responsible practices across the enterprise. Review what capabilities you’re missing in your teams and provide targeted AI literacy and enablement to support staff working on AI initiatives. Also consider adjusting KPIs to promote AI experimentation in a ‘safe-to-fail environment to support continuous learning.

5. ‘Wild west’ data practices

The problem: These arise when there are no established rules. Some typical red flags you might see include data getting copied and altered ad-hoc by different teams for their local needs, which breaks lineage and trust; overlooking critical considerations like bias, diversity and inclusion in AI models; inconsistent or even completely absent security, privacy and ethical guidelines for data collection, retention and usage. 

The solution: Address data practices by instituting lightweight but disciplined data governance with clear, practical policies and automated checks. Establish human-centered AI principles that guide development and deployment, and define consistent standards for data security and privacy.

6. Adoption headaches

The problem: Even well-designed AI solutions fail if employees don't use them. Adoption can be hampered by a lack of engagement, fear of change, internal politics, perceived lack of personal benefit or inadequate training.

The solution: Treat this as any other change management challenge. Engage with all users and stakeholders early and often. Clearly communicate the purpose and benefits, explaining "what's in it for them." Identify and empower internal champions. Invest in capability-building and training programs to demystify AI and equip employees with the skills needed to work with new tools and processes confidently.

Build robust foundations to generate lasting value with AI

Concentrating on acquiring the latest technology while neglecting strategy, data foundations, portfolio management, organizational design and disciplined practices is a common cause of the high failure rates of data and AI initiatives.

We’ve outlined critical pain points regularly observed across diverse organizations. Once these pain points have been identified, they can be addressed with deliberate effort. By shifting focus from purely technical implementation, businesses can dramatically improve their odds of success — they’ll be able to move beyond the hype and begin generating sustainable, meaningful value from their data and AI investments.