Patterns of enterprise intelligence

Introduction

This is the second in our Intelligent Enterprise Series of articles. In the first article we introduced Continuous Intelligence, the virtuous cycle of steps in transforming data to insight to action and of continuously improving this process through iteration. We also introduced the Enterprise Intelligence Model  to overlay this cycle onto enterprise operating models in which the business value and experiences of customers, colleagues, and partners may be improved through insights and automated decisions. We also introduced an Intelligence Maturity Model for helping you evaluate how advanced your organization is in creating insights from data and using those insights to drive decisions, actions and learnings. These thinking frameworks serve as a basis for this and other upcoming articles in this series. We’ll be examining the technical and business enablers for speeding up the Continuous Intelligence cycle in your organization.

In this article, we’ll explore common patterns of enterprise intelligence and identify the points of friction and opportunities for improvement in the Continuous Intelligence cycle. This article series is best read as a collection of dimensions for incremental improvements — things that are usually best taken in steps, on a test-and-learn basis, to find what works best for your unique situation. This approach requires executive support and sponsorship as well as grass roots buy-in from the knowledgeable colleagues in your organization. 

Organizational Archetypes

The Enterprise Intelligence Model introduced in the first article in this series offers a means of enabling Continuous Intelligence by examining those points of friction and opportunity in the intelligence cycle which are most likely to deliver results quickly, and to mature capabilities. In our experience, those considering how to enable Continuous Intelligence in their organization find it most helpful to identify common stages along the path of intelligence maturity as organizational archetypes. We’ve found framing these archetypes in the Enterprise Intelligence Model provides a clear way to express these stages of maturity. Future articles in this series will unpack the what, why, and how of the techniques in this article that we identify with the more mature stages of Enterprise Intelligence.

• From an organizational standpoint, we’ve observed that the following hurdles most commonly cause friction in creating value from data:The data being captured doesn’t reflect the richness of the customer and user experience 
• Insights gained losing much of their practical applicability because too much time is spent refining data models and schema for insight enablement and too much focus on reports as standalone artifacts
• There is a disconnect between the decision-making process and the insights needed to support that process, leading to suboptimal decisions.
• Feedback loops are ineffective because decisions aren’t carried out entirely as intended and so aren’t reflected in the user experience

Figure 1: Hurdles for data value creation

Each of these hurdles is represented as quadrant boundaries in Figure 4. Archetypes presented in this article are at various stages of overcoming some or all of the hurdles presented above.

Archetype #1: The legacy state

Business and tech behavior: Data analysis within these organizations centers around a conventional data warehouse architecture with some combination of enterprise data warehouse (EDW) and / or a collection of subject-area data marts (Figure 2). These patterns of data development usually impose heavy up-front quality and governance criteria that severely limit what type, condition and proportion of data is made available for analysis. The consumption of information is predominantly via business intelligence (BI) tools, which are typically used to handle only historical analyses. Therefore, any links between past events and future decisions are not structured and captured in enterprise systems. Even worse, these decisions are often arrived at via the proliferation of informal BI artifacts such as “spreadmarts” — or user created spreadsheets, that may not be well governed. 

Insights gleaned from these artifacts often rely on the reader’s interpretation because the structured analysis doesn’t clearly frame either predictions or decisions for execution. These predictions, decisions and actions are commonly ad hoc and manual, and are therefore not easily explained, understood, audited, repeated, reviewed or refined. This tends to severely limit the sophistication of any one decision, as well as the granularity that can be achieved across different decisions. Likewise, it often takes weeks or months to define and execute decisions based on these artifacts, due to the lack of a defined implementation process.

Even where decisions can be arrived at more quickly, delays in sourcing data tend to remain. Data is primarily sourced from operational systems using batch extraction, transformation, and load (ETL) jobs that run nightly or weekly, meaning that information latency is — at best — one full day after business events occurred.  

While BI capability is “table stakes” for most companies to maintain parity with competitors, it’s not a differentiator. Companies that exclusively rely on data warehousing approaches and BI to power the intelligence cycle fall into the ripe for disruption or intelligence envy quadrants of the Intelligence Maturity Model.  

State of Intelligence Cycle: There are many opportunities for improvement in intelligence speed and effectiveness in these organizations — for instance by adopting modern, adaptive data architectures to ease ongoing access to all forms of data; use of advanced analytical capabilities to generate clear predictions; and better business and technology collaboration methods to frame more effective decisions. In such organizations, the conversion of data into information for analysis isn’t very effective. Decisions and actions are rather ad hoc in this state and most decisions are instinct- not data-driven. Often data analysis is performed post-facto rather than during or before decision making. Hurdles exist across all four quadrants.

Figure 2: The legacy state archetype

Archetype #2: The data lake state

Business and tech behavior: The advent of big data technologies and public cloud infrastructure has given rise to companies that pursue a data hub or data lake at the heart of their data management architecture (see Figure 3). Sometimes these are replacing data warehouses or marts, but more often they are complementary to and separate from data warehouses. The more tech- and data-enabled of these companies are embracing event sourcing from operational systems to capture ongoing changes, and streaming techniques for publishing data to be consumed by downstream subscribers such as the data hub. This greatly reduces information latency and speeds up the Intelligence Cycle. In this pattern, the data warehouse consumes data from the hub and remains valuable for producing essential BI artifacts. 

State of Intelligence Cycle: Decisions and actions remain rather ad hoc in this state and improvements of the business experience may be sporadic and take months to implement. Nonetheless, the conversion of data into information for analysis is a substantial improvement over the legacy state. Reports are a primary artifact of insight communication for most decision-making purposes. Decisions are taken in a “batch” sense at human timescale rather than near real time as and when events unfold. Insight creation happens in close collaboration with data and information creation. The decision owner using the report isn’t very well captured or transparent to the larger data ecosystem. Also, to an extent the hub removes the insight enablement hurdle.

Figure 3: The modern data hub state

Archetype #3: The Data Science State

Business and tech behavior: Introducing data scientists can increase the value of the insights gleaned through the use of statistical and machine learning techniques to uncover patterns, create classifications and detect important signals (see Figure 4). Companies in consumer retail and finance were among the earliest adopters of data science (formerly data mining). Today, modern architectures enable high-speed processing of high-volume data of numerous types (e.g, images, text, etc.). This offers data scientists a much richer playground for experimentation, new discoveries and insights. In some data-science driven organizations, trained models are “bench models” (versus deployment on the data pipeline) that are presented to business stakeholders who interpret the results and decide what actions are appropriate. These organizations are generally in the upper left quadrant of the Intelligence Maturity model. They have advanced analytical methods but decisions and actions are ad hoc and subjective. Data science enables maturity along the Y axis of the Intelligence Maturity model. To mature along the X axis, they need to combine product thinking with encapsulating intelligence and decisioning.

State of Intelligence Cycle: What sets this stage apart is how closely the decision owner is involved, so that they’re part of insight creation journey. There is less friction between information creation, insight creation and decision makers. Science and experimentation also is another key differentiator at this stage. Most of these organizations exhibit analysis-paralysis behaviors, in which data science models, however sophisticated, remain in a POC-esque (proof of concept) lab state and fail to see real world utility. The fact that conventional BI capabilities co-exist alongside modern data science and ML capabilities is a step in the right direction. But translation of decisions to actions on the digital estate still remains poor, moreover, the ability to instrument, measure and act on the consequence of these decisions is very difficult in this stage. Decision to action is still a hurdle at this stage.

Figure 4: The data science state

Archetype #4: The insight product state 

Business and Tech behavior: Organizations in this state think about decisions and decision-centric products. (see Figure 5). In this state, analysts, scientists, decision- and delivery-owners are formally organized into a cross-functional product team with a clear charter to build business-aligned intelligence into insight products or directly into the data pipeline — and to rapidly combine this intelligence with automated decisioning to drive predictable actions. This collaborative work results in insights products that drive appropriate actions based on embedded intelligent models. For example, consider a credit card company using a loan application fraud model to trigger automated alerts and actions when a fraudulent loan application is detected. The model is trained by data scientists in collaboration with decision owners, and is baked into a software insight product by data engineers and developers and deployed into production for execution.

State of Intelligence Cycle: Here, the biggest improvement is the ability to think of decision journeys and encapsulate data, insights, decisions into a product and apply product thinking to Intelligence cycles. The Insight Product shown in Figure 5 is able to facilitate actions to be placed back in the operational tech estate to bring accountability to decision making. The biggest lever for business is that they move away from being reliant only on reports and ad hoc analysis for decision making, but instead, get into the mode of tuning decisions as insight products. This facilitates smooth decision journeys and at times provide ability to simulate scenarios quickly and experiment from a business point of view, with the ability to react to changes in the digital estate more quickly and predictably. Also, another great thing about this stage is that the hurdles are removed to a large extent and issues of speed exist in discrete places.

Figure 5: The intelligence state

Archetype #5: The Continuous Intelligence state 

Business and Tech behavior: The addition of continuous delivery for ML (CD4ML), platform thinking and a data devops culture leads to this state (see Figure 6). A machine learning platform is a data science enabler that reduces friction by enabling data scientists to focus on model training over data wrangling. It supports the sharing of machine learning elements within data science teams and enables versioning and model management. Continuous intelligence can be thought of continuous delivery for data. As data scientists create more refined and accurate models, they can easily deploy these into production as replacements for prior models. For example, at AutoScout24, the largest online automobile marketplace in Europe, price evaluation models can be continuously updated. This includes automated verification using live data in a continuous delivery pipeline that enables models to be updated with confidence. Continuous Delivery for Machine Learning (CD4ML) will be a deep-dive topic in upcoming articles in this series. It is in addition to the normal continuous delivery for data products and data infrastructure. It’s an essential component, and significant friction reducer, needed to optimize the Continuous Intelligence cycle. Creating a data devops culture is a necessary complement to the technical elements of ML platforms and continuous intelligence pipelines.

State of Intelligence Cycle: Adding to the product thinking in the previous stage, being able to create products which learn and complete the intelligence cycle in a continuous fashion is what sets this stage apart. The loops become more seamless and most of the hurdles are removed. Loops become tighter and faster with more use and more experimentation, which is a key indicator of the health of intelligence cycle.  

Figure 6: The Continuous Intelligence state

Intelligence enablers. Figure 7 explicitly calls out all of the enablers represented by the archetypes previously outlined. These enablers include technical efforts such as: 

• Modern data architectures, domain event publishing and streaming, and continuous delivery for machine learning (CD4ML); capability advancements such as data science and machine learning 
• Operating model changes such as platform thinking and product thinking, to reduce friction and create sustainable outcomes; focus moves from making data and moving data to creating products which are fuelled by data and creation of intelligence cycles
• Cultural changes such as building cross-functional product teams