ThoughtWorks
  • Contact
  • Español
  • Português
  • Deutsch
  • 中文
Go to overview
  • Engineering Culture, Delivery Mindset

    Embrace a modern approach to software development and deliver value faster

    Intelligence-Driven Decision Making

    Leverage your data assets to unlock new sources of value

  • Frictionless Operating Model

    Improve your organization's ability to respond to change

    Platform Strategy

    Create adaptable technology platforms that move with your business strategy

  • Experience Design and Product Capability

    Rapidly design, deliver and evolve exceptional products and experiences

    Partnerships

    Leveraging our network of trusted partners to amplify the outcomes we deliver for our clients

Go to overview
  • Automotive
  • Cleantech, Energy and Utilities
  • Financial Services and Insurance
  • Healthcare
  • Media and Publishing
  • Not-for-profit
  • Public Sector
  • Retail and E-commerce
  • Travel and Transport
Go to overview

Featured

  • Technology

    An in-depth exploration of enterprise technology and engineering excellence

  • Business

    Keep up to date with the latest business and industry insights for digital leaders

  • Culture

    The place for career-building content and tips, and our view on social justice and inclusivity

Digital Publications and Tools

  • Technology Radar

    An opinionated guide to technology frontiers

  • Perspectives

    A publication for digital leaders

  • Digital Fluency Model

    A model for prioritizing the digital capabilities needed to navigate uncertainty

  • Decoder

    The business execs' A-Z guide to technology

All Insights

  • Articles

    Expert insights to help your business grow

  • Blogs

    Personal perspectives from ThoughtWorkers around the globe

  • Books

    Explore our extensive library

  • Podcasts

    Captivating conversations on the latest in business and tech

Go to overview
  • Application process

    What to expect as you interview with us

  • Grads and career changers

    Start your tech career on the right foot

  • Search jobs

    Find open positions in your region

  • Stay connected

    Sign up for our monthly newsletter

Go to overview
  • Conferences and Events
  • Diversity and Inclusion
  • News
  • Open Source
  • Our Leaders
  • Social Change
  • Español
  • Português
  • Deutsch
  • 中文
ThoughtWorksMenu
  • Close   ✕
  • What we do
  • Who we work with
  • Insights
  • Careers
  • About
  • Contact
  • Back
  • Close   ✕
  • Go to overview
  • Engineering Culture, Delivery Mindset

    Embrace a modern approach to software development and deliver value faster

  • Experience Design and Product Capability

    Rapidly design, deliver and evolve exceptional products and experiences

  • Frictionless Operating Model

    Improve your organization's ability to respond to change

  • Intelligence-Driven Decision Making

    Leverage your data assets to unlock new sources of value

  • Partnerships

    Leveraging our network of trusted partners to amplify the outcomes we deliver for our clients

  • Platform Strategy

    Create adaptable technology platforms that move with your business strategy

  • Back
  • Close   ✕
  • Go to overview
  • Automotive
  • Cleantech, Energy and Utilities
  • Financial Services and Insurance
  • Healthcare
  • Media and Publishing
  • Not-for-profit
  • Public Sector
  • Retail and E-commerce
  • Travel and Transport
  • Back
  • Close   ✕
  • Go to overview
  • Featured

  • Technology

    An in-depth exploration of enterprise technology and engineering excellence

  • Business

    Keep up to date with the latest business and industry insights for digital leaders

  • Culture

    The place for career-building content and tips, and our view on social justice and inclusivity

  • Digital Publications and Tools

  • Technology Radar

    An opinionated guide to technology frontiers

  • Perspectives

    A publication for digital leaders

  • Digital Fluency Model

    A model for prioritizing the digital capabilities needed to navigate uncertainty

  • Decoder

    The business execs' A-Z guide to technology

  • All Insights

  • Articles

    Expert insights to help your business grow

  • Blogs

    Personal perspectives from ThoughtWorkers around the globe

  • Books

    Explore our extensive library

  • Podcasts

    Captivating conversations on the latest in business and tech

  • Back
  • Close   ✕
  • Go to overview
  • Application process

    What to expect as you interview with us

  • Grads and career changers

    Start your tech career on the right foot

  • Search jobs

    Find open positions in your region

  • Stay connected

    Sign up for our monthly newsletter

  • Back
  • Close   ✕
  • Go to overview
  • Conferences and Events
  • Diversity and Inclusion
  • News
  • Open Source
  • Our Leaders
  • Social Change
Blogs
Select a topic
View all topicsClose
Technology 
Agile Project Management Cloud Continuous Delivery  Data Science & Engineering Defending the Free Internet Evolutionary Architecture Experience Design IoT Languages, Tools & Frameworks Legacy Modernization Machine Learning & Artificial Intelligence Microservices Platforms Security Software Testing Technology Strategy 
Business 
Financial Services Global Health Innovation Retail  Transformation 
Careers 
Career Hacks Diversity & Inclusion Social Change 
Blogs

Topics

Choose a topic
  • Technology
    Technology
  • Technology Overview
  • Agile Project Management
  • Cloud
  • Continuous Delivery
  • Data Science & Engineering
  • Defending the Free Internet
  • Evolutionary Architecture
  • Experience Design
  • IoT
  • Languages, Tools & Frameworks
  • Legacy Modernization
  • Machine Learning & Artificial Intelligence
  • Microservices
  • Platforms
  • Security
  • Software Testing
  • Technology Strategy
  • Business
    Business
  • Business Overview
  • Financial Services
  • Global Health
  • Innovation
  • Retail
  • Transformation
  • Careers
    Careers
  • Careers Overview
  • Career Hacks
  • Diversity & Inclusion
  • Social Change
Retail Evolutionary ArchitectureTransformationInnovationBusinessTechnology

Experimentation helps retailers optimize their supply chain - here’s how

Praful Todkar Praful Todkar

Published: Jun 25, 2019

We have been collaborating with a large US based fashion retailer to improve their supply chain efficiency. One of the key factors that help solve supply chain puzzles is making sure online orders are fulfilled from the most profitable location – a warehouse or a store that maximizes the profit margin on the order. This is the responsibility of the Order Routing algorithm. To identify the most profitable fulfilment location, the Order Routing algorithm uses a variety of ‘levers’ such as the shipping address, shipping service (example, Expedited vs Regular shipping), inventory, among others.

The client’s Order Routing team wanted to improve their algorithm by leveraging demand forecast and further optimize for gross margin per online order. The principle here is that demand forecast pinpoints the most ‘disadvantaged’ location – a location where the product is likely to go on markdown (a 50% off sale) in the near future. Therefore, margin on the online order can be improved by fulfilling the product from such a disadvantaged location.

Let’s dig into this a little with an example. It’s fall season and the retailer has received an online order for a pair of shorts. There are two stores that stock the product. One, based out of Chicago and other in San Diego. Our demand forecast tells us that the approaching winter in Chicago will ensure less demand for the product, which means the shorts will go on markdown. As opposed to this, in sunny San Diego the product will sell full price. The order should be fulfilled from Chicago, the disadvantaged location, to yield a better margin.

A case for the Test-and-Learn Platform

The Order Routing algorithm is a ‘sensitive’ algorithm and small changes to it can have a big impact on the margin of fulfilled orders. One example could be; accounting for higher shipping costs of one location over another. This is why prototyping is a preferred path when evolving the algorithm.

The project team’s new demand forecast approach went through the prototype stage as well but, with certain limitations. The prototype could not be built for the entire data set due to sheer volume which meant inaccurate results. There was also no real time feedback from the changes planned for the production software, as the prototype was disconnected from the production systems. We needed to find a better solution!

Genesis of the Platform

Our client partnered with ThoughtWorks not just for our software development expertise but also to inculcate the culture and capabilities of Lean Thinking (test-and-learn). And, given the algorithm’s sensitivity, a real time feedback loop for software changes was essential.

While a solution was identified based on prototype results, it did not guarantee a lot of confidence. In response, the team built a test-and-learn platform that pitted multiple solutions against one another, and used business metrics to determine the best solution.

While test-and-learn is a commonly used technique to test variations of user interfaces, this situation was unique because we were testing business logic. Moreover, it had to be carried out on an enterprise backend, low latency system within a complex business domain and a high risk environment.

Getting this right was paramount and called for a well thought out architecture that would support the business need.

Test-and-Learn Platform Architecture

Let’s get into a little more details of the kind of architecture we were working with. A test-and-learn platform conventionally has 3 components:
  • An experimentation framework to run multiple experiments in parallel
  • An analytics framework to validate the experiments
  • A business impact dashboard to monitor business health metrics in real time
Low-fi implementation for faster feedback
Test-and-learn platform architecture
The algorithm in production is called the ‘active’ algorithm. The new solutions are implemented as experiment algorithms in ‘passive’ mode –  running in isolation without affecting the active algorithm. This safety feature was instrumental in getting us the green light to build the platform. The active algorithm’s input parameters are passed on to the experiment algorithms, along with the necessary transient data. This helps with an ‘apples-to-apples’ comparison.

The analytics framework compares the experiment algorithms’ output with the active algorithm’s output using complex business calculations (gross margin lift). The winning experiment algorithm is promoted to production, making it active – merging it with the active algorithm or running it alongside the active algorithm.

The aforementioned business impact dashboard monitors the newly promoted experiment algorithms’ impact, on the health metrics. These metrics are the Order Routing’s Key Performance Indicators (KPIs).

The Challenges and their Solutions

Although the test-and-learn platform has been a great success, the project team did overcome their fair share of challenges to meet this success.


Testing high-risk experiments

Running the experiments in passive mode was the perfect way to avoid negative fallout from the experiments on the Order Routing algorithm. Business users try new improvements without worrying about consequences. Additionally, business could turn off a newly promoted experiment after reading the business impact dashboard, and immediately have orders routed through the earlier, safe version of the algorithm.


Receiving fast feedback in an enterprise setting

Quick feedback is the lifeblood of experimentation, and the team built a low-fi implementation for the dependency; demand forecast. The team imported a small subset of real data into the application database and manually refreshed it at regular intervals. This low-fi implementation meant the team could launch the experiment in just a few weeks instead of a few months, and receive valuable feedback from production. The low-fi and hi-fi implementation shared a common interface which resulted in minimal change (and a lot less throw away work) when migrating to the hi-fi implementation.

Test-and-learn platform architecture
Low-fi implementation for faster feedback


Using real data for testing

The real data used in the Low-Fi version was periodically refreshed from the external dependency. This was a temporary hassle but it helped uncover a lot of issues early on – not only in the algorithm but also in the data from the dependency. For example, of different demand forecast variants, we requested the appropriate variant based on initial learnings. This helped us to influence the development of the external dependency as well.


Receiving fast feedback in a complex domain

Conventionally, web based A/B test results can be analyzed relatively quickly. However, in our case, the algorithm’s effectiveness, measured in gross margin lift, could only be carried out after 6-12 weeks. In response to this longer loop, the project team came up with the "feedback confidence levels" concept. A quick but low confidence level was used to make small tweaks to the algorithm, while the highest confidence level (that took longer) was used to determine eligibility for promotion to production.


Identifying direct cause and effect relationship KPIs

The business impact dashboard’s KPIs needed to have a direct cause and effect relationship with the code being changed, while also staying aligned to the organization’s KPIs. For example, shipping costs is an organizational KPI. But this metric could be affected by multiple organizational initiatives that sometimes even run contrary to each other.

To avoid system ‘noise’, the project team picked a metric that was only affected by our system while giving us a directional read on the organizational KPI. One such metric was ‘percentage of shipments from stores versus warehouses’. The code in our system directly impacted that percentage. As a general rule of thumb, shipments from warehouses are cheaper than stores and hence if the percentage of shipments skewed largely in favor of stores after a certain experiment went live, it would increase shipping costs, thus negatively affecting the organization’s ‘shipping cost’ KPI.

The metrics were identified by the same team developing the experiment which meant they would be prioritized as per the project’s needs. It was also encouraging to witness the team’s emotional investment in the success of the experiment since the metric was real time validation not only of the experiment but also the hard work put in.

Additional benefits of the Platform

One of the unintended but pleasant consequences of having a business impact dashboard was relooking at existing business inefficiencies and identifying new opportunities. For example, store shipments would skew from normal to becoming excessively high during certain business hours.  

The test-and-learn platform created highly specialized algorithms for certain types of orders, like expedited orders, which have incidentally been a huge upgrade for the product.

Conclusion

It has been a game changing experience to solve business problems with creative solutioning and validating the right solution through rapid experimentation. In our particular scenario, during the test phase, we averaged a +2% in gross margin per single SKU transaction.

This effectively meant that with the new test-and-learn platform, there had been a drastic reduction in cost of validating business hypotheses. And, the project team had gone on to enhance the Order Routing product, and consequently the architecture, to support experimentation and analytics as first-class concepts.

Contact Us

We'd love to help you on your digital journey.

Get in touch
Related blogs
Technology Strategy

The Art of Platform Thinking

Ryan Murray
Learn more
Transformation

The well-factored approach to securing ROI on your service investment: Part 1

Praful Todkar
Ryan Murray
Learn more
Experience Design

How to Implement Hypothesis-Driven Development

Barry O'Reilly
Learn more
  • What we do
  • Who we work with
  • Insights
  • Careers
  • About
  • Contact

WeChat

×
QR code to ThoughtWorks China WeChat subscription account

Media and analyst relations | Privacy policy | Modern Slavery statement ThoughtWorks| Accessibility | © 2021 ThoughtWorks, Inc.