Master
ThoughtWorks
Menu
Close
  • What we do
    • Go to overview
    • Customer Experience, Product and Design
    • Data Strategy, Engineering and Analytics
    • Digital Transformation and Operations
    • Enterprise Modernization, Platforms and Cloud
  • Who we work with
    • Go to overview
    • Automotive
    • Healthcare
    • Public Sector
    • Cleantech, Energy and Utilities
    • Media and Publishing
    • Retail and E-commerce
    • Financial Services and Insurance
    • Not-for-profit
    • Travel and Transport
  • Insights
    • Go to overview
    • Featured

      • Technology

        An in-depth exploration of enterprise technology and engineering excellence

      • Business

        Keep up to date with the latest business and industry insights for digital leaders

      • Culture

        The place for career-building content and tips, and our view on social justice and inclusivity

    • Digital Publications and Tools

      • Technology Radar

        An opinionated guide to technology frontiers

      • Perspectives

        A publication for digital leaders

      • Digital Fluency Model

        A model for prioritizing the digital capabilities needed to navigate uncertainty

      • Decoder

        The business execs' A-Z guide to technology

    • All Insights

      • Articles

        Expert insights to help your business grow

      • Blogs

        Personal perspectives from ThoughtWorkers around the globe

      • Books

        Explore our extensive library

      • Podcasts

        Captivating conversations on the latest in business and tech

  • Careers
    • Go to overview
    • Application process

      What to expect as you interview with us

    • Grads and career changers

      Start your tech career on the right foot

    • Search jobs

      Find open positions in your region

    • Stay connected

      Sign up for our monthly newsletter

  • About
    • Go to overview
    • Our Purpose
    • Awards and Recognition
    • Diversity and Inclusion
    • Our Leaders
    • Partnerships
    • News
    • Conferences and Events
  • Contact
Global | English
  • United States United States
    English
  • China China
    中文 | English
  • India India
    English
  • Canada Canada
    English
  • Singapore Singapore
    English
  • United Kingdom United Kingdom
    English
  • Australia Australia
    English
  • Germany Germany
    English | Deutsch
  • Brazil Brazil
    English | Português
  • Spain Spain
    English | Español
  • Global Global
    English
Blogs
Select a topic
View all topicsClose
Technology 
Agile Project Management Cloud Continuous Delivery  Data Science & Engineering Defending the Free Internet Evolutionary Architecture Experience Design IoT Languages, Tools & Frameworks Legacy Modernization Machine Learning & Artificial Intelligence Microservices Platforms Security Software Testing Technology Strategy 
Business 
Financial Services Global Health Innovation Retail  Transformation 
Careers 
Career Hacks Diversity & Inclusion Social Change 
Blogs

Topics

Choose a topic
  • Technology
    Technology
  • Technology Overview
  • Agile Project Management
  • Cloud
  • Continuous Delivery
  • Data Science & Engineering
  • Defending the Free Internet
  • Evolutionary Architecture
  • Experience Design
  • IoT
  • Languages, Tools & Frameworks
  • Legacy Modernization
  • Machine Learning & Artificial Intelligence
  • Microservices
  • Platforms
  • Security
  • Software Testing
  • Technology Strategy
  • Business
    Business
  • Business Overview
  • Financial Services
  • Global Health
  • Innovation
  • Retail
  • Transformation
  • Careers
    Careers
  • Careers Overview
  • Career Hacks
  • Diversity & Inclusion
  • Social Change
Software TestingTechnology

Just re-run the build - it should go green

Neil Philip Craven Neil Philip Craven

Published: Jan 1, 2014

"Just re-run the build - it should go green"

Do you hear this on your project? Are your tests flaky? Does the build change from green to red and back again without any real explanation? What are you doing about it?

Recently I joined a project that had this problem. It's quite normal to encounter it and it's quite common to get around it by just running the test again. Some teams had actually added scripts to pick up failed tests and run them again automatically at the end of the same build, in case they pass the second time around.

Flaky test or flaky system?

But how can you tell the difference between a flaky test and a flaky system? We certainly couldn't - it turned out that the system was flaky and our users experienced problems. This is not an easy problem to solve. Here are two approaches that helped us:

1. Sniff out the non-deterministic tests:

You should attempt to work out whether the problem is in the test code or the app code. Read Martin Fowler's bliki on Non-deterministic tests. Think about your own test suites - do you have any non-deterministic tests? You probably do unless you've been very careful. You might try to deconstruct your flaky tests and implement them at a lower level. You may lose a certain degree of coverage but the results of the test will be reliable. At the very least, you could watch the logs in real-time and where appropriate, check the UI of the test machine as the steps are running to see if they're behaving as expected.

2. Don’t ignore them, fix them.

Understanding your problem doesn't make it go away. You need to leave adding new features by the wayside to go and fix this stuff. Using the head-in-the-sand approach, our functional test suite quickly descended from being mostly green to mostly red. The day we had 13 out of 17 builds go red was the last straw and we decided more drastic action needed to be taken. What was our action? Simply having better discipline to follow the process we already had in place. We decided that we'd remove the option to re-run the suite without a code change, and we reinforced the rule that you were only allowed to push on a red build if what you were pushing was intended to fix the build.

Sure enough, by the end of the next day there were 7 people crowded around the build machine trying to fix these flaky build. As a result we discovered some badly written tests and identified six genuine issues with the application. The build light was still red, but instead of crossed fingers we had six juicy, clearly defined bugs to fix. 

Master
Privacy policy | Modern Slavery statement | Accessibility
Connect with us
×

WeChat

QR code to ThoughtWorks China WeChat subscription account
© 2021 ThoughtWorks, Inc.