ThoughtWorks
  • Contact
  • Español
  • Português
  • Deutsch
  • 中文
Go to overview
  • Engineering Culture, Delivery Mindset

    Embrace a modern approach to software development and deliver value faster

    Intelligence-Driven Decision Making

    Leverage your data assets to unlock new sources of value

  • Frictionless Operating Model

    Improve your organization's ability to respond to change

    Platform Strategy

    Create adaptable technology platforms that move with your business strategy

  • Experience Design and Product Capability

    Rapidly design, deliver and evolve exceptional products and experiences

    Partnerships

    Leveraging our network of trusted partners to amplify the outcomes we deliver for our clients

Go to overview
  • Automotive
  • Cleantech, Energy and Utilities
  • Financial Services and Insurance
  • Healthcare
  • Media and Publishing
  • Not-for-profit
  • Public Sector
  • Retail and E-commerce
  • Travel and Transport
Go to overview

Featured

  • Technology

    An in-depth exploration of enterprise technology and engineering excellence

  • Business

    Keep up to date with the latest business and industry insights for digital leaders

  • Culture

    The place for career-building content and tips, and our view on social justice and inclusivity

Digital Publications and Tools

  • Technology Radar

    An opinionated guide to technology frontiers

  • Perspectives

    A publication for digital leaders

  • Digital Fluency Model

    A model for prioritizing the digital capabilities needed to navigate uncertainty

  • Decoder

    The business execs' A-Z guide to technology

All Insights

  • Articles

    Expert insights to help your business grow

  • Blogs

    Personal perspectives from ThoughtWorkers around the globe

  • Books

    Explore our extensive library

  • Podcasts

    Captivating conversations on the latest in business and tech

Go to overview
  • Application process

    What to expect as you interview with us

  • Grads and career changers

    Start your tech career on the right foot

  • Search jobs

    Find open positions in your region

  • Stay connected

    Sign up for our monthly newsletter

Go to overview
  • Conferences and Events
  • Diversity and Inclusion
  • News
  • Open Source
  • Our Leaders
  • Social Change
  • Español
  • Português
  • Deutsch
  • 中文
ThoughtWorksMenu
  • Close   ✕
  • What we do
  • Who we work with
  • Insights
  • Careers
  • About
  • Contact
  • Back
  • Close   ✕
  • Go to overview
  • Engineering Culture, Delivery Mindset

    Embrace a modern approach to software development and deliver value faster

  • Experience Design and Product Capability

    Rapidly design, deliver and evolve exceptional products and experiences

  • Frictionless Operating Model

    Improve your organization's ability to respond to change

  • Intelligence-Driven Decision Making

    Leverage your data assets to unlock new sources of value

  • Partnerships

    Leveraging our network of trusted partners to amplify the outcomes we deliver for our clients

  • Platform Strategy

    Create adaptable technology platforms that move with your business strategy

  • Back
  • Close   ✕
  • Go to overview
  • Automotive
  • Cleantech, Energy and Utilities
  • Financial Services and Insurance
  • Healthcare
  • Media and Publishing
  • Not-for-profit
  • Public Sector
  • Retail and E-commerce
  • Travel and Transport
  • Back
  • Close   ✕
  • Go to overview
  • Featured

  • Technology

    An in-depth exploration of enterprise technology and engineering excellence

  • Business

    Keep up to date with the latest business and industry insights for digital leaders

  • Culture

    The place for career-building content and tips, and our view on social justice and inclusivity

  • Digital Publications and Tools

  • Technology Radar

    An opinionated guide to technology frontiers

  • Perspectives

    A publication for digital leaders

  • Digital Fluency Model

    A model for prioritizing the digital capabilities needed to navigate uncertainty

  • Decoder

    The business execs' A-Z guide to technology

  • All Insights

  • Articles

    Expert insights to help your business grow

  • Blogs

    Personal perspectives from ThoughtWorkers around the globe

  • Books

    Explore our extensive library

  • Podcasts

    Captivating conversations on the latest in business and tech

  • Back
  • Close   ✕
  • Go to overview
  • Application process

    What to expect as you interview with us

  • Grads and career changers

    Start your tech career on the right foot

  • Search jobs

    Find open positions in your region

  • Stay connected

    Sign up for our monthly newsletter

  • Back
  • Close   ✕
  • Go to overview
  • Conferences and Events
  • Diversity and Inclusion
  • News
  • Open Source
  • Our Leaders
  • Social Change
Blogs
Select a topic
View all topicsClose
Technology 
Agile Project Management Cloud Continuous Delivery  Data Science & Engineering Defending the Free Internet Evolutionary Architecture Experience Design IoT Languages, Tools & Frameworks Legacy Modernization Machine Learning & Artificial Intelligence Microservices Platforms Security Software Testing Technology Strategy 
Business 
Financial Services Global Health Innovation Retail  Transformation 
Careers 
Career Hacks Diversity & Inclusion Social Change 
Blogs

Topics

Choose a topic
  • Technology
    Technology
  • Technology Overview
  • Agile Project Management
  • Cloud
  • Continuous Delivery
  • Data Science & Engineering
  • Defending the Free Internet
  • Evolutionary Architecture
  • Experience Design
  • IoT
  • Languages, Tools & Frameworks
  • Legacy Modernization
  • Machine Learning & Artificial Intelligence
  • Microservices
  • Platforms
  • Security
  • Software Testing
  • Technology Strategy
  • Business
    Business
  • Business Overview
  • Financial Services
  • Global Health
  • Innovation
  • Retail
  • Transformation
  • Careers
    Careers
  • Careers Overview
  • Career Hacks
  • Diversity & Inclusion
  • Social Change
Software TestingBrazilTechnology

Functional Tests - How to Decide What to Automate?

Daniel Amorim Daniel Amorim
Hugo Vares Vieira Hugo Vares Vieira

Published: Dec 15, 2015

When we are working on software delivery projects, we will eventually face the decision of whether or not we should automate a test for a specific scenario. The question of whether we should automate or not, is what we will discuss in this article.

Why we Automate

Generally, we automate to avoid repeated manual work, get faster feedback, save time on running tests over and over again, and ensure we are always executing tests consistently with the same preconditions and expectations.

Repeated manual tests are boring. Also, humans executing a bunch of tests manually, doesn't provide the same result each time. Factors such as confirmation bias and human error come into play. At some point, a person could execute a test manually and miss an important step, thereby failing to identify a bug in the application being tested.

How to decide what to automate or not

Fast feedback is important for any successful continuous integration strategy, and critical as you move towards Continuous Delivery. It allows you to identify if there is any regression in the quality at the time the code is committed, allowing the team to fix problems as soon as they are introduced. Fast feedback also allows us to have our test scenarios reproducible at any time we need to run them again. The team will also rely on it in order to implement changes, or to re-factor the code.

But, even with all of these as benefits, it doesn't kill the need to have exploratory tests in the application at some point in the pipeline.

What we Automate

When deciding what tests to automate, firstly we need to look to Mike Cohn’s Test Pyramid. In essence, the pyramid prescribes that there are many more unit tests than higher level integration, service or UI automated tests.

The natural instinct of someone who is new to test automation is to attempt to cover every single scenario that they would test manually. Continuous integration requires automated tests, because for every small change, we are able to re-execute all regressions and make sure we aren't breaking anything.

However, functional tests can unintentionally create a huge set of test cases, thereby generating inconvenient consequences. Even if for each new story we add a small set of functional tests after some iterations, our regression suite will take considerable time to run. A long running regression test suite quickly becomes a bottleneck when committing changes. As a consequence, developers will commit less frequently, with larger commits.

As an application evolves, we don’t simply want to maintain an ever growing set of test cases that are passing forever. We want a test set that guarantees that the new and existing features are working as expected. A more efficient approach would be to cover the areas which are more important from a business perspective, or are known to be unstable and have a higher probability to introduce defects.

Our focus should be to create a high value test suite, which focuses on the business critical areas of our product. To achieve our goal, we need to take both views into account by associating two concepts with each story: business relevance and rate of error of each feature. This way, we can build a matrix to determine which features have more impact and should have more tests.

For example:

Features Business Relevance Error Rate Total
Inbox 2 3 5
New mail 1 2 3
Login 3 1 4
Settings 1 1 2

As we know, the business relevance changes over time and the probability of failure in a test, that is passing for a long time, is low. Test results should be reviewed constantly to determine which should remain in our functional test set. So, we suggest that you evaluate your tests at every iteration.

Conclusion

When implementing continuous integration, it is important to have a suite of automated tests that executes quickly and provides confidence in your product. Being able to deliver business value quicker than your competitors is becoming very important in a digital marketplace. A good test automation strategy that can ensure quality software faster, puts you a step ahead of your competitors. But, we know that there is no silver bullet when we talk about test strategy. It will always depend on the project and the kind of tests we need to focus on, to achieve the project goals.

This post was designed and produced by Kaifeng Zhang

  • What we do
  • Who we work with
  • Insights
  • Careers
  • About
  • Contact

WeChat

×
QR code to ThoughtWorks China WeChat subscription account

Media and analyst relations | Privacy policy | Modern Slavery statement ThoughtWorks| Accessibility | © 2021 ThoughtWorks, Inc.