An introduction to effective test assertions
In the software industry, before we deliver a product, we need to run many test assertions to make sure the product has met the customer's expectation. So what is a test assertion?
What is a test assertion?
In software industry, before we deliver a product, we need to run many test assertions to make sure the productmet customer's expectation.
So what's a test assertion? We can find from internet that, one of the popular definitions is "a test is a condition that must be tested to confirm conformance to a requirement". I quite like thisas it explains quite well what a test assertion is – a condition. And from functional test point of view,the conformance to a piece of requirement often needs more than one assertion.
Due to this, QAs need to work with lots of assertion in their everyday work, either manually or automatically,both depending on the complexity and time. Although no matter whether we run those assertions manually or, there is one thing that we cannot escape, which is designing them in advance, according to our. Good assertion is necessary; otherwise we risk gathering wrong/inaccurate information from QA work.can delay the delivery of software or leave issues uncovered in the product. Since designing good assertionso vital, how to design a good assertion?
Many might say, that's quite easy. Ok, before we come to a conclusion of whether it is easy or not, let'sa look at a very simple example using ThoughtWorks website.
Here we have an acceptance criteria (we use AC for short) as following: Given users on ThoughtWorks homepageusers click on 'Technology' link under 'How we work' drop-down menu Then users navigate to 'Technology'
Let's first take a look at what the homepage looks like:
And what technology page looks like:
Test the acceptance criteria
To test this AC, we need to click on the link ‘Technology’ within drop-down menu ‘How we work’, which is‘Given’(aka pre-condition) of the AC; then make sure we are navigated to technology page, which is‘When’ in AC.
If we try to do that with some code, it can look something like below:
[Action] browser.open("https://www.thoughtworks.com/");
browser.click("/html/body/div[@id='container']/div[@id='header']/div[@id='primary-navigation']/div[@id='block-1']/ul/li[5]/ul/li[1]/a");
browser.waitForPageToLoad("5000");
Once the action is done, we can verify the expected action result ‘Then’ of the AC.
There can be several ways to do the assertions for this simple AC, we show some of them here:
[Assertions]
Way I: We search if the page we navigate to contains 'Technology' and 'How we work'
assertTrue(
browser.isTextPresent('Technology')
); assertTrue(
browser.isTextPresent('How we work')
);
Way II: We check if we are navigated to correct URL
assertEquals( "https://www.thoughtworks.com/technology", browser.getLocation() );
Way III: We check if the title of the current page we are navigated to is calledTechnology - www.thoughtworks.com'
assertEquals(
"Technology - www.thoughtworks.com", browser.getTitle()
);
Way IV: We check if the left hand sidebar has technology highlighted:
assertEquals( browser.getAttribute("//div[@id='container']/div[3]/div[2]/a[1]@class"), "Active");
Re-visit the assertions
What do you think of the above assertions? Right, none of them are good enough for this very simple AC!
Let's take a closer look at them one by one and see what's wrong with them.
Way I:
assertTrue(
browser.isTextPresent("Technology")
); assertTrue(
browser.isTextPresent("How we work")
);
It checks if text 'Technology' and 'How we work' present on the page we are on. But they both exist on more thanpage, such as website footer, navigation bar, side bar which are present not only on Technology page. Usingway of doing assertion will hardly see any test fails. As a consequence, we risk passing software to clientfalse confidence.
Way II:
assertEquals( "https://www.thoughtworks.com/technology", browser.getLocation() );
This group checks if we are navigated to right location. If the URL is correct, then it means we are directed tocorrect page. However, this is not enough as the page content itself might be incorrect, there can bebetween the page location and content itself. So test can pass, but the functionality is broken.
Way III:
assertEquals(
"Technology - www.thoughtworks.com", browser.getTitle()
);
It's good to use different elements on the page for assertion, though it also has the same issue with Way II,test or not won't be able to provide us with accurate information about the status of theunder test.
Way IV:
assertEquals( browser.getAttribute("//div[@id='container']/div[3]/div[2]/a[1]@class"), "Active");
Writing correct assertions
Believe it or not, this one is quite a common mistake we can see in daily testing -- we are testing more than whatneed to. There is an extra business logic tested along with our target functionality; that is why we seeTechnology' option on sidebar highlighted when we are on Technology page. We have a failed test due to this function not working fine while our target functionality is running perfectly.
To do assertion right for this simple case, we have to remember the important principle of writing good assertions - The goal when testing an implementation is to achieve consistent reproducible results that accurately indicate the conformance of the implementation to the specification. In another word: the test assertion will fail if and only if the tested function is broken.
To achieve the goal, we have to examine the part of the software to test carefully, find out what's unique about it, and what we need to find out as a sufficient proof of its working.
Now let's come back to the Technology page and see what we can use.
First, we can find that the URL is supposed to be used only by 'Technology' page.
Second, the title of the page should also be correct while we are on the 'Technology' page.
But we have not yet proven that the page content is correct. So we have to find something unique in the page content. Examining carefully we can see that the purple span is something unique. It could be in another color which can be described on another AC. So we don't need that in this case, but have a span named Technology which should be unique.
Once we get these three identifiers, we can start to write our automated test again if we want.
It could be code as following:
assertEquals( "https://www.thoughtworks.com/technology", browser.getLocation() );
assertEquals( "Technology - www.thoughtworks.com", browser.getTitle());
assertEquals(browser.getText("/html/body/div[@id='container']/div[3]/p[1]/span"), 'Technology');
Now our test fails only when our functionality is not working. We can confidently say when this test is passed; that this piece of software is ready to be delivered.
So to be able to write good assertion, we need to understand the underlying checkpoints of a piece of requirement, make sure we go from the correct entry to the point we need to test, we need to find out unique identifiers to avoid confusion of the test result. Also, we need to be aware of not including implicit business logic that is not involved in the requirement to test, and, we have to understand that to do good assertion of a piece of requirement, we might need more than one assertion.
To sum up, analysis is just too important for writing good assertion.