Speeding up Test Execution with Appium

Published: Jan 29, 2016

Background

Mobile Test Automation is increasingly becoming very important. Almost all web applications are responsive these days and it's very important to test how the application works across devices. The same is true with the native application as well. At the same time, the number of devices and the custom OS versions on devices are also vast. This means that it's harder for a tester to manually run the automated tests over a list of devices to get device coverage and quicker results over every feature development.

Hence there is a need to run the tests parallelly, which provides faster feedback about the quality to your team.There are a couple of open source tools in the market to automate mobile applications. One of them is Appium, a cross platform automation tool, which can run tests on both Android and iOS. There are a number of good blogs, issue threads on github and other discussion forums, which talk about how to run Appium Android tests in parallel across multiple devices with the help of Selenium Grid.

Existing Challenge

Although there is a lot to learn from these helpful discussions, the setup is expensive and hard to maintain. Here is why: every instance of Appium server with the Android UDID has to be registered with the hub (check how selenium grid works with appium), which means that if you want to run your tests across ten devices, you need to run ten nodes.

Imagine how much system memory you will need if you have to run your tests across fifteen or twenty devices. Also, this becomes more tedious when you have to run your tests with CI across devices.

Solution

I did a lot of reading on how to make this process (of running tests across devices) easier and came up with the below mentioned approach, which would help everyone run tests in a distributed fashion, to get quicker results and to run tests concurrently, to get better device coverage. In the image below, you can see that the devices are connected in parallel and tests are executed across the devices.

Right now with v3.0.3, this tool supports only running Android (Native,Hybrid and Web) tests across devices in parallel; support for iOS parallel execution over real devices will soon be implemented.

Usage is very simple. All you need to do is to add the dependencies to your project, create a runner class and connect N+1 devices to a single machine.

Implementation details

To Run in Parallel Mode

The goal is to get device coverage, so we trigger N threads for N devices in parallel and push all the tests to each of the N threads. The Appium server generates runtime ports, starts the server and pushes the tests to the devices.

To Run in Distributed Mode

The goal is to get faster feedback on every feature completed. So we trigger N threads for N devices in parallel. However, instead of pushing all tests to all devices, we distribute the tests across the available threads, and the Appium server generates runtime ports, starts the server and pushes the tests to the devices.

Executor: TestNG

For example: pom.xml should look like this:

<repositories>
	<repository>
		<id>jitpack.io</id>
		<url>https://jitpack.io</url>
	</repository>
</repositories>

<dependencies>
	<dependency>
		<groupId>com.github.saikrishna321</groupId>
		<artifactId>AppiumTestDistribution</artifactId>
		<version>3.0.3</version>
	</dependency>
<dependencies>

Runner.java should look like this:

public class Runner {
	@Test
	public static void testApp() throws Exception {
		ParallelThread parallelThread = new ParallelThread();
		parallelThread.runner("com.test.site");
	}
}

Initialize the AppiumParallelTest runner, which takes care of triggering the Appium session in parallel.

public class UserBaseTest extends AppiumParallelTest {

	@BeforeMethod()
	public void startApp(Method name) throws Exception {
		startLogResults(name.getName());
	}

	@AfterMethod()
	public void killServer(ITestResult result) {
		endLogTestResults(result);
		getDriver().resetApp();
	}

	public AppiumDriver getDriver() {
		return driver;
	}

	@BeforeClass()
	public void beforeClass() throws Exception {
		driver = startAppiumServerInParallel(getClass().getSimpleName());
	}

	@AfterClass()
	public void afterClass() throws InterruptedException, IOException {
		killAppiumServer();
	}
}

You need to specify the Android APK file path, LaunchActivity and TargetPackage of the Android application under test, in the config.properties file.

APP_PATH=/Users/saikrisv/Documents/workspace/PagePatternAppium/build/wordpress.apk
APP_PACKAGE=org.wordpress.android
APP_ACTIVITY=org.wordpress.android.ui.WPLaunchActivity
RUNNER=parallel
APPIUM_JS_PATH=/usr/local/lib/node_modules/appium/bin/appium.js
APP_WAIT_ACTIVITY=org.wordpress.android.ui.accounts.SignInActivity
APP_TYPE=native
BROWSER_TYPE=chrome

You can trigger the test in two different ways -Distribute and Parallel - which means that when you set the Runner to ‘Parallel’ in the config, the same set of tests run on all the devices connected, which helps us get device coverage.

When you set the Runner to ‘Distribute’ in the config, the framework will distribute the tests across devices, which helps us to get quicker results.

Roadmap

Implement parallel run at method level. Currently, when the execution mode is set to Distribute, the tests are distributed at class level to the devices. In future, the scope is to distribute tests at method level.
Implement parallel run for iOS real devices for Native,Hybrid and Web apps. Current scope is to run tests in parallel across Android devices. The same implementation will be added to iOS real devices.
Implement ScreenCast as attachment for Android tests. Current scope is to attach a screenshot on failure to the reports, with future improvements to include adding screen-recording as well for test failures
Image Comparison for UI Verification(using ImageMagick API). Add visual assertions to compare two images and validate

The framework does generate detailed HTML reports (using Extent API) which includes AppiumServerLogs and ADBLogs. For test failures, it includes screenshots.