On my last project, the client categorically asked the team to ensure that the Django app we were developing had first-class support for Internet Explorer 8. Now I know a number of developers who would cringe at the mere mention of Internet Explorer (IE). I have even heard it said that the only use of Internet Explorer is to download an alternative browser! What do you do when you need to set up a test suite that supports the running of functional tests with Internet Explorer? Do you:
- Switch roles?
- Feign illness?
- Run away?
- All of the above
Well, here are some more valiant options (that we actually considered and tried out), so you now have a longer list of possibilities!
Option 1: Use a Cloud-based Testing Service
Thank you SaaS/PaaS/IaaS! We now have a couple of platforms that offer testing as a service. The most prominent ones I have come across are SauceLabs and BrowserStack.
Using either of these, you can have your IE testing environment up in a snap without the pain of setting up any additional infrastructure. Moreover, they both offer options to test your internally hosted applications as well as to run tests in parallel. Additionally, the documentation on either site provides a lot of helpful information for getting started.
The Good
- Using a cloud testing service such as SauceLabs means there is no need to setup and/or maintain testing infrastructure. All that is required is (a connection to the cloud and) a code snippet to let your app know that you are using a remote webdriver (in this case, one hosted by SauceLabs or BrowserStack). This snippet would also point your test executor to the appropriate SauceLabs or BrowserStack (or whatever other testing service) node. Ordinarily, this should not be anything more than a couple of lines of code.
- SauceLabs in particular supports a wealth of platforms including Windows, mobile and desktop (last I checked 304 Device/OS/Browser Platforms were supported), all the way from the primeval IE6 to beta versions of Firefox and Chrome
- SauceLabs and BrowserStack both have a lot of useful documentation on their site to help you get started. This includes code snippets for most of the popular languages and platforms including Java, Python, Ruby, Node.js, PHP and .Net.
- Both automated and manual testing are supported. With manual testing, because the tests are actually being run in a VM, all the useful tools you need for debugging are already available. With automated tests, the test results come with full logs including video screencasts (in the case of SauceLabs) and a command history. Furthermore, each new test run is performed on a clean VM.
- SauceLabs offers a free tier for open source projects. This comes with unlimited testing minutes as well as (three) parallel VMs!
The Bad
- Using any of these cloud-testing services is way slower than using a local testing environment. On my team, we noticed increases in test execution time of 200% or even more in some cases.
- From experience, the likelihood of flakiness of tests tends to increase when using these cloud-based testing services. This is not too surprising, given the introduction of an additional dependency on communication over a network. Having an unreliable, slow, or high-latency network thus becomes a huge factor that determines if your tests pass or fail (or whether the tests run to completion at all).
- Depending on your project, using such a service may turn out to be quite expensive. Billing tends to be per minute, and your bill will directly reflect those automated tests that hang or take way longer than usual.
Option 2: Set up a Windows Virtual Machine
The Good
- Provides much faster feedback than using cloud-based testing services, since it avoids traversal over the network - which would inherently result in an increase in test execution times
- The tests are less likely to suffer from the flakiness that would have been caused by the dependency on communication over a network. This is particularly true as long as the VM is located on the local dev machine, or collocated in a datacenter with the dev machine and/or CI-tools
- Depending on your particular setup, this option is much more likely to be cheaper or even entirely free (for instance if using the freely available IE images from modern.ie on a VM hosted on your local machine)
- Particularly for manual testing, this option provides the most accurate representation of how a user would interact with the system
The Bad
- Choosing to use a Windows VM requires the setup and maintenance of additional infrastructure, including the possibility of having to manage Windows licenses. This also involves setting up a Selenium server on the specified VM and making sure that the server is accessible to and from the test initiator. Generally speaking, setup and maintenance is no simple feat.
- If the goal is to run tests across numerous platforms such as various versions of IE, each of these platforms will require a separate VM (unless you decide to use something such as IECollection which gives you various versions of IE that can be run at the same time). In any case, the corresponding setup/maintenance overhead will most likely quickly spiral out of control.
Bottom Line?
The choice to go with either of the above options ultimately depends on a trade-off between convenience and scalability on the one hand; and reliability and shorter feedback cycles on the other. Take your pick.