Should your quality assurance processes and tools change if you are building an application for the cloud? The answer is obvious - "it depends." Despite having one of the clearest definitions amongst recent buzzwords, there are many types of clouds and many ways to integrate them with your application or development processes. If you have an experienced team that can make informed choices, then these options are a great thing for quality. The team can take advantage of the cloud as a tool to assure quality, rather than viewing it as an extra aspect that needs to be tested.
There are a few clear definitions of the cloud that I find useful.
The first one from Gartner only describes a single cloud use case, but is a great way to promote cloud quality:
Virtual lab environment. These customers typically try to provide self-service infrastructure to a group of technical users, such as developers, scientists or engineers, for the purposes of test and development, or scientific computing or other batch computing. (From Gartner's "Magic Quadrant for Public Cloud Infrastructure as a Service")
The self-service nature is very important. If you only need an environment for 30 minutes to run a spike, than it is wasteful if you need to spend more than 30 minutes budgeting or scheduling the creation of that environment. One of my colleagues quipped: “I can spin up a VM in minutes, destroy it whenever, and repeat this process as frequently as I need. Anything less is just yesterday's infrastructure. If I have to email someone, open a ticket, submit a request to get a VM, etc. then it's yesterday's infrastructure.”
Let’s see how we can we take advantage of these cloud characteristics for the 8 dimensions of quality assurance. In Part I we’ll examine how the cloud helps with Performance, Features, Reliability and Conformance. In Part II we’ll describe the positive effects of the cloud on Durability, Serviceability, Aesthetics and Perceived Quality.
Performance refers to a product's primary operating characteristics and involves measurable attributes. I believe performance is the #1 business feature. There have been many studies1 showing that even a small reduction in performance can have a big business impact. The cloud helps you improve your performance in several ways.
Since rapid elasticity is an essential characteristic of cloud computing, you have the option to ramp up compute resources to match spikes or recurring demand trends. For example, you could run more servers during weekdays than on the weekend. You're able to make these fine-tuned adjustments without blowing the budget, because another essential characteristic - measured service - means you are usually billed per-hour (or even in smaller chunks of time) for the resources.
The broad network access characteristic of the cloud makes it easy to take advantage of high-speed networks, fast DNS services, and CDNs like Akamai to speed up your application. This is especially important for mobile applications, where networks are slower, making every byte or millisecond of latency count. A CDN can help you cache content closer to your users, compress your content, and make sure it supports conditional requests. All of these can significantly speed up your site.
There are many cloud services offered to help you test or increase the performance of your site. It is difficult and expensive to build your own performance test framework that can rapidly ramp up to simulate a large number of users. Often homegrown frameworks run into trouble because they themselves run into network or performance problems before the application they are testing, or because they lack features to simulate users in other parts of the world, or due to slow connections. Instead of building your own, you can use a service like Blitz.io. Blitz can economically simulate 1000 users in the network of your choice for 1 minute and even scale this up to 50000 users for 20 minutes.
If you need to test a more complex workflow, you can use something like BlazeMeter, which lets you run JMeter or Selenium tests from their cloud.
Features are additional characteristics that enhance the appeal and value of the product or service to the user. However, this is just a hypothesis until you put the feature in front of real users and validate its value. Functionality that is well-tested but not useful is not a feature, it is "high-quality bloat". The cloud can help you avoid bloat and quickly test that your features are properly implemented.
In order to quickly test features, we can take advantage of the elastic capabilities of the cloud. You can use tools like [jclouds-jenkins] to make sure your Continuous Delivery pipeline can handle a spike in commits without running out of nodes. You can use tools like Vagrant (with your choice of providers like VMWare or vagrant-rackspace, and provisioner like Puppet, Chef or Ansible) to quickly create a test environment, and destroy it once you're finished. You can also use SaaS testing services, like SauceLabs, Appium, Xamarin, Appium or Soasta so you can speed up your test suite by running tests in parallel or on multiple browsers or mobile devices.
One strategy to avoid bloat is to keep your application focused on solving the business domain, and trusting SaaS partners for related services. A good example is sending mail. It is not easy to quickly process a mail template and send it to a large number of subscribers. In addition to building the capacity to quickly process templates and send emails, you need to process unsubscription requests, spam complaints, email campaign schedules, analytics and much more. Providers like Mailgun let you outsource these problems and focus on your core business. These sorts of providers are available for many ancillary needs, like video encoding through Zencoder or payment processing through PayPal.
Applications such as Mailgun support analytics and A/B testing, two methods that help ensure quality in the features you deliver. Analytics, A/B and multivariate testing make it possible to run experiments to see what features or content provides the most value. Other cloud services that help you implement these experiments include Google Analytics Content Experiments, Optimizely and VWO. You can also integrate them directly into your application, using libraries like the split gem for Ruby.
If you use these techniques, you can:
The reason we recommended so many SaaS solutions in this because they are highly reliable. PayPal has shared their cloud infrastructure is the secret to their reliability.
Reliability is the likelihood that a product will not fail within a specific time period, and can be especially critical for certain domains. Reliability may be the dimension of quality where the cloud provides the biggest advantage. Even though individual components on the cloud may fail, the cloud makes it very easy to "design for failure". Thus you will have a resilient application that can survive even the most severe outages.
Most public cloud providers offer services from several data centers across the world. You can take advantage of this to easily build a highly redundant application or create several disaster recovery sites. If that's not enough, you could use RightScale, which helps you manage multiple public clouds across the globe, as well as your own private clouds. If you want to make sure your application never goes down, you could deploy it in several sites across the world on Rackspace, Amazon, and your own private OpenStack instance.
The cloud also makes it extremely simple to manage backups. Object Storage services like Amazon S3 and Rackspace Cloud Files provide data redundancy and virtually unlimited storage. You can simply click (or schedule) when you want a backup.
There are also many great cloud monitoring systems, so you can detect a deteriorating service before it becomes a major problem. You can set up monitoring alerts for most cloud infrastructure through the vendor, and can also integrate your application with great monitoring and log management services like NewRelic, Loggly or Splunk Storm.
Conformance is the precision with which the product or service meets specified standards. The cloud can help with conformance or compliance needs either by letting you avoid the need for compliance to a standard by using a service provider, or by providing consistent infrastructure to help ensure conformance.
Often, you can defer an ancillary feature to a compliant SaaS provider, so you don't need to deal with the requirement yourself. You can use Mailgun's expertise on the CAN-SPAM Act, like ensuring your e-mails have an unsubscribe link and a physical mailing address. You can let PayPal or Braintree Payment Gateway deal with the majority of your PCI-DSS compliance concerns. If you need strong auditing and security to guarantee compliance there are services like Dome9, which provide advanced security and compliance auditing solutions for your cloud setup.
Let's not forget about the less intimidating compliance tests either. Are you conforming with IEEE or W3C standards? If you're configuring your own caching servers you might make a subtle mistake that causes problems or slows performance for some users. If you're using the caching layer provided by your cloud's CDN then it is unlikely they would be configured in a way that causes problems.
Tools like RedBot, which "finds common protocol mistakes", Google Page Speed or YSlow can help even further, and check on caching, content-negotiation and compression.
1 Amazon found every 100ms of latency cost them 1% in sales.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.