What’s your Test Strategy?
When I heard that the application my team was building was going to be hosted on the cloud, one of the first thoughts that crossed my mind was, “Is there anything particularly different about testing an application on the cloud?”
My instinct was that when you come right down to it, there is no particular difference in the way I would approach testing this application versus the others I had tested before, that were hosted on physical servers. I then did some research, spoke to other Testers, and googled. But while the lack of conversation around this topic on the web validated in part my original instinct that testing on the cloud isn’t fundamentally different, I came to realize that “cloud awareness” is a must.
So what is “Cloud awareness”?
When the team actually started testing the application, there were a number of different types of tests we ended up doing, that we had not actually paid much attention to in earlier projects. I’ll get into the details in a bit, but the main takeaway for me was "cloud awareness" -- the very fact that this application was hosted on the cloud meant that there were some cloud-specific checks and balances built into the application. And that obviously meant we had to test all of them to make sure they work as intended. This in turn, meant that I had to develop a basic understanding of Cloud Computing, especially with regard to the aspects outlined below:
Security
The application under test is basically a set of custom payment gateways for credit and debit cards.
Since that is pretty sensitive information, there were two levels of authentication. The application is deployed on a public network, but the database resides in a private network. In order to get into the private network, we had to be on the VPN and use a key generated on a registered mobile device to log in.
Additionally sensitive data such as credit card numbers was encrypted with a frequently updated encryption key.
Exploratory testing to verify that authentication and encryption works as expected needed to be done. This apart from the standard security tests for APIs such as SQL injection into the payload.
Deploy. Rinse. Repeat. And don’t forget to backup.
The main reason the application is hosted on the cloud is that this is expected to be a highly scalable, highly available system, ensuring that when the number of customers increase, more instances can be deployed quickly to meet the additional demand. This means that apart from the usual functional testing, we had to pay particular attention to issues around data integrity, data backup and exception handling. Some examples are:
- Populate the database so an alert is received when the limit is reached. Once the database size is increased, verify that the alert is not received.
- Bring down the Master database - verify that the Slave comes up. Verify an alert is received.
- Verify that the Master and Slave are in sync, also do operations on the Master while Slave is down and vice versa
- Setup cron jobs in the QA environment to verify that the scheduler is doing its job of backing up the data at the specified time
- When the database backup file has reached its limit, verify that a new one is created. Verify that the backup can be applied on the database to return it to the last known stable state.
And so on.
The Fun Part
In most other projects hosted in traditional environments, getting a QA environment can be a bit of a struggle. Here we were able to cloudify our tests, and run any number of experiments without having to wait in the queue for allocation of environments. QAs basically went trigger happy - building and deploying instances at will. The not so fun part? Getting a frantic email from the customer “Would you please tear down the QA instances? We just got a huge bill from Amazon!” Oops.
Just the beginning...
Is our testing by any means complete? Did we cover most of the high risk scenarios? I suspect that the more we deploy applications on the Cloud, the more we as a Software Community will understand the specific challenges and risks in doing so. And only then will we be able to incorporate that learning into Quality.