Business needs, Business solutions
Software Testing has found its place in the software industry, with more and more organizations understanding the crucial role that it plays in quality software production. As business requirements grow, so does the pressure on IT organizations to deliver more products with fewer resources, in reduced time and with high quality.
This scenario emphasizes the need of:
A phased approach, keeping in mind different contributing factors, would provide a solution to this problem.
This approach would address:
This approach does not attempt to replace any of the industry's standard test automation tools. It only suggests the process for effective test automation framework development using any tool, by overcoming tool limitations and accelerating its productivity.
In today's business environment, project teams are expected to do more and deliver higher quality systems in lesser time with fewer resources. And when companies tighten their budgetary belt, software testing is often one of the first systems-development item to be done away with.
IT systems that don't solve real business problems or don't perform as promised, impose a similar economic toll on business costs and results. More than half of all software projects fail to meet objectives or suffer significant schedule and budget slippage because defects are discovered too late. All these factors combined result in a high percentage of "Defect Leakage" in the production line, resulting in poor customer satisfaction and less ROI from the product.
What needs to be done?
All the above listed elements can be addressed by a time-to-market solution.
"Test Automation - Build a Test Automation Framework"
This paper provides details on building a Test Automation Framework using a systematic approach, which walks through a 10 different process stages to be followed in order to reap key-benefits.
Key factors required for a Test Automation to be successful include:
Dedicated Budget A dedicated budget to be allocated, which includes costs related to test tool, development, deployment, resource and training. In addition, the maintenance cost for automated tests and tools must be included.
Well-defined Testing Process
Dedicated Resources A dedicated team is needed for effective test automation. Non-dedicated team will execute the test automation with their own limitations, which will lead to:
Management and the project team must be keep realistic expectations and should keep them in mind during the entire test automation life-cycle.
A framework defines the organization's way of doing things - a 'Single Standard'. Following this standard would result in the project team achieving:
Test automation framework development is a multi-stage process. And passing through each stage involves multiple challenges to be addressed. Key challenges to be addressed are detailed below:
Clear vision of what needs to be achieved out of this automation must be defined and documented. To formulate the clear vision, the following needs to be identified:
Tool identification process is a crucial one, as it involves critical factors to be considered, which include:
Framework design involves identifying requirements from multiple areas. At a high level, this includes (not limited to):
Each organization believes in its own requirements in software test automation. Considering the organization's requirements, test automation activities can be performed with three different scopes:
Subsequent to the testing scope identification, product/application/modules under the testing scope need to be identified. Based on the product/application/module requirement, types of testing that need to be performed are identified. For example, em Scenario: For an 'Enterprise-oriented' testing scope, product A would require a functional testing, product B would require a web-service testing, a product C would require a performance testing and also complete project management etc.......
Priority must be assigned to each type of testing, based on the schedule for product release.
Testing requirements and their nature is studied for the product/application/modules. Each requirement has its own actions, validations for testing. For example,
Scenario 1: For an application, form validation functionalities, database validation and accessibility functionalities needs to be validated.
Scenario 2:For an application, all the web-service methods needs to be validated. This would also include the delay time for the request's reponse from third party systems
All the identified requirements are assigned priority. This would help in identifying 'Build-Verification Test' (BVT) requirements that should never fail.
Identified testing types and requirements, acts as a base criterion for test automation tool evaluation.
Checklist - An exhaustive evaluation checklist needs to be created which is in-line with the requirements and the tool is evaluated against this checklist for positive results. Checklist needs to cover (not limited to):
Identify Tools - Next task would be identifying different industry-standard tools based on the requirements. This can also include tools to be purchased or open source tools. Identified tools must be evaluated against the checklist.
Sample Run - Tools claim that the tool supports specific requirements, but finally when we try creating our scripts it fails. Best way to evaluate is to create a sample run, this includes different types of actions and requirements we need to perform. Create sample scripts and execute the same for results.
Rate and Select Tools - Based on the sample run, supportive tools could be identified and rated. Also there may be scenarios, where multiple tools satisfy the requirements. In the said scenario, we may need to choose more than one tool, for test automation.
Implementation and Training - High rated tools will be procured/open-source licensed. Training needs to be conducted for the project team, on how to use these tools.
Every tool has its own limitations. A feasibility study needs to be conducted for the requirements against the tools. This study would result in listing requirements that can be automated. Also based on the nature of the requirements, automation feasibility needs to be identified.
Generally, testers start creating test scripts based on the scenarios. This includes multiple actions to be performed against each objects. This approach leads to an ad-hoc test script creation and duplicate testing effort, i.e. testers, would create test scripts for a single action in different scenarios.
Our approach takes a different path as explained below. For designing a framework, various elements need to be taken into consideration. Utilities/Components (re-usable) would be designed for the following elements that include (not limited to):
Test automation framework would be designed based on the listed factors, using the following guidelines.
Types of input data files supported by the tools, needs to be identified. Based on the requirements, input files can be categorized as (not limited to),
For all the files types, file format needs to be identified and prototyped based on the input data storage.
Framework development is facilitated using the same set of identified tools. Scripting language supported by the test automation tool is used to create the components. Tool extensibility utility/component can be developed using a different language. Utility functions/components created based on framework design is explained in Step 7.
In addition to the re-usable components driver scripts and worker scripts needs to be created.
Approaches for developing re-usable utilities/components include:
Input data store needs to be populated based on the file structure defined in Step 6. Data can be populated either manually or in an automated fashion from different data-sources. Test data would be populated based on parent-child hierarchy. For example,
"A transaction would be a parent hierarchy and Input to the test cases would be the child"
Scheduler requirement needs to be identified. Schedulers can be configured to run a worker script (batch script) on a specific time period. This approach benefits in a way that even a business user can configure the scheduler and make the test execution happen.
- Test automation processes, a single standard is established across the organization. This helps the organization as they follow the standard processes as compared to pre-empted ad-hoc processes, which yield no results.
- Complete coding and component usage standards are defined in production. Organization benefits include:
- Requirements are collected from an overall organization's perspective (for eg: Product suite on multiple technologies .net and java etc). Overall coverage of re-usable components which includes (data communications, system communications, schedulers, loggers, reporters etc)
This overall coverage minimizes the testing effort during the later stages of the releases, for the entire product suite across the organization.
- Organizations need not worry about testing future enhancements. Only the validations related to the enhancements need to be added to the existing base framework, and that too with minimal effort.
- At the end of Step 6, the complete cost for the framework development can be estimated. This cost includes,
Having seen the benefits, the importance of testing and test automation will be increasingly realized across domains and to accelerate the process. "Test Automation Framework" would find greater acceptance and importance in the industry.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.