Test Automation Tools Selections
Traditional Test Automation vs. Manual Testing
The most significant benefits of traditional test automation are the decreased test execution time and cost, and increased test coverage for each testing cycle. The downside of test automation is the increased test development cost as on quality assurance. In a typical test automation scheme, each test is first documented by a test engineer then scripted by an automation engineer. The cost benefit of automated testing comes through the repeated execution of the test automation suite. Since the cost of executing an automated test is far less than the cost of executing a manual test, eventually the decreased test execution cost will make up for the increased development costs.
For example, suppose that a test engineer costs your organization $60/hr, and an automation engineer costs your organization $100/hr. The test engineer can create 5 new test cases per hour during test plan development, and can execute 10 test cases per hour during test execution. The automation engineer can develop 2 test cases per hour. Once those test cases are completed, a test engineer can execute and analyze the results of 50 tests in one hour (typically the test engineer will run the tests overnight, or while he is doing some other work, then analyze the results once the test is done).
Suppose you create a test plan with 1000 test cases:
- For manual testing, your initial test development cost is: 1000 tests * 1 hour/5 test cases * $60/hour = $12,000
- The cost of executing all of the tests manually is: 1000 tests * 1 hour/10 test cases * $60/hour = $6,000
- For automated testing, your initial cost is: $12,000 + 1000 tests * 1 hour/2 test cases * $100/hour = $65,000
- The cost of executing all of the test cases w/ test automation is: 1000 tests * 1 hour/50 test cases * $60/hour = $600
The cost benefit of test automation is realized by running the tests several times (i.e. with each weekly or nightly build of your application).
In this case, we begin to see a cost savings if we run the tests more than 12 times.
There are several reasons why people choose to automate tests:
- Speed up testing to accelerate releases
- Allow testing to happen more frequently
- Reduce costs of testing by reducing manual labor
- Improve test coverage
- Ensure consistency
- Improve the reliability of testing
- Allow testing to be done by staff with less skill
- Define the testing process and reduce dependence on the few who know it
- Make testing more interesting
- Develop programming skills