Course Outline

Chapter 1: Fundamentals of Testing

Chapter 2: Testing Throughout the Software Development
Lifecycle
Chapter 3: Static Testing
Chapter 4: Test Techniques
Chapter 5: Test Management
Chapter 6: Tool Support for Testing
 Course Outline

Chapter 6: Tool Support for Testing

No. of Session: 01
Session 14: 6 - 6.2.3
 Chapter 6: Tool Support for Testing

Session 14
6.1 Test Tool Considerations
6.1.1 Test Tool Classification
6.1.2 Benefits and Risks of Test Automation
6.1.3 Special Considerations for Test Execution and Test
Management
 Chapter 6: Tool Support for Testing

Session 14 (continued)
6.2 Effective Use of Tools
6.2.1 Main Principles for Tool Selection
6.2.2 Pilot Projects for Introducing a Tool into an
Organization
6.2.3 Success Factors for Tools
 Chapter 6: Tool Support for Testing
Test Tool Considerations /1
There are a number of ways in which tools can be used to support
testing activities:
❖ To start with the most visible use, we can use tools directly in
testing. This includes test execution tools and test data
preparation tools.
❖ We can use tools to help us manage the testing process. This
includes tools that manage requirements, test cases, test
procedures, automate test scripts, test results, test data and
defects, as well as tools that assist with reporting and monitoring
test execution progress.
 Chapter 6: Tool Support for Testing
Test Tool Considerations /2
❖ We can use tools as part of exploration, investigation and
evaluation. For example, we can use tools to monitor file activity for
an application.
❖ We can use tools in a number of other ways, in the form of any tool
that aids in testing. This would include spreadsheets when used to
manage test assets or progress, or as a way to document manual
or automated tests.
 Chapter 6: Tool Support for Testing
Test Tool Classification /1
Main categories - based on the functionality
Tool support for
➢ Tool support for
management of testing and testware management of testing and testware
➢ Tool support for static testing
static testing
➢ Tool support for
test design and specification test design and specification

➢ Tool support for test execution and logging test execution and logging
➢ Tool support for performance performance measurement and
measurement and dynamic analysis dynamic analysis
➢ Tool support for specialized testing needs specialized testing needs
 Chapter 6: Tool Support for Testing
Test Tool Classification /2
Tool support for
❏ Tool support for management of testing
and testware ➢ management of testing and
➢ Test management tools testware
➢ Requirements management tools static testing
➢ Defect management tools
test design and specification
➢ Configuration management tools
➢ Continuous integration tools (D) test execution and logging

performance measurement and

dynamic analysis

specialized testing needs

 Chapter 6: Tool Support for Testing
Test Tool Classification /3 Tool support for
❏ Tool support for static testing
➢ Tools that support reviews management of testing and testware
➢ Static analysis tools (D) ➢ static testing
❏ Tool support for test design and
➢ test design and specification
specification
➢ Test design tools test execution and logging

➢ Model-based testing tools performance measurement and

➢ Test data preparation tools dynamic analysis

➢ Test-driven development (TDD) specialized testing needs

tools (D)
➢ Acceptance test-driven development (ATDD) and
behaviour-driven development (BDD) tools
 Chapter 6: Tool Support for Testing
Test Tool Classification /4 Tool support for

❏ Tool support for test execution and logging

management of testing and testware
➢ Test execution tools
➢ Coverage tools static testing

➔ Requirements coverage test design and specification

➔ Code coverage (D) ➢ test execution and logging
➢ Test harnesses (D)
performance measurement and
➢ Unit test framework tools (D) dynamic analysis

specialized testing needs

 Chapter 6: Tool Support for Testing
Test Tool Classification /5
❏ Tool support for performance measurement and dynamic analysis
➢ Performance testing tools Tool support for
➢ Monitoring tools
management of testing and testware
➢ Dynamic analysis tools (D)
❏ Tool support for specialized testing needs static testing
➢ Data quality assessment test design and specification
➢ Data conversion and migration
test execution and logging
➢ Usability testing
➢ Accessibility testing ➢ performance measurement
and dynamic analysis

➢ specialized testing needs

 Chapter 6: Tool Support for Testing
Test Tool Classification /6
❏ Tool support for specialized testing needs (continued)
➢ Localization testing Tool support for
➢ Security testing
management of testing and testware
➢ Portability testing
static testing

test design and specification

test execution and logging

performance measurement and

dynamic analysis

➢ specialized testing needs

 Chapter 6: Tool Support for Testing
Benefits and Risks of Test Automation /1

Benefits:
There are many benefits that can be gained by using tools to support
testing, whatever the specific type of tool. Benefits include:

❏ Reduction in repetitive manual work.

❏ Greater consistency and repeatability.
❏ More objective assessment.
❏ Easier access to information about tests or testing.
 Chapter 6: Tool Support for Testing
Benefits and Risks of Test Automation /2

Risks:
There are many risks that are present when tool support for testing is
introduced and used, whatever the specific type of tool. Risks include:

❏ Unrealistic expectations for the tool.

❏ Underestimating the time, cost and effort for the initial introduction
of a tool.
 Chapter 6: Tool Support for Testing
Benefits and Risks of Test Automation /3

Risks (continued):

❏ Underestimating the time and effort needed to achieve significant

and continuing benefits from the tool.
❏ Underestimating the effort required to maintain the test assets
generated by the tool.
❏ Over-reliance on the tool.
 Chapter 6: Tool Support for Testing
Special Considerations for Test Execution and Test Management

Test execution tools /1

There are different levels of scripting. Five are described in [Fewster
and Graham, 1999]:

➔ Linear scripts, which could be created manually or captured by

recording a manual test.
➔ Structured scripts, using selection and iteration programming
structures.
 Chapter 6: Tool Support for Testing
Special Considerations for Test Execution and Test Management
Test execution tools /2
➔ Shared scripts, where a script can be called by other scripts so can
be reused: shared scripts also require a formal script library under
configuration management.
➔ Data-driven scripts, where test data is in a file or spreadsheet to be
read by a control script.
➔ Keyword-driven scripts, where all of the information about the test
is stored in a file or spreadsheet, with a single control script that
implements the tests described in the file using shared and
keyword scripts.
 Chapter 6: Tool Support for Testing
Special Considerations for Test Execution and Test Management

Test management tools

Test management tools need to interface with other tools (including
spreadsheets for example) for various reasons, including:
➔ To produce useful information in a format that fits the needs of the
organization.
➔ To maintain consistent traceability to requirements in a
requirements management tool.
➔ To link with test object version information in the configuration
management tool.
 Chapter 6: Tool Support for Testing
❏ Effective Use of Tools
⇛ Main principles for tool selection
⇛ Pilot project
⇛ Success factors for tools
 Chapter 6: Tool Support for Testing
Effective Use of Tools: Main Principles for Tool Selection /1
The following factors are important in selecting a tool:
● Assessment of the organization's maturity (e.g. readiness for
change).
● Identification of the areas within the organization where tool
support will help to improve testing processes.
● Understanding the technologies used by the test object(s), so that
a tool will be selected that is compatible with those technologies.
● Knowledge of any build and continuous integration tools already
being used within the organization, to make sure that the new
tool(s) will integrate with them and be compatible.
 Chapter 6: Tool Support for Testing
Effective Use of Tools: Main Principles for Tool Selection /2
● Evaluation of tools against clear requirements and objective
criteria.
● Consideration of any free trial period for the tool (for commercial
tools) to ensure that this gives adequate time to evaluate the tool.
● Evaluation of the vendor (including training, support and other
commercial aspects) or support for non-commercial tools (open
source).
● Identification of internal requirements for coaching and mentoring
in the use of the tool.
 Chapter 6: Tool Support for Testing
Effective Use of Tools: Main Principles for Tool Selection /3
● Evaluation of training needs for those who will use the tools directly
and indirectly (for example without technical detail), taking into
account testing skills and test automation skills (for those working
directly with the tools).
● Consideration of pros and cons of different licencing models (for
example commercial or open source).
● Estimation of a cost-benefit ratio based on a concrete and realistic
business case (if required).
 Chapter 6: Tool Support for Testing
Effective Use of Tools: Pilot Projects for Introducing a Tool into an
Organization /1 Tool selection
The objectives for a pilot project for a new tool are:
● To learn more about the tool Successful
➔ more detail proof-of-concept
➔ more depth
● To see how the tool would fit with existing processes Pilot project
or documentation, how those would need to change to
work well with the tool and how to use the tool to streamline
existing processes.
 Chapter 6: Tool Support for Testing
Effective Use of Tools: Pilot Projects for Introducing a Tool into an
Organization /2
● To decide on standard ways of using the tool that will work for all
potential users (e.g. naming conventions, creation of libraries,
defining modularity, where different elements will be stored, how
they and the tool itself will be maintained).
● To evaluate the pilot project against its objectives (have the
benefits been achieved at reasonable cost?)
 Chapter 6: Tool Support for Testing
Effective Use of Tools: Pilot Projects for Introducing a Tool into an
Organization /3

● To understand (and experiment with) metrics that you want the

tool(s) to collect and to report, and configuring the tool(s) to ensure
that your goals for these metrics can be achieved.
 Chapter 6: Tool Support for Testing
Effective Use of Tools: Success Factors for Tools /1

Here are some of the factors that have contributed to success:

➢ incremental roll-out (after the pilot) to the rest of the organization.
➢ adapting and improving processes, testware and tool artefacts to
get the best fit and balance between them and the use of the tool.
➢ providing adequate training, coaching and mentoring of new users.
 Chapter 6: Tool Support for Testing
Effective Use of Tools: Success Factors for Tools /2

➢ defining and communicating guidelines for the use of the tool,

based on what was learned in the pilot.
➢ implementing a continuous improvement mechanism as tool use
spreads through more of the organization.
➢ monitoring the use of the tool and the benefits achieved and
adapting the use of the tool to take account of what is learned.
Thank you!

⇰Happy learning..
⇉Learn more...