System Test

By: Dung Nguyen Date: June 10, 2007

Contents
Introduction System Test Overview System Test Plan Content System Test Cases Execution of System Test Cases Reporting Test Results Regression Test Cases Appendix I - Example Test Plan Appendix II - Example Test Execution References

Feb. 04, 2002

Confidential

INTRODUCTION
Purpose and Scope of this document: This document provides information on the requirements of the system test phase and its deliverables. The document is intended to be used as a guide for the preparation and execution of system testing.

Feb. 04, 2002

Confidential

System Test Overview

System Test Overview - Objectives

System Test is responsible for: Developing the System Test Plan and test cases Executing the testing based on the test cases developed Generating the test results and reporting weekly, monthly, or by release depending upon the need Generating and closing problem reports The objective of System Test is to test the system capabilities and behaviour of the product on different types of platforms on which the product will be available.

Feb. 04, 2002

Confidential

System Test Overview - Scope of System Test

The System Test test areas are usually: Feature and Feature Interaction Tests, Performance and Memory Measurement, Capacity Limits Tests, Load and Longevity Tests, System Recovery and Robustness, System Security Tests, System Operations (including new Installation, Software Conversion, and System Expansion), and End-User Documentation.

Feb. 04, 2002

Confidential

System Test Overview - Scope of System Test (Cont)

Feature and Feature Interaction Tests: These test verify on functionality and features that are the product release Performance and Memory Measurement: These tests measure the system performance, memory usage, and various data throughput rates to compare these against those specified in the product specification and engineering guidelines. Capacity Limits Tests: These tests focus on system behavior when the system reaches its capacity limits as defined in the product specification.
Feb. 04, 2002 Confidential 7

System Test Overview - Scope of System Test (Cont)

Load and Longevity Tests:
Load tests will be run to check service standards related to service availability, lost message rate, damaged message rate, dropped calls, etc., as defined in the product specification. Longevity tests test the system behavior over time when subjected to a specified load

System Recovery and Robustness: These tests will start with inserting both software and hardware faults into the system and observing:
How the system detects faults, How the system recovers when faults are detected, How robust the system is in general, and How well faults are reported.
Feb. 04, 2002 Confidential 8

System Test Overview - Scope of System Test (Cont)

System Security Tests : These tests will focus on the security of the system from all kinds of terminals as:
Screen based, telset, Access (remote, local), Client/server, and Different kinds of users, operators, and administrators.

A key goal of these tests is to ensure that system operation functions do not violate system security.

Feb. 04, 2002

Confidential

System Test Overview - Scope of System Test (Cont)

System Operations: System Operations for the product refer to the system configuration utilities provided for the product, including:
Installation, Expansion, Conversion, and A number of less general utilities.

Testing will cover all operations on a variety of hardware platforms

Feb. 04, 2002

Confidential

10

System Test Overview - Scope of System Test (Cont)

End-User Documentation: End-user documentation procedural testing and review is part of the function of System Test. This can be achieved efficiently by using end-user documentation when testing the various system functions.

Feb. 04, 2002

Confidential

11

System Test Plan Content

System Test Plan: Responsibility

The system test plan is the responsibility of the System Test group

Feb. 04, 2002

Confidential

13

System Test Plan: Major Sections

The System Test Plan must minimally include the following sections: Executive Summary Product Overview Schedule Acceptance Criteria Test Coverage Test Environment Needs Assumptions, Dependencies, and Risks Reporting Glossary References
Feb. 04, 2002 Confidential 14

System Test Plan - Executive summary

The executive summary provides an overview of the purpose of this document. Key points outlined here include:
Resources, Schedule time-frames, Assumptions, Dependencies, and Risks.

Feb. 04, 2002

Confidential

15

System Test Plan - Product Overview

The product overview provides a brief description of the services offered in the product as well as the platforms and connectivity of the product. If the product is a new release of an existing product, discussion also describes new release capabilities as well as existing base release capabilities.

Feb. 04, 2002

Confidential

16

System Test Plan - Schedule

This section describes
System test milestones and Detailing the resource (staffing) requirements before, during, and after the system test phase.

A detailed schedule is presented here for the actual testing phase including
Personnel, platforms, and type of testing

during the planned actual testing period.

Feb. 04, 2002

Confidential

17

System Test Plan - Acceptance Criteria

The acceptance criteria provide details in the following terms: Entrance criteria, Exit criteria, Suspension criteria, and Deliverables criteria to System Test.
Note: The use of entrance and exit criteria is to ensure that those responsible for the preceding integration test phase and subsequent product verification phases are aware of the quality that is expected and will be delivered in the product.

Feb. 04, 2002

Confidential

18

System Test Plan - Test Coverage

This section provides a description of the testing that will be performed on the product for each of the system testing test areas. Test areas might consist of the following: Features and Feature Interactions Performance and Memory Measurements Capacity Limits Load and Longevity System Recovery and Robustness System Security System Operations End-User Documentation

Feb. 04, 2002

Confidential

19

System Test Plan - Test Environment Needs

This section describes the hardware, and software required during the system test phase (including all test tools required to perform system testing).

Feb. 04, 2002

Confidential

20

System Test Plan - Assumptions, Dependencies, Risks

Assumptions, Dependencies, and Risks:
Assumptions are things that are outside the scope of the project but that must happen for the system integration test cycle to be successful. Dependencies are things that are inside the scope of the project over which integration has no control. Risks are statements about things for which the integration prime can plan contingencies. There must be at least one contingency for each risk.

Feb. 04, 2002

Confidential

21

System Test Plan - Glossary

Provide a detailed glossary of the terms and acronyms used within the system test plan document

Feb. 04, 2002

Confidential

22

System Test Plan - References

Provide a reference to other documents to which the system test plan relates

Feb. 04, 2002

Confidential

23

System Test Plan: Review

The test plan is reviewed subsequently by: System Test Group, and Product Verification

Feb. 04, 2002

Confidential

24

System Test Plan: Modification

All changes to the test plan are then reviewed.

Feb. 04, 2002

Confidential

25

System Test Plan: Archival

The reviewed system test plan is archived as described in the Product or Project Development Plan and follows version control.

Feb. 04, 2002

Confidential

26

System Test Case

System Test Cases: Coverage

System Test cases focus upon testing the system capabilities of the product as well as individual features with the goal of ensuring that the system conforms with the product and feature specifications. System Test test cases are intended to test:
All new functionality on at least one applicable platform and Regression test existing functionality.

Test cases will be traced to the feature specification requirements, to ensure proper requirement coverage.

Feb. 04, 2002

Confidential

28

System Test Cases: Creation

Test cases for System Test are developed from :
Product and Feature Specifications.

Development of test cases can proceed after, and only after:

The Product Specification and Related Feature Specifications

have been reviewed and signed off.

Note: Design Specifications may be used if the Product and Feature Specifications do not contain all necessary information.

Feb. 04, 2002

Confidential

29

System Test Cases: Creation (Cont.)

In cases where free-form test cases (i.e. test cases executed by a tester for which no written test case exists) result in the detection of a problem, the Test Prime may decide to formalize the freeform test case into a written one. The decision on whether to do this will be made at the Process Post-Mortem meeting.
Note: In order to facilitate the formalization of a free-form test case, any problem report resulting from the test case must describe all the steps leading to the product defect.

Feb. 04, 2002

Confidential

30

System Test Cases: Format

Appendix II Section 7.2 provides a recommended form for writing test cases.

Feb. 04, 2002

Confidential

31

System Test Cases: Automated

Wherever practical, automated test cases are developed for the system testing phase. Certain test areas, such as capacity limits, load testing, and performance measurements, lend well to automation, Whereas other test areas, such as system operations and system security testing require a high degree of human (tester) interaction

Feb. 04, 2002

Confidential

32

System Test Cases: Review

Group consisting of members from System Test, Design, and Product Verification is responsible for reviewing
a particular test area or test item (feature),

Feb. 04, 2002

Confidential

33

System Test Cases: Modification

Test cases may be added, deleted, or modified. New and modified test cases are subject to review

Feb. 04, 2002

Confidential

34

System Test Cases: Archival

All reviewed system test cases are archived and follow version control

Feb. 04, 2002

Confidential

35

System Test Cases: Execution

Test cases are executed on the target platform during the system testing phase. Prior to the beginning of test case execution, the product must meet the System Test Entrance Criteria as outlined in the System Test Plan. System Test will execute test cases only on approved releases of the product.

Feb. 04, 2002

Confidential

36

System Test Results

System Test Results: Creation

System test results must provide the following information:
Name of tester(s). Software Release. Date and time of test execution. Pass, Fail, or Inconclusive indication. Problem Report number (if applicable).

Feb. 04, 2002

Confidential

38

System Test Results: Review

The test results are reviewed by System Team and Product Verification group

Feb. 04, 2002

Confidential

39

System Test Results: Modification

System rest results are not modified.

Feb. 04, 2002

Confidential

40

System Test Results: Archival

System test results are stored in the Information Resources Centre.

Feb. 04, 2002

Confidential

41

Regression Test Cases

Regression Test Cases - Purpose

Regression testing as the name implies is used to determine whether changes or additions to software or hardware have caused degradation in the quality of existing functionality.
Regression tests therefore, can never be run on an initial release of software or hardware.

Regression test cases are chosen from the full suite of test cases which were written for the entity under test. Once the regression test cases have been executed, the number of test cases which failed provides an indication of whether more extensive testing needs to be done.

Feb. 04, 2002

Confidential

43

Regression Test Cases Purpose (Cont.)

The system test plan must identify the required pass rate.
Note: Regression test suites may only be run on subsystems and on the full system.

Feb. 04, 2002

Confidential

44

Regression Test Cases - Planning

The System Test Team is responsible for determining the amount of regression testing required.

Feb. 04, 2002

Confidential

45

Regression Test Cases - Selecting

The regression test suite is chosen from the full suite of integration test cases for the entity under test. Each regression test case chosen must have already been executed successfully. The ideal regression test case is one which exercises major functionality. The frequency, with which regression test cases are executed, makes regression test cases ideal candidates for automation.

Feb. 04, 2002

Confidential

46

Regression Test Cases - Executing

In the event that a software defect has to be corrected and a new release is made, the full suite system test cases must be run, followed by the regression test suite If the software changes are minor, a partial regression may be executed.
Note:
However, if the number or extent of the software changes are significant, then either a full regression or the complete integration test suite for the subsystem(s) affected will be executed. If the entire system is similarly affected, then the complete integration test suite is executed for the system.

This determination is made by the System Test Team in conjunction with the Product Verification

Feb. 04, 2002

Confidential

47

References
[1] Boris Beizer, Software Systems Testing and Quality Assurance, 1984. [2] Boris Beizer, Software Testing Techniques, 1990. [3] Ross Collard, Systems Testing and Quality Assurance Techniques, Volumes I & II, May 26th, 1994. [4] Yuk-Wha Li, Meridian Mail Products - System Test Responsibilities Issue 1.0, January

Feb. 04, 2002

Confidential

48