Overview
Unit testing
Code Review
Integration testing
Big bang
Bottom up
Top down
Sandwich
Continuous
System testing
Functional
Performance
Acceptance testing
Summary
Unit Testing
Code Review
Code walkthrough
Code inspection
Integration Testing
The entire system is viewed as a collection
of subsystems (sets of classes) determined
during the system and object design
Goal: Test all interfaces between subsystems
and the interaction of subsystems
The Integration testing strategy determines
the order in which the subsystems are
selected for testing and integration.
Why do we do integration testing?
Unit tests only test the unit in isolation
Many failures result from faults in the interaction of subsystems
Often many Off-the-shelf components are used that cannot be unit
tested
Without integration testing the system test will be very time
consuming
Failures that are not discovered in integration testing will be
discovered after the system is deployed and can be very expensive.
Stubs and drivers
Driver:
A component, that calls the
TestedUnit
Controls the test cases
Driver
Tested
Unit
Stub:
A component, the TestedUnit
depends on
Partial implementation
Returns fake values.
Stub
Example: A 3-Layer-Design
A
Spread
A
SheetView
B
Entity
Data
B
Model
BinaryFile
E
Storage
Calculator
C
XMLFile
F
Storage
Layer I
D
Currency
D
Converter
Layer II
Currency
G
DataBase
Layer III
Big-Bang Approach
B
Test A
Test B
E
Test C
Test D
Test E
Test F
Test G
Test
A, B, C, D,
E, F, G
Bottom-up Testing Strategy
The subsystems in the lowest layer of the call
hierarchy are tested individually
Then the next subsystems are tested that call
the previously tested subsystems
This is repeated until all subsystems are
included
Drivers are needed.
Bottom-up Integration
Test E
Test B, E, F
Test F
Test C
Test G
Test D,G
Test
A, B, C, D,
E, F, G
Pros and Cons of Bottom-Up
Integration Testing
Con:
Tests the most important subsystem (user interface)
last
Drivers needed
Pro
No stubs needed
Useful for integration testing of the following systems
Object-oriented systems
Real-time systems
Systems with strict performance requirements.
Top-down Testing Strategy
Test the top layer or the controlling
subsystem first
Then combine all the subsystems that are
called by the tested subsystems and test
the resulting collection of subsystems
Do this until all subsystems are
incorporated into the test
Stubs are needed to do the testing.
Top-down Integration
B
Test A
Test A, B, C, D
Test
A, B, C, D,
E, F, G
Layer I
Layer I + II
All Layers
Pros and Cons of Top-down
Integration Testing
Pro
Test cases can be defined in terms of the
functionality of the system (functional
requirements)
No drivers needed
Cons
Writing stubs is difficult: Stubs must allow all
possible conditions to be tested.
Large number of stubs may be required,
especially if the lowest level of the system
contains many methods.
Some interfaces are not tested separately.
Sandwich Testing Strategy
Combines top-down strategy with
bottom-up strategy
The system is viewed as having three
layers
A target layer in the middle
A layer above the target
A layer below the target
Testing converges at the target layer.
Sandwich Testing Strategy
B
Test A
E
Test E
Test A,B,C, D
Test B, E, F
Test F
Test D,G
Test G
Test
A, B, C, D,
E, F, G
Pros and Cons of Sandwich Testing
Top and Bottom Layer Tests can be done
in parallel
Problem: Does not test the individual
subsystems and their interfaces
thoroughly before integration
Solution: Modified sandwich testing
strategy
Modified Sandwich Testing Strategy
Test in parallel:
Middle layer with drivers and stubs
Top layer with stubs
Bottom layer with drivers
Test in parallel:
Top layer accessing middle layer (top layer replaces
drivers)
Bottom accessed by middle layer (bottom layer
replaces stubs).
Modified Sandwich Testing
B
Test A
Test A,C
Test C
Test B
Test E
Test B, E, F
Test F
Test D
Test D,G
Test G
Test
A, B, C, D,
E, F, G
Continuous Testing
Continuous build:
Build from day one
Test from day one
Integrate from day one
System is always runnable
Requires integrated tool support:
Continuous build server
Automated tests with high coverage
Tool supported refactoring
Software configuration management
Issue tracking.
Continuous Testing Strategy
A
Spread
SheetView
B
Data
Model
BinaryFile
Storage
Sheet View
Calculator
Layer I
D
Currency
Converter
Layer II
Currency
DataBase
XMLFile
Storage
+ Cells
+ Addition
Layer III
+ File Storage
Steps in Integration Testing
1. Based on the integration strategy,
select a component to be tested.
Unit test all the classes in the
component.
2. Put selected component together;
do any preliminary fix-up
necessary to make the
integration test operational
(drivers, stubs)
3. Test functional requirements:
Define test cases that exercise all
uses cases with the selected
component
4. Test subsystem decomposition:
Define test cases that exercise all
dependencies
5. Test non-functional requirements:
Execute performance tests
6. Keep records of the test cases and
testing activities.
7. Repeat steps 1 to 7 until the full
system is tested.
The primary goal of integration testing
is to identify failures with the
(current) component configuration.
System Testing
Functional Testing
Validates functional requirements
Performance Testing
Validates non-functional requirements
Acceptance Testing
Validates clients expectations
Functional Testing
Goal: Test functionality of system
Test cases are designed from the
requirements analysis document (better:
user manual) and centered around
requirements and key functions (use
.
cases)
The system is treated as black box
Unit test cases can be reused, but new
test cases have to be developed as well.
Performance Testing
Goal: Try to violate non-functional requirements
Test how the system behaves when overloaded.
Can bottlenecks be identified? (First candidates for redesign in
the next iteration)
Try unusual orders of execution
Call a receive() before send()
Check the systems response to large volumes of data
If the system is supposed to handle 1000 items, try it with 1001
items.
What is the amount of time spent in different use cases?
Are typical cases executed in a timely fashion?
Types of Performance Testing
Stress Testing
Try to violate security
requirements
Stress limits of system
Volume testing
Test what happens if large amounts of
data are handled
Quality testing
Test reliability, maintain- ability &
availability
Recovery testing
Test systems response to
presence of errors or loss of data
Timing testing
Evaluate response times and time to
perform a function
Environmental test
Test tolerances for heat, humidity,
motion
Compatibility test
Test backward compatibility with existing
systems
Configuration testing
Test the various software and hardware
configurations
Security testing
Human factors testing
Test with end users.
Acceptance Testing
Goal: Demonstrate system is ready
for operational use
Client uses the software at the
developers environment.
Choice of tests is made by client
Software used in a controlled
setting, with the developer
always ready to fix bugs.
Many tests can be taken from
integration testing
Acceptance test is performed by
the client, not by the developer.
Alpha test:
Beta test:
Conducted at clients
environment (developer is not
present)
Software gets a realistic workout
in target environ- ment
Testing has many activities
Establish the test objectives
Design the test cases
Write the test cases
Test the test cases
Execute the tests
Evaluate the test results
Change the system
Do regression testing
Test Team
Professional
Tester
Programmer
Analyst
User
Test
Team
Configuration
Management
Specialist
System
Designer
too familiar
with code
The 4 Testing Steps
1. Select what has to be tested
Analysis: Completeness of
requirements
Design: Cohesion
Implementation: Source code
2. Decide how the testing is done
Review or code inspection
Proofs (Design by Contract)
Black-box, white box,
Select integration testing strategy
(big bang, bottom up, top down,
sandwich)
3. Develop test cases
A test case is a set of test data or
situations that will be used to
exercise the unit (class, subsystem,
system) being tested or about the
attribute being measured
4. Create the test oracle
An oracle contains the predicted
results for a set of test cases
The test oracle has to be written
down before the actual testing takes
place.
Summary
Testing is still a black art, but many rules and
heuristics are available
Testing consists of
Unit testing
Integration testing
System testing
Acceptance testing
Design patterns can be used for integration
testing
Testing has its own lifecycle
Test Report
Completed copy of each test script with
evidence that it was executed (i.e., dated with
the signature of the person who ran the test)
Copy of each SPR showing resolution
List of open or unresolved SPRs
Identification of SPRs found in each baseline
along with total number of SPRs in each
baseline
Regression tests executed for each software
baseline
Bug Fixing
A bug fix is a change to a system or
product designed to handle a
programming bug/glitch. Many different
types of programming bugs that create
errors with system implementation may
require specific bug fixes that are
successfully resolved by a development or
other IT team.
A bug fix is also known as a program
temporary fix (PTF).
Common Types of Bugs
Arithmetic bugs
- Division by zero
- Arithmetic overflow or underflow.
- Loss of arithmetic precision due to rounding or numerically
unstable algorithms
Logic bugs
- Infinite loops and infinite recursion
- Off by one error, counting one too many or too few when
looping
Syntax bugs
- Use of the wrong operator, such as performing assignment
instead
of equality test
Common Types of Bugs
Resource bugs
- Null pointer dereference.
- Using an uninitialized variable.
- Using an otherwise valid instruction on the
wrong data type
- Access violations.
- Buffer overflow, in which a program tries
to
store data past the end of allocated
storage.
This may or may not lead to an access
violation
or storage violation.
Common Types of Bugs
Interfacing bugs
-
Incorrect
Incorrect
Incorrect
Incorrect
API usage.
protocol implementation.
hardware handling.
assumptions of a particular platform.
Performance bugs
- Too high computational complexity of algorithm.
- Random disk or memory access.
Difference between Error, Bug and Defect
Error: It is deviation from logic, syntax or execution.
There are 3 types of errors
syntactical error: it is deviation from syntax of program
logical error: it is deviation from logic of the program.
Executional error: this occurs during executing the
program.
Defect: when the error is discovered by the tester then it
is called deffect.
Bug:if the developer accepts the deffect then it is called
bug.
Stress testing
Stress testing (sometimes called torture testing) is a form of
deliberately intense or thorough testing used to determine
the stability of a given system or entity. It involves testing
beyond normal operational capacity, often to a breaking
point, in order to observe the results.
Reasons can include:
- to determine breaking points or safe usage limits
- to confirm intended specifications are being met
- to determine modes of failure (how exactly a system fails)
- to test stable operation of a part or system outside
standard usage
Performance Testing
Performance testing is the assessment of very different
sets of functional skills or abilities. Performance testing is
extremely common in the computer hardware and
software industry.
If a business fails to performance test its system, the
system may run slowly or even crash when subjected to
high-volume traffic and peak-load data quantities. Such
problems can result in:
- Loss of revenue
- Loss of customers
- Underuse of expensive company systems
- A backlog customer orders
- Negative publicity from the media and blog sites
Test automation
In software testing, test automation is the use of
special software to control the execution of tests
and the comparison of actual outcomes with
predicted outcomes.
Test automation can automate some repetitive
but necessary tasks in a formalized testing
process already in place, or add additional
testing that would be difficult to perform manually.
Test automation interface
Test automation interface are platforms that provide a
single workspace for incorporating multiple testing tools
and frameworks for System/Integration testing of
application under test.
Test automation interface are expected to improve the
efficiency and flexibility of maintaining test scripts.
Test Automation Interface consists of the following core
modules:
- Interface Engine
- Interface Environment
- Object Repository
Test automation interface
Interface engine
Interface engines are built on top of Interface Environment.
Interface engine consists of a parser and a test runner.
Interface environment
Interface environment consists of Product/Project Library and
Framework Library. Framework Library have modules related with
the overall test suite while the Product/Project Library have
modules specific to the application under test.
Object repository
Object repositories are a collection of UI/Application object data
recorded by the testing tool while exploring the application under
test.