Introduction to Software

Testing
Module 1
Introduction
Module - 1 (Introduction to Software Testing)
Some Popular Errors – Ariane 5, Therac 25, Intel Pentium Bug. What is
Software testing? Why should it be tested? Software Quality, Role of
Testing. Testing Process - Level 0 thinking, Level 1 thinking, Level 2
thinking, Level 3 thinking, Level 4 thinking. Software Testing
Terminologies - Verification, Validation and Testing, Faults, Error and
Bug, Test cases, Coverage Criteria. Types of Testing- Unit testing,
integration testing, System testing, Acceptance testing, Beta testing,
Functional testing, Stress testing, Performance testing, Usability
testing, and Regression testing. Testing Methods - Black Box testing,
WhiteBox testing, Grey Box testing.

2
 Testing in the 21st Century
• Software defines behavior
• network routers, finance, switching networks, other infrastructure
• Today’s software market :
• is much bigger Industry is going through
• is more competitive a revolution in what
• has more users testing means to the
• Embedded Control Applications success of software
• airplanes, air traffic control – PDAs products
• spaceships – memory seats
• watches – DVD players
• ovens – garage door openers
• remote controllers – cell phones

• Agile processes put increased pressure on testers

• Programmers must unit test – with no training, education or tools !
• Tests are key to functional requirements – but who builds those tests ?
3
The First Bugs
Hopper’s
“bug” (moth
stuck in a
relay on an
early machine)

“an analyzing process must

equally have been performed in “It has been just so in all of my
order to furnish the Analytical inventions. The first step is an
Engine with the necessary intuition, and comes with a burst,
operative data; and that herein then difficulties arise—this thing
may also lie a possible source of gives out and [it is] then that
error. Granted that the actual 'Bugs'—as such little faults and
mechanism is unerring in its difficulties are called—show
processes, the cards may give it themselves and months of intense
wrong orders. ” – Ada, Countess watching, study and labor are
Lovelace (notes on Babbage’s requisite. . .” – Thomas Edison
Analytical Engine)
4
 What is a computer bug?
• In 1947 Harvard University was operating a room-
sized computer called the Mark II.
• mechanical relays
• glowing vacuum tubes
• technicians program the computer by reconfiguring it
• Technicians had to change the occasional vacuum tube.
• A moth flew into the computer and was zapped
by the high voltage when it landed on a relay.
• Hence, the first computer bug!
• I am not making this up :-)
Bugs a.k.a. …

• Defect • Failure
• Fault • Inconsistency
• Problem • Product
• Error Anomaly
• Incident • Product
• Anomaly Incidence
• Variance • Feature :-)
Costly Software Failures
NIST report, “The Economic Impacts of Inadequate Infrastructure for
Software Testing” (2002)
– Inadequate software testing costs the US alone between $22 and $59 billion
annually
– Better approaches could cut this amount in half
Huge losses due to web application failures
– Financial services : $6.5 million per hour (just in USA!)
– Credit card sales applications : $2.4 million per hour (in USA)
In Dec 2006, [Link]’s BOGO offer turned into a double discount
2007 : Symantec says that most security vulnerabilities are due to
faulty software

7
 NASA’s Mars lander: September 1999, crashed
due to a units integration fault
Spectacular Software Failures Ariane 5:
Mars Polar exception-handling
Lander crash bug : forced self
site? destruct on maiden
flight (64-bit to 16-bit
Toyota brakes : Dozens dead, thousands of conversion: about
crashes 370 million $ lost)
Major failures: Ariane 5 explosion, Mars Polar
Lander, Intel’s Pentium FDIV bug
Poor testing of safety-critical software can cost
lives :
THERAC-25 radiation machine: 3 dead THERAC-25 design

We need our software to be

dependable
Testing is one way to assess dependability 8
 Ariane 5 Rocket Goes Boom (1996)
• Cost: $500 million
• Disaster: Ariane 5, an unmanned rocket of the European Space Agency, was
intentionally destroyed 37 seconds after launch on its maiden flight. Also destroyed
was its cargo of four scientific satellites to study how the Earth’s magnetic field interacts
with solar winds.
• Cause: Shutdown occurred when the guidance computer tried to convert the sideways
rocket velocity from 64b floating point to a 16b signed integer format. The number was
too big, and an overflow error resulted. When the guidance system shut down, control
passed to an identical redundant unit, which also failed because it was running the
same algorithm.
Software is a Skin that Surrounds
Our Civilization

Quote due to Dr. Mark Harman

10
 Pentium Fails Long Division (1993)
• Cost: $475 million, corporate credibility
• Disaster: Intel’s highly-promoted Pentium chip occasionally made
mistakes when dividing floating-point numbers within a specific
range. For example, dividing 4195835.0/3145727.0 yielded 1.33374
instead of 1.33382, an error of 0.006%. Although the bug affected
few users, it become a public relations nightmare. With an estimated
5 million defective chips in circulation, Intel offered to replace
Pentium chips only for consumers who could prove they needed high
accuracy. Eventually Intel replaced the chips for anyone who
complained.
• Cause: The divider in the Pentium floating point unit had a flawed
division table, missing about five of a thousand entries and resulting
in these rounding errors.
 Airbus 319 Safety Critical
Software Control

Loss of autopilot

Loss of most flight deck lighting and intercom

Loss of both the commander’s and the co-pilot’s

primary flight and navigation displays !
12
Northeast Blackout of 2003
508 generating
units and 256
power plants shut
down

Affected 10 million
people in Ontario,
Canada
Affected 40 million
people in 8 US
states
Financial losses of
$6 Billion USD

The alarm system in the energy management system failed due

to a software error and operators were not informed of the power
overload in the system
13
What Does This Mean?

Software testing is getting

more important

14
 Testing in the 21st Century
• More safety critical, real-time software
• Embedded software is ubiquitous … check your pockets
• Enterprise applications means bigger programs, more users
• Paradoxically, free software increases our expectations !
• Security is now all about software faults
• Secure software is reliable software
• The web offers a new deployment platform
• Very competitive and very available to more users
• Web apps are distributed
• Web apps must be highly reliable

Industry desperately needs our inventions !

15
Cost of Testing
You’re going to spend at least half of
your development budget on testing,
whether you want to or not
• In the real-world, testing is the principle post-design
activity

• Restricting early testing usually increases cost

• Extensive hardware-software integration requires more

testing

16
What is..........
Software Testing Definition
• Software testing is a process,
• to evaluate the functionality of a software application
with an intent
• to find whether the developed software met the specified
requirements or not and
• to identify the defects
• to ensure that the product is defect-free in order to
produce the quality product.
 Test Design in Context
• Test Design is the process of designing input values
that will effectively test software

• Test design is one of several activities for testing

software
• Most mathematical
• Most technically challenging

19
 Types of Test Activities
• Testing can be broken up into four general types of activities
1. Test Design 1.a) Criteria-based
2. Test Automation 1.b) Human-based
3. Test Execution
4. Test Evaluation
• Each type of activity requires different skills, background knowledge,
education and training
• No reasonable software development organization uses the same
people for requirements, design, implementation, integration and
configuration control

20
 1. Test Design – (a) Criteria-Based
• This is the most technical job in software testing
• Requires knowledge of :
• Discrete math
• Programming
• Testing
• Requires much of a traditional CS degree
• This is intellectually stimulating, rewarding, and challenging
• Test design is analogous to software architecture on the development
side
• Using people who are not qualified to design tests is a sure way to get
ineffective tests

21
1. Test Design – (b) Human-Based
• This is much harder than it may seem to developers
• Criteria-based approaches can be blind to special situations
• Requires knowledge of :
• Domain, testing, and user interfaces
• Requires almost no traditional CS
• A background in the domain of the software is essential
• An empirical background is very helpful (biology, psychology, …)
• A logic background is very helpful (law, philosophy, math, …)
• This is intellectually stimulating, rewarding, and challenging
• But not to typical CS majors – they want to solve problems and
build things
22
2. Test Automation
• This is slightly less technical
• Requires knowledge of programming
• Fairly straightforward programming – small pieces and simple
algorithms
• Requires very little theory
• Very boring for test designers
• Programming is out of reach for many domain experts
• Who is responsible for determining and embedding the expected
outputs ?
• Test designers may not always know the expected outputs
• Test evaluators need to get involved early to help with this

23
3. Test Execution
• This is easy – and trivial if the tests are well automated
• Requires basic computer skills
• Interns
• Employees with no technical background
• Asking qualified test designers to execute tests is a sure
way to convince them to look for a development job
• If, for example, GUI tests are not well automated, this
requires a lot of manual labor
• Test executors have to be very careful and meticulous with
bookkeeping

24
4. Test Evaluation
• This is much harder than it may seem
• Requires knowledge of :
• Domain
• Testing
• User interfaces and psychology
• Usually requires almost no traditional CS
• A background in the domain of the software is essential
• An empirical background is very helpful (biology, psychology, …)
• A logic background is very helpful (law, philosophy, math, …)
• This is intellectually stimulating, rewarding, and challenging
• But not to typical CS majors – they want to solve problems and
build things

25
 Other Activities
• Test management: Sets policy, organizes team, interfaces with development,
chooses criteria, and decides how much automation is needed, …
• Test maintenance: Save tests for reuse as software evolves
• Requires cooperation of test designers and automators
• Deciding when to trim the test suite is partly policy and partly technical –
and in general, very hard !
• Tests should be put in configuration control
• Test documentation : All parties participate
• Each test must document “why” – criterion and test requirement satisfied
or a rationale for human-designed tests
• Ensure traceability throughout the process
• Keep documentation in the automated tests

26
Types of Test Activities – Summary
1a. Design Design test values to satisfy engineering goals
Criteria Requires knowledge of discrete math, programming and
testing
1b. Design Design test values from domain knowledge and intuition
Human Requires knowledge of domain, UI, testing
2. Automation Embed test values into executable scripts
Requires knowledge of scripting
3. Execution Run tests on the software and record the results
Requires very little knowledge
4. Evaluation Evaluate results of testing, report to developers
Requires domain knowledge
• These four general test activities are quite different
• It is a poor use of resources to use people inappropriately
27
 Organizing the Team
• A mature test organization needs only one test designer to work with several
test automators, executors and evaluators
• Putting the wrong people on the wrong tasks leads to inefficiency, low job
satisfaction and low job performance
• A qualified test designer will be bored with other tasks and look for a job in
development
• A qualified test evaluator will not understand the benefits of test criteria
• Test evaluators have the domain knowledge, so they must be free to add tests
that “blind” engineering processes will not think of

28
 Model-Driven Test Design
refined
model / test
requirements /
structure requirements
test specs
test
requirements DESIGN
ABSTRACTION
LEVEL
IMPLEMENTATION
software ABSTRACTION
artifact LEVEL input
values

pass / test test test

fail results scripts cases

29
 Model-Driven Test Design – Steps
criterion refine refined
model / test
requirements /
structure requirements
test specs generate

analysis test
domain
requirements DESIGN
analysis ABSTRACTION
LEVEL
IMPLEMENTATION
software ABSTRACTION
artifact LEVEL input
values

execute prefix
evaluate automate
postfix
pass / test test test expected
fail results scripts cases

30
 Model-Driven Test Design –
Activities refined
model / test
requirements /
structure requirements
test specs
Test Design DESIGN
ABSTRACTION
LEVEL
Raising our abstraction level makes
IMPLEMENTATION
ABSTRACTION
software
test design MUCH easier
LEVEL input
artifact values

pass / test test test

fail results scripts cases
Test Test
Evaluation Execution
31
Small Illustrative Example
Control Flow Graph
Software Artifact : Java Method
/**
* Return index of node n at the 1 i = 0
* first position it appears,
* -1 if it is not present
*/ 2
i < [Link]()
public int indexOf (Node n)
{
for (int i=0; i < [Link](); i++)
3 if
if ([Link](i).equals(n))
return i;
return -1;
} 5 4
return -1 return i

32
Example (2)
Graph Edges 6 requirements for
Abstract version 12 Edge-Pair Coverage
1 23 1. [1, 2, 3]
32 2. [1, 2, 5]
34 3. [2, 3, 4]
2 25 4. [2, 3, 2]
Initial Node: 1 5. [3, 2, 3]
3 Final Nodes: 4, 5 6. [3, 2, 5]
Test Paths
5 4 [1, 2, 5]
[1, 2, 3, 2, 5] Find values …
[1, 2, 3, 2, 3, 4] 33
Goal of a software tester
• … to find bugs
• … as early in the software development processes as possible
• … and make sure they get fixed.
 How to define Software Testing
Principles
• Testing
The execution of a program to find its faults
• Verification
The process of proving the programs correctness.
• Validation
The process of finding errors by executing the program in a real
environment
• Debugging
Diagnosing the error and correct it
Fundamentals of Software Testing
Verification ????? Validation ??????
Validation & Verification (IEEE)
• Validation: The process of evaluating software at the end
of software development to ensure compliance with
intended usage

• Verification: The process of determining whether the

products of a given phase of the software development
process fulfill the requirements established during the
previous phase

37
 Verification & Validation-Example
Lets say we are writing a program for addition.
a+b = c

• Verification: “Are we building the product right?”

Are we getting some output for a+b? 1+1 = 6

• Validation: “Are we building the right product?”

Are we getting correct output for a+b? 1+1=2
Fundamentals of Software Testing
Validation Vs Verification
Verification Validation

1 Involves comparing a work Involves checking the outcome

product with process, of developed product and
standards and guidelines. process used with respect to
standards and expectations of a
customer.

2 Are we building the Are we building the right

product right? product?
3 Ensures that the software Ensures that the functionalities
system meets all meets the users needs and
functionalities. requirements
4 Done by Developers Done by Testers
 Validation Vs Verification
Verification Validation
5 Verification uses methods Validation uses methods like
like inspections, reviews, black box (functional) testing,
walkthroughs, etc. gray box testing, and white box
(structural) testing etc.
6 It generally comes first-done It generally follows
before validation. after verification.(Testing &
validating actual product)
7 It is human based checking of It is computer based execution
documents and files. of program.
8 It does not involve executing It always involves executing the
the code code
Test Engineer & Test Managers
• Test Engineer: An IT professional who is in charge of one or more technical test
activities
• designing test inputs
• producing test values
• running test scripts
• analyzing results
• reporting results to developers and managers

• Test Manager: In charge of one or more test engineers

• sets test policies and processes
• interacts with other managers on the project
• otherwise helps the engineers do their work

42
Static and Dynamic Testing
• Static Testing: Testing without executing the program.
• This includes software inspections and some forms of analyses
• Very effective at finding certain kinds of problems – especially
“potential” faults, that is, problems that could lead to faults
when the program is modified

• Dynamic Testing : Testing by executing the program with real

inputs

43
 Software Faults, Errors & Failures
• Software Fault: A static defect in the software

• Software Failure: External, incorrect behavior with respect

to the requirements or other description of the expected
behavior

• Software Error: An incorrect internal state that is the

manifestation of some fault

44
Testing & Debugging
• Testing: Finding inputs that cause the software to
fail

• Debugging: The process of finding a fault given a

failure

45
Fault & Failure Model
Three conditions necessary for a failure to be observed

1. Reachability : The location or locations in the program that contain

the fault must be reached

2. Infection : The state of the program must be incorrect

3. Propagation : The infected state must propagate to cause some

output of the program to be incorrect

46
Test Case
• Test Case Values: The values that directly satisfy one test
requirement

• Expected Results: The result that will be produced when

executing the test if the program satisfies it intended
behavior

47
Testing Goals Based on Test Process Maturity
▪ Level 0 : There’s no difference between testing and debugging

▪ Level 1 :The purpose of testing is to show correctness

▪ Level 2 : The purpose of testing is to show that the software
doesn’t work
▪ Level 3 : The purpose of testing is not to prove anything
specific, but to reduce the risk of using the software
▪ Level 4 : Testing is a mental discipline that helps all IT
professionals develop higher quality software

48
 Level 0 Thinking
• Testing is the same as debugging

• Does not distinguish between incorrect behavior and

mistakes in the program

• Does not help develop software that is reliable or safe

This is what we teach undergraduate CS majors

49
Level 1 Thinking
• Purpose is to show correctness
• Correctness is impossible to achieve
• What do we know if no failures?
• Good software or bad tests?
• Test engineers have no:
• Strict goal
• Real stopping rule
• Formal test technique
• Test managers are powerless

This is what hardware engineers often expect

50
Level 2 Thinking
• Purpose is to show failures

• Looking for failures is a negative activity

• Puts testers and developers into an adversarial relationship

• What if there are no failures?

This describes most software companies.

How can we move to a team approach ??
51
Level 3 Thinking
• Testing can only show the presence of failures

• Whenever we use software, we incur some risk

• Risk may be small and consequences unimportant

• Risk may be great and consequences catastrophic

• Testers and developers cooperate to reduce risk

This describes a few “enlightened” software companies
52
Level 4 Thinking
A mental discipline that increases quality

• Testing is only one way to increase quality

• Test engineers can become technical leaders of the project

• Primary responsibility to measure and improve software

quality

• Their expertise should help the developers

This is the way “traditional” engineering works
53
Where Are You?
Are you at level 0, 1, or 2 ?
Is your organization at work at level
0, 1, or 2 ?
Or 3?
We hope to teach you to become “change
agents” in your workplace …
Advocates for level 4 thinking
54
How to Improve Testing ?
• Testers need more and better software tools
• Testers need to adopt practices and techniques that lead to more
efficient and effective testing
• More education
• Different management organizational strategies
• Testing / QA teams need more technical expertise
• Developer expertise has been increasing dramatically
• Testing / QA teams need to specialize more
• This same trend happened for development in the 1990s

55
Software Testing Foundations

Testing can only show the presence of

failures

Not their absence

56
Testing & Debugging
• Testing : Evaluating software by observing its execution

• Test Failure : Execution of a test that results in a software

failure

• Debugging : The process of finding a fault given a failure

Not all inputs will “trigger” a fault

into causing a failure
57
 Fault & Failure Model (RIPR)
Four conditions necessary for a failure to be observed

1. Reachability: The location or locations in the program that contains the

fault must be reached
2. Infection: The state of the program must be incorrect
3. Propagation: The infected state must cause some output or final state of
the program to be incorrect
4. Reveal: The tester must observe part of the incorrect portion of the
program state

58
RIPR Model
Test
Final Program State
Reaches Observed Final
• Reachability Observed
Program State
Final Program
• Infection Fault State
Incorrect
Final
Infects
• Propagation State
Propagates Reveals
• Revealability Incorrect
Program
State Test
Oracles
59
Traditional Testing Levels Acceptance testing : Is
main Class P the software acceptable
to the user?

System testing : Test the

overall functionality of
the system
Class A Class B
Integration testing : Test
method mA1() method mB1() how modules interact
with each other
method mA2() method mB2()
Module testing
(developer testing) : Test
each class, file, module,
component
Unit testing (developer
This view obscures underlying testing) : Test each unit
similarities (method) individually
60
Object-Oriented Testing
Inter-class testing : Test
Levels
multiple classes together

Class A Class B
Intra-class testing : Test an
method mA1() method mB1() entire class as sequences
of calls
method mA2() method mB2()
Inter-method testing :
Test pairs of methods in
the same class
Intra-method testing :
Test each method
individually
61
New : Test Coverage Criteria
A tester’s job is simple : Define a model of the software, then find
ways to cover it

Test Requirements: A specific element of a software artifact that a

test case must satisfy or cover

Coverage Criterion: A rule or collection of rules that impose test

requirements on a test set

Testing researchers have defined dozens of criteria, but

they are all really just a few criteria on four types of
structures …
62
Coverage Criteria
• Even small programs have too many inputs to fully test them all

• private static double computeAverage (int A, int B, int C)

• On a 32-bit machine, each variable has over 4 billion possible values

• Over 80 octillion possible tests!!
• Input space might as well be infinite
• Testers search a huge input space
• Trying to find the fewest inputs that will find the most problems
• Coverage criteria give structured, practical ways to search the input space
• Search the input space thoroughly
• Not much overlap in the tests
63
Advantages of Coverage Criteria
• Maximize the “bang for the buck”

• Provide traceability from software artifacts to tests

• Source, requirements, design models, …

• Make regression testing easier

• Gives testers a “stopping rule” … when testing is finished

• Can be well supported with powerful tools

64
Test Requirements and Criteria
• Test Criterion : A collection of rules and a process that define test
requirements
̶ Cover every statement
̶ Cover every functional requirement
• Test Requirements : Specific things that must be satisfied or covered
during testing
• Each statement might be a test requirement
• Each functional requirement might be a test requirement
Testing researchers have defined dozens of criteria, but they are
all really just a few criteria on four types of structures …

1. Input domains 3. Logic expressions

2. Graphs 4. Syntax descriptions
65
Source of Structures
• These structures can be extracted from lots of software
artifacts
• Graphs can be extracted from UML use cases, finite state machines,
source code, …
• Logical expressions can be extracted from decisions in program
source, guards on transitions, conditionals in use cases, …
• This is not the same as “model-based testing,” which derives
tests from a model that describes some aspects of the system
under test
• The model usually describes part of the behavior
• The source is explicitly not considered a model

66
Criteria Based on Structures
Structures : Four ways to model software

1. Input Domain A: {0, 1, >1}

B: {600, 700, 800}
Characterization C: {swe, cs, isa, infs}
(sets)
2. Graphs

3. Logical Expressions (not X or not Y) and A and B

if (x > y)
4. Syntactic Structures else
z = x - y;

(grammars) z = 2 * x;
67
Example : Jelly Bean Coverage
Flavors : Colors :
1. Lemon 1. Yellow (Lemon, Apricot)
2. Pistachio 2. Green (Pistachio)
3. Cantaloupe 3. Orange (Cantaloupe,
4. Pear Tangerine)
5. Tangerine 4. White (Pear)
6. Apricot

Possible coverage criteria :

1. Taste one jelly bean of each flavor
• Deciding if yellow jelly bean is Lemon or Apricot is a controllability
problem
2. Taste one jelly bean of each color

68
Coverage
Given a set of test requirements TR for coverage
criterion C, a test set T satisfies C coverage if and only
if for every test requirement tr in TR, there is at least
one test t in T such that t satisfies tr
• Infeasible test requirements : test requirements that
cannot be satisfied
• No test case values exist that meet the test requirements
• Example: Dead code
• Detection of infeasible test requirements is formally undecidable for most
test criteria
• Thus, 100% coverage is impossible in practice

69
More Jelly Beans
T1 = { three Lemons, one Pistachio, two Cantaloupes,
one Pear, one Tangerine, four Apricots }
• Does test set T1 satisfy the flavor criterion ?

T2 = { One Lemon, two Pistachios, one Pear, three

Tangerines }

Does test set T2 satisfy the flavor criterion ?

Does test set T2 satisfy the color criterion ?

70
Types of Testing
• Levels of testing include different methodologies that can be
used while conducting software testing.
• The main levels of software testing are −
• Functional Testing
• Non-functional Testing
Functional Testing
• This is a type of black-box testing that is based on the specifications
of the software that is to be tested.
• The application is tested by providing input and then the results are
examined that need to confirm to the functionality it was intended
for.
• Functional testing of a software is conducted on a complete,
integrated system to evaluate the system's compliance with its
specified requirements.
• There are four main types of functional testing.
• There are five steps that are involved while testing an
application for functionality.

I The determination of the functionality that the intended application is meant

to perform.

II The creation of test data based on the specifications of the application.

III The output based on the test data and the specifications of the application.

IV The writing of test scenarios and the execution of test cases.

V The comparison of actual and expected results based on the executed test
cases.
[Link] Testing
• Unit testing is a type of software testing which is done on an individual unit or
component to test its corrections.
• Typically, Unit testing is done by the developer at the application development
phase.
• Each unit in unit testing can be viewed as a method, function, procedure, or
object.
• Developers often use test automation tools such as NUnit, Xunit, JUnit for the
test execution.
• Unit testing is important because we can find more defects at the unit test
level.
• For example, there is a simple calculator application. The developer can write
the unit test to check if the user can enter two numbers and get the correct
sum for addition functionality.
[Link] Testing
• a) White Box Testing
• White box testing is a test technique in which the internal structure or code of
an application is visible and accessible to the tester.
• In this technique, it is easy to find loopholes in the design of an application or
fault in business logic.
• Statement coverage and decision coverage/branch coverage are examples of
white box test techniques.
• b) Gorilla Testing
• Gorilla testing is a test technique in which the tester and/or developer test the
module of the application thoroughly in all aspects.
• Gorilla testing is done to check how robust your application is.
 Unit Testing (White Box)
• Individual components are tested.
• It is a path test.
• To focus on a relatively small segment of code and aim to
exercise a high percentage of the internal path
• Disadvantage: the tester may be biased by previous
experience. And the test value may not cover all possible
values.
2) Integration Testing
• Integration testing is a type of software testing where two or more modules
of an application are logically grouped together and tested as a whole.
• The focus of this type of testing is to find the defect on interface,
communication, and data flow among modules.
• Top-down or Bottom-up approach is used while integrating modules into the
whole system.
• This type of testing is done on integrating modules of a system or between
systems.
• For example, a user is buying a flight ticket from any airline website. Users
can see flight details and payment information while buying a ticket, but
flight details and payment processing are two different systems. Integration
testing should be done while integrating of airline website and payment
processing system.
 Integration Testing
• Top-down Integration Test
• Bottom-up Integration Test
 Top-down Integration Test
• The control program is tested first. Modules are integrated one
at a time. Emphasize on interface testing
• Advantages: No test drivers needed
Interface errors are discovered early
Modular features aid debugging
• Disadvantages: Test stubs are needed
Errors in critical modules at low levels are found late.
 Bottom-up Integration Test
• Allow early testing aimed at proving feasibility
Emphasize on module functionality and performance
• Advantages: No test stubs are needed
Errors in critical modules are found early
• Disadvantages: Test drivers are needed
Interface errors are discovered late
 Bottom-up testing

Test Test Test

Drivers Drivers Drivers

Level N Level N Level N

Test Test
Drivers Drivers

Level N-1 Level N-1

3) System Testing
• System testing is type of testing where the tester evaluates the whole system
against the specified requirements.
a) End to End Testing
• It involves testing a complete application environment in a situation that
mimics real-world use, such as interacting with a database, using network
communications, or interacting with other hardware, applications, or systems
if appropriate.
• For example, a tester is testing a pet insurance website. End to End testing
involves testing buying an insurance policy, LPM, tag, adding another pet,
updating credit card information on users’ accounts, updating user address
information, and receiving order confirmation emails and policy documents.
b) Black Box Testing
• Blackbox testing is a software testing technique in which testing is performed
without knowing the internal structure, design, or code of a system under
test. Testers should focus only on the input and output of test objects.
3) System Testing
c) Smoke Testing
• Smoke testing is performed to verify that basic and critical functionality of
the system under test is working fine at a very high level.
• Whenever a new build is provided by the development team, then the
Software Testing team validates the build and ensures that no major issue
exists. The testing team will ensure that the build is stable, and a detailed
level of testing will be carried out further.
d) Sanity Testing
• Sanity testing is performed on a system to verify that newly added
functionality or bug fixes are working fine. Sanity testing is done on stable
build. It is a subset of the regression test.
• For example, a tester is testing a pet insurance website. There is a change
in the discount for buying a policy for second pet. Then sanity testing is
only performed on buying insurance policy module.
4) Acceptance Testing
• Acceptance testing is a type of testing where
client/business/customer test the software with real-time business
scenarios.
• The client accepts the software only when all the features and
functionalities work as expected. This is the last phase of testing, after
which the software goes into production. This is also called User
Acceptance Testing (UAT).
• a) Alpha Testing
• Alpha testing is a type of acceptance testing performed by the team in
an organization to find as many defects as possible before releasing
software to customers.
• For example, the pet insurance website is under UAT. UAT team will
run real-time scenarios like buying an insurance policy, buying annual
membership, changing the address, and ownership transfer of the pet
in a same way the user uses the real website. The team can use test
credit card information to process payment-related scenarios.
• b) Beta Testing
• Beta Testing is a type of software testing which is carried out by the
clients/customers. It is performed in the Real Environment before
releasing the product to the market for the actual end-users.
• Beta Testing is carried out to ensure that there are no major failures
in the software or product, and it satisfies the business requirements
from an end-user perspective. Beta Testing is successful when the
customer accepts the software.
• c) Operational acceptance testing (OAT)
• Operational acceptance testing of the system is performed by operations
or system administration staff in the production environment. The
purpose of operational acceptance testing is to make sure that the
system administrators can keep the system working properly for the
users in a real-time environment.
The focus of the OAT is on the following points:
• Testing of backup and restore.
• Installing, uninstalling, upgrading software.
• The recovery process in case of natural disaster.
• User management.
• Maintenance of the software.
 Non-Functional Testing
• 1) Security Testing
• It is a type of testing performed by a special team. Any hacking method can
penetrate the system.
• Security Testing is done to check how the software, application, or website is
secure from internal and/or external threats. This testing includes how much
software is secure from malicious programs, viruses and how secure & strong
the authorization and authentication processes are.
• It also checks how software behaves for any hacker’s attack & malicious
programs and how software is maintained for data security after such a hacker
attack.
• a) Penetration Testing

• Penetration Testing or Pen testing is the type of security testing

performed as an authorized cyberattack on the system to find out the
weak points of the system in terms of security.
• Pen testing is performed by outside contractors, generally known as
ethical hackers. That is why it is also known as ethical hacking.
Contractors perform different operations like SQL injection, URL
manipulation, Privilege Elevation, session expiry, and provide reports
to the organization.
2) Performance Testing
• Performance testing is testing of an application’s stability and response
time by applying load.
• The word stability means the ability of the application to withstand in
the presence of load. Response time is how quickly an application is
available to users. Performance testing is done with the help of tools.
[Link], JMeter, LoadRunner, etc. are good tools available in the
market.
• a) Load testing
• Load testing is testing of an application’s stability and response time by
applying load, which is equal to or less than the designed number of
users for an application.
• For example, your application handles 100 users at a time with a
response time of 3 seconds, then load testing can be done by applying a
load of the maximum of 100 or less than 100 users. The goal is to verify
that the application is responding within 3 seconds for all the users.
b) Stress Testing
• Stress testing is testing an application’s stability and response time by
applying load, which is more than the designed number of users for
an application.
• For example, your application handles 1000 users at a time with a
response time of 4 seconds, then stress testing can be done by
applying a load of more than 1000 users. Test the application with
1100,1200,1300 users and notice the response time. The goal is to
verify the stability of an application under stress.
C) Scalability Testing
• Scalability testing is testing an application’s stability and response time by
applying load, which is more than the designed number of users for an
application.
• For example, your application handles 1000 users at a time with a response
time of 2 seconds, then scalability testing can be done by applying a load of
more than 1000 users and gradually increasing the number of users to find out
where exactly my application is crashing.
• Let’s say my application is giving response time as follows:
• 1000 users -2 sec
• 1400 users -2 sec
• 4000 users -3 sec
• 5000 users -45 sec
• 5150 users- crash – This is the point that needs to identify in scalability testing
• d) Volume testing (flood testing)
• Volume testing is testing an application’s stability and response time
by transferring a large volume of data to the database. Basically, it
tests the capacity of the database to handle the data.
• e) Endurance Testing (Soak Testing)
• Endurance testing is testing an application’s stability and response
time by applying load continuously for a longer period to verify that
the application is working fine.
• For example, car companies soak testing to verify that users can drive
cars continuously for hours without any problem.
3) Usability Testing
• Usability testing is testing an application from the user’s perspective
to check the look and feel and user-friendliness.
• For example, there is a mobile app for stock trading, and a tester is
performing usability testing.
• Testers can check the scenario like if the mobile app is easy to operate
with one hand or not, scroll bar should be vertical, background color
of the app should be black and price of and stock is displayed in red
or green color.

• The main idea of usability testing of this kind of app is that as soon as
the user opens the app, the user should get a glance at the market.
• a) Exploratory testing

• Exploratory Testing is informal testing performed by the testing team.

The objective of this testing is to explore the application and look for
defects that exist in the application. Testers use the knowledge of the
business domain to test the application. Test charters are used to
guide the exploratory testing.

• b) Cross browser testing

• Cross browser testing is testing an application on different browsers,

operating systems, mobile devices to see look and feel and
performance.
• c) Accessibility Testing

• The aim of Accessibility Testing is to determine whether the software

or application is accessible for disabled people or not.

• Here, disability means deafness, color blindness, mentally disabled,

blind, old age, and other disabled groups. Various
Black-Box Testing
• The technique of testing without having any knowledge of the interior
workings of the application is called black-box testing.
• The tester is oblivious to the system architecture and does not have
access to the source code.
• Typically, while performing a black-box test, a tester will interact with
the system's user interface by providing inputs and examining outputs
without knowing how and where the inputs are worked upon.
• Black box testing can be applied to three main types of tests:
functional, non-functional, and regression testing.
Regression Testing
• Regression testing is testing of unchanged features of the application to
make sure that any bug fixes, adding new features, deleting, or updating
existing features, are not impacting the working application.

• To find out regression scope is an important part of Regression Testing.

• To find out the regression scope, the Tester needs to find out the area of
application where changes happened and the Impact of those changes on
the entire application.
• It is difficult to cover the whole regression test suite in every release, so
Automation Testing Tools are used in regression testing.
White-Box Testing
• White-box testing is the detailed investigation of internal logic and
structure of the code.
• White-box testing is also called glass testing or open-box testing.
• In order to perform white-box testing on an application, a tester
needs to know the internal workings of the code.
• The tester needs to have a look inside the source code and find out
which unit/chunk of the code is behaving inappropriately.
Grey-Box Testing

• Grey-box testing is a technique to test the application with having a limited

knowledge of the internal workings of an application.
• In software testing, the phrase the more you know, the better carries a lot
of weight while testing an application.
• Mastering the domain of a system always gives the tester an edge over
someone with limited domain knowledge.
• Unlike black-box testing, where the tester only tests the application's user
interface; in grey-box testing, the tester has access to design documents
and the database.
• Having this knowledge, a tester can prepare better test data and test
scenarios while making a test plan.
• Gray Box Testing is a software testing method, which is a
combination of both White Box Testing and Black Box Testing
method.
• In White Box testing internal structure (code) is known
• In Black Box testing internal structure (code) is unknown
• In Grey Box Testing internal structure (code) is partially known
• Techniques used for Grey box Testing are-

• Matrix Testing: This testing technique involves defining all the

variables that exist in their programs.
• Regression Testing: To check whether the change in the previous
version has regressed other aspects of the program in the new
version. It will be done by testing strategies like retest all, retest risky
use cases, retest within a firewall.
• Orthogonal Array Testing or OAT: It provides maximum code
coverage with minimum test cases.
• Pattern Testing: This testing is performed on the historical data of the
previous system defects. Unlike black box testing, gray box testing
digs within the code and determines why the failure happened
Black Box Testing vs. White Box Testing vs.
Grey Box Testing