Introduction to Software Testing

CHAPTER 1: OVERVIEW
Content
□ What is Testing ?
□ Why we must test?
□ Why Testing is so hard?
□ Objectives of Testing
□ Testing group
□ Key Lessons-Learned
□ Software Development Lifecycles (SDLC)
and Testing
Lê Ngô Thục Vi Introduction to Software Testing 2
What is testing?

Executing software/system in a simulated or real

environment, using inputs selected somehow

Lê Ngô Thục Vi Introduction to Software Testing 3

What is testing?
□ Testing vs. Debugging
■ Testing: Finding inputs that cause the software
to fail
■ Debugging: The process of finding a fault give
a failure
□ Fault: incorrect piece of code
□ Failure: result of a fault

Lê Ngô Thục Vi Introduction to Software Testing 4

What is testing
□ Debugging process

Lê Ngô Thục Vi Introduction to Software Testing 5

Why we must test?
□ Testing in the 1980s and 1990s
■ Software engineering was very different
□ Low quality software was normal and expected
□ The cost of building better software outweighed
the benefits
■ Software was smaller:
□ Most software was bundled, shrink-wrapped,
contracted
□ Industry became dominated by one monopolistic
vendor

Lê Ngô Thục Vi Introduction to Software Testing 6

Why we must test?
□ Testing in the 21st Century
■ The field has dramatically changed
■ Today’s software market :
□ is much bigger
□ is more competitive
□ has more users
□ used in more places
■ The web offers a new deployment platform
□ Very competitive and very available to more users

Lê Ngô Thục Vi Introduction to Software Testing 7

Why we must test?
□ Testing in the 21st Century
■ Enterprise applications means bigger
programs and more users
■ Embedded software is now ubiquitous …
■ Paradoxically, free software increases user
expectations !

Lê Ngô Thục Vi Introduction to Software Testing 8

 Why we must test
□ Cost of late testing
Planning &
Requirements Design Development Deploy to
Testing AUT
Production
Fault
Origination 10% 40% 50%

Fault 6% 13% 20% 20% 36% 5%

Discovery

Requirements Design Development Functional System

Test Planning Review Production
Unit Testing Testing Testing
Cost per Fault 1X 1X 1X 5X 10X 50X

Current State $6K $13K $20K $101K $363K $252K

Lê Ngô Thục Vi Introduction to Software Testing 9

Why we must test?

Software testing is fast becoming

essential to an essential problem in the
software industry

Lê Ngô Thục Vi Introduction to Software Testing 10

Why we must test?
□ Future of Software Testing
■ Increased specialization in testing teams will lead
to more efficient and effective testing
■ Testing and QA teams will have more technical
expertise
■ Developers will have more knowledge about
testing and motivation to test better
■ Agile processes puts testing first—putting pressure
on both testers and developers to test better

Lê Ngô Thục Vi Introduction to Software Testing 11

 Why testing is so hard?
□ Example
The program’s specification:
■ This program is
designed to add two
numbers, which you
will enter
■ Each number should
be one or two digits

Lê Ngô Thục Vi Introduction to Software Testing 12

Why testing is so hard?

Lê Ngô Thục Vi Introduction to Software Testing 13

Why testing is so hard?
□ How many test case?
There are 199 x 199 = 39,601 test cases for valid values:
■ definitely valid: 0 to 99
■ might be valid: -99 to -1

There are infinitely many invalid cases:

■ 100 and above
■ -100 and below
■ anything non-numeric

Lê Ngô Thục Vi Introduction to Software Testing 14

Why testing is so hard?
□ * Smallest/Largest Possible Values Allowed
via UI

□ 8 □ 1 □ 9
* *
□ - □ Partition’s Valid
□ α
Inputs
α

□ 2 □ 5
□ L □ U
B B
□ Lower □ 3 □ Upper Boundary □ 6
+ +
Boundary (LB) (UB)
□ L
1 □ U
1
B□ 4 B□ 7
- -
1 1

Lê Ngô Thục Vi Introduction to Software Testing 15

Why testing is so hard?
□ Determining whether or not outputs are
correct.
□ Comparing resulting internal states to
expected states.
□ Determining whether adequate testing has
been done.
□ Determining what you can say about the
software when testing is completed.
□ Measuring performance characteristics.

Lê Ngô Thục Vi Introduction to Software Testing 16

Why testing is so hard?
□ Testing resources (staff, time, tools, labs)
are limited.
□ Specifications are frequently
unclear/ambiguous and changing (and not
necessarily complete and up-to-date).
□ Systems are almost always too large to
permit test cases to be selected based on
code characteristics.

Lê Ngô Thục Vi Introduction to Software Testing 17

Objectives of Testing
□ Detect faults
□ Establish confidence in software
□ Evaluate properties of software
■ Reliability
■ Performance
■ Memory Usage
■ Security
■ Usability

Lê Ngô Thục Vi Introduction to Software Testing 18

Objectives of Testing
□ Should not be to verify that the program works correctly
because:
■ If you can’t test the program completely, you can’t verify that
it works correctly.
■ The program doesn’t work correctly, so you can’t verify that it
does.
■ Then as a tester, you reach your goal every time you find an
error.
■ You’ll be more likely to miss problems than if you want and
expect the program to fail

Lê Ngô Thục Vi Introduction to Software Testing 19

Objectives of Testing
The Objective of Testing a Program is
to Find Problems.
Finding problems is the core of your work. You should
want to find as many problems as possible. The more
serious the problem tester find, the better tester is.
A test that reveals a problem is a
success. A test that did not reveal a
problem is (often) a waste of time!

Lê Ngô Thục Vi Introduction to Software Testing 20

Objectives of Testing
The Purpose of Finding Problems is to
Get Them Fixed.
The point of the exercise is quality improvement!

□ The best tester is not the one who finds the most
bugs or who embarrasses the most programmers.
□ The best tester is the one who gets the most bugs
fixed.

Lê Ngô Thục Vi Introduction to Software Testing 21

Testing group
□ The Testing Group provides important technical
services, including:
■ Finding and reporting bugs.
■ Identifying weak areas of the program.
■ Identifying high risk areas of a project.
■ Explaining findings in ways that help
□ Engineering solve or fix the problems.
□ The customer service staff help customers
□ Management make sound business decisions about each bug.

Lê Ngô Thục Vi Introduction to Software Testing 22

Testing group
□ Find Problems
■ Find bugs.
■ Find design issues.
■ Find more efficient ways to find bugs.

□ Communicate Problems
■ Report bugs and design issues.
■ Report on testing progress.
■ Evaluate and report the program’s stability.

□ More Senior Testers Manage/Supervise Testing Projects

■ Prepare test plans and schedules.
■ Estimate testing tasks, resources, time and budget.
■ Measure and report testing progress against milestones.
■ Teach other testers to find bugs.

Lê Ngô Thục Vi Introduction to Software Testing 23

Testing group
□ Quality Assurance (QA): A process for providing
adequate assurance that the software products and
processes in the product life cycle conform to their
specific requirements and adhere to their established
plans
□ Quality Control (QC): A set of activities designed to
evaluate a developed working product.
■ Testing: The process of executing a system with the intent of
finding defects including test planning prior to the execution
of the test cases.

Lê Ngô Thục Vi Introduction to Software Testing 24

Testing group
Quality Assurance Quality Control
An activity that establishes
An activity which verifies if
and evaluates the
the product meets pre-
processes to produce the
defined standards.
products.
Helps establish processes. Implements the process.
Sets up measurements Verifies if specific attributes
programs to evaluate are in a specific product or
processes. Service

Identifies weaknesses in Identifies defects for the

processes and improves primary purpose of correcting
them. defects.
Lê Ngô Thục Vi Introduction to Software Testing 25
Testing group
Quality Assurance Quality Control

QA is the responsibility of QC is the responsibility of the

the entire team. tester.

Prevents the introduction of Detects, reports and corrects

issues or defects defects
QA evaluates whether or not
QC evaluates if the
quality control is working for
application is working for the
the primary purpose of
primary purpose of
determining whether or not
determining if there is a flaw /
there is a weakness in the
defect in the functionalities.
process.

Lê Ngô Thục Vi Introduction to Software Testing 26

Testing group

Quality Assurance Quality Control

QA improves the process
QC improves the
that is applied to multiple
development of a specific
products that will ever be
product or service.
produced by a process.
QA personnel should not
QC personnel may perform
perform quality control
quality assurance tasks if
unless doing it to validate
and when required.
quality control is working.

Lê Ngô Thục Vi Introduction to Software Testing 27

 Testing group

Quality Assurance is interested in the process

whereas
Quality Control are interested in the product

Lê Ngô Thục Vi Introduction to Software Testing 28

Key lessons-learned
□ The Program Doesn’t Work. Your task is to
find errors.
□ Be Methodical
□ Complete Testing is Impossible
□ Use Powerful Representatives of Tests
□ Communication Must be Effective
□ Change is Normal

Lê Ngô Thục Vi Introduction to Software Testing 29

Key lessons-learned
□ The Program Doesn’t Work. Your task is to
find errors.
■ All programs have bugs. Nobody would pay you to test if their
program didn’t have bugs.
■ Any change to a program can cause new bugs, and any aspect of
the program can be broken.
■ You DON’T “verify that the program is working.” You FIND bugs.

If you set your mind to show that a program works

correctly, you’ll be more likely to miss problems than if
you want and expect the program to fail.

Lê Ngô Thục Vi Introduction to Software Testing 30

Key lessons-learned
□ Be Methodical
■ Testing isn’t just banging away at the keyboard.
■ To have any hope of doing an efficient job, you must be
methodical:
□ Break the testing project down into tasks and subtasks so
that you have a good idea of what has to be done and what
has been done.
□ Track what has been done so that people don’t repeat the
same tasks and you know what tasks are left.
□ Prioritize the tasks

Lê Ngô Thục Vi Introduction to Software Testing 31

Key lessons-learned
□ Complete Testing is Impossible
■ There are a nearly infinite number of paths through any
non-trivial program.
■ There are a virtually infinite set of combinations of data
that you can feed the program.
You can’t test them all
■ Therefore, your task is to find bugs -- not to find all the
bugs
■ You want to
□ Find as many bugs as possible
□ Find the most serious bugs
□ Find bugs as early as possible
Lê Ngô Thục Vi Introduction to Software Testing 32
Key lessons-learned
□ Use Powerful Representatives of Tests
■ Equivalence classes
■ Boundary conditions
■ Input combinations
■ Data/functionality relationships
■ …………………..

Lê Ngô Thục Vi Introduction to Software Testing 33

Key lessons-learned
□ Communication Must be Effective
■ Your bug reports advise people of difficult
situations. The problems that you report can affect
□ The project schedule
□ The company’s cash flow
■ The clearer your reports are, the more likely it will
be that the company will make reasonable
business decisions based on them
■ Persuasive and technical writing, oral argument,
face-to-face negotiation, and diplomacy are core
skills for your job
Lê Ngô Thục Vi Introduction to Software Testing 34
Key lessons-learned
□ Change is Normal
■ Project requirements change, as do design and
specifications
■ The market changes
■ Development groups come to understand the product more
thoroughly as it is being built
■ Some companies accept late changes into their products as
a matter of course
As a new tester, you might decide quickly that this is a bad,
amateurish way to do business -> It might be, and it might
not be.
Take steps to make yourself more effective in dealing with late
changes.

Lê Ngô Thục Vi Introduction to Software Testing 35

Software Development Lifecycles
(SDLC) and Testing
□ SDLC and Testing
□ Waterfall Model
□ Spiral Model
□ V-Model
□ Concurrent Model
□ Agile Model
□ Other SDLC Models

Lê Ngô Thục Vi Introduction to Software Testing 36

SDLC and Testing
■How testing is directly affected by the SDLC
choice:
●Documentation availability to test against
●Time to test
●Time to automate or produce effective
automation
●Knowledge and understanding of the application
as we plan our tests
●Amount of regression testing

Lê Ngô Thục Vi Introduction to Software Testing 37

Waterfall Model
Requirements
Definition

verify Functional
Design

verify Technical
Design
• The Waterfall model is a
verify Coding
sequential software
development model.
• Transition between phases is verify Testing
done by a formal review.
• The review is a checkpoint to
see that you are on the right verify Deployment
track.

Lê Ngô Thục Vi Introduction to Software Testing 38

Testing in Waterfall Model
□ Testing is not inherent to every phase of the Waterfall
model.
□ Constant testing from the design, implementation and
verification phases is required to validate the phases
preceding them.
□ Testing phase: Only after coding phase, software
testing begins. Different testing methods are available
to detect the bugs that were committed during the
previous phases.

Lê Ngô Thục Vi Introduction to Software Testing 39

Role of testing team in Waterfall
Model
Requirements Testing or QA
Definition
Team
verify Functional
Design

verify Technical
Design

verify Coding

verify Testing

verify Deployment

Lê Ngô Thục Vi Introduction to Software Testing 40

Spiral Model

Lê Ngô Thục Vi Introduction to Software Testing 41

Testing in Spiral Model
□ Planned and structured releases
□ Usually has documentation to test against
□ Each spiral iteration can be thought of as a
“mini-waterfall”; there are defined testing
phases
□ Previous releases must be regression tested

Lê Ngô Thục Vi Introduction to Software Testing 42

V-Model

Lê Ngô Thục Vi Introduction to Software Testing 43

Testing in V-Model
□ Verification: Checks that a deliverable is complete (Contains all
requires information, follows standards; verify a procedure).
□ Validation: Checks that the deliverables satisfy requirements
specified in the previous stage or an earlier stage, and that the
business case is met (Validate a function or requirement).
□ Testing: Ensures that the specification is properly assembled and
implemented (Test to see if it works).
□ In testing, these are important changes in software development
from the V-model:
●Write the test cases at requirements review.
●Unit testing

Lê Ngô Thục Vi Introduction to Software Testing 44

Concurrent Model
None Analysis activity

Under
development

Awaiting Under
changes review

Under
revision Baselined

Done

All activities concurrent but reside in different states

Lê Ngô Thục Vi Introduction to Software Testing 45
Testing in Concurrent Model
□ Planning is concurrent to design; design is concurrent to
development—everything is happening at the same time.
□ The whole project is not well planned or well structured.
□ Planning, design and development are most dynamic. Product is
in constant change. Very difficult to test; impossible to effectively
plan testing project.
□ Often no documentation to test against.
□ Testing is ad hoc.
□ Coverage usually cannot be measured. Structured regression
testing is impossible.
□ Bugs will be missed because of so much change and so little
planning.
□ Risk analysis and reporting is crucial

Lê Ngô Thục Vi Introduction to Software Testing 46

Agile Model

Lê Ngô Thục Vi Introduction to Software Testing 47

Testing in Agile Model
□ Delivery cycles are short. Development is very
focused. Diagrams, use cases, user stories, index
cards, or discussions of functionality serve as
“documentation” to test against.
□ Agile projects usually have more unit testing by
developers.
□ Dynamic nature of development needs structured
regression testing.
□ Testing is often ad hoc, but focused.
□ Dynamic style or projects include much change but
the change is discussed and side-effects noted.

Lê Ngô Thục Vi Introduction to Software Testing 48

Other SDLC Model
□ Rapid Application Development (RAD) Model
□ Test First
□ Rational Unified Process (RUP)
□ eXtreme Programming (XP)
□ …the list goes on

Lê Ngô Thục Vi Introduction to Software Testing 49

 ASK and ANSWER

Lê Ngô Thục Vi Introduction to Software Testing 50

Work at home
□ Read:
■ Textbook: Chapter 1, Chapter 2
■ Paper: “What is software Testing?”

Lê Ngô Thục Vi Introduction to Software Testing 51