Software Engineering

Testing Basics

CUHK Shenzhen
Pinjia He
1. What is Software Testing
2. Why Software Testing
3. Stages and Categories of Testing
4. Coverage Metrics

Pinjia He @ CUHK Shenzhen

What is Software Testing
Software testing is the process of evaluating and
verifying that a software product or application
does what it is supposed to do.
—— IBM
[Link]

Find as many bugs as possible.

Pinjia He @ CUHK Shenzhen

Bugs
why we call flaws in software “bugs”?

Logbook page of the Mark II Aiken Relay Calculator with the first bug (1947)
Recorded by Grace Hopper
Pinjia He @ CUHK Shenzhen
The Patriot Accident
• The Patriot missile air defense system
tracks and intercepts incoming
missiles

• On February 25, 1991, a Patriot

system ignored an incoming Scud
missile

• 28 soldiers died; 98 were injured

Pinjia He @ CUHK Shenzhen

Patriot Bug
• The tracking algorithm measures time in 1/10s
• Time is stored in a 24-bit register
Ø Precise binary representation of 1/10 (infinite):
0.00011001100110011001100110011001…
Ø Truncated value in 24-bit register:
0.00011001100110011001100
Ø Rounding error: ca. 0.000000095s every 1/10s
• After 100 hours of operation error is 0.34s
• A Scud travels at about 1.7km/s, and so travels more
than 0.5kms in this time

Pinjia He @ CUHK Shenzhen

Windows Bug

[Link]

Pinjia He @ CUHK Shenzhen

14 Years Later

[Link]

Pinjia He @ CUHK Shenzhen

And Many Others…
• Bitcoin Hack, Mt. Gox, 2011 - $1.5 million
• Bitcoin Hack, Mt. Gox, 2014 - $0.5 billion
• Knight in bad trades, 2012 - $440 million
• Airbus A400M Crash – human life
• Heartbleed bug in OpenSSL – leak of password
•…

Pinjia He @ CUHK Shenzhen

Why Does Software Contain
Bugs?
• Out limited ability to predict code behavior
Ø Software is extremely complex
Ø No developer can understand the whole system

• We humans make mistakes

Ø Unclear requirements, miscommunication
Ø Wrong assumptions (e.g., behavior of OS)
Ø Design errors (e.g., capacity of data structure too small)
Ø Coding errors (e.g., wrong loop condition)

Pinjia He @ CUHK Shenzhen

History of Software Testing
• Debugging era
• Demonstration era
• Destruction era
• Evaluation era
• Prevention era

Pinjia He @ CUHK Shenzhen

Debugging Era
• Early 1950s
• No distinction between testing & debugging
• Focus: fixing bugs
• Testing? Testers?

Pinjia He @ CUHK Shenzhen

Demonstration Era
• 1957-1978
• Testing was carried out as a separate activity
• Software requirements satisfied?

Pinjia He @ CUHK Shenzhen

Destruction Era
• 1979-1982
• Focus: break code and find errors
• Separate debugging from testing

Pinjia He @ CUHK Shenzhen

Evaluation Era
• 1983-1987
• Focus: evaluate and measure the quality
• When to stop?

Pinjia He @ CUHK Shenzhen

Prevention Era
• 1988-2000
• Testable and non-testable code
• Key: identifying the testing techniques

Pinjia He @ CUHK Shenzhen

Testing and Analysis
• Verify whether your intents were correctly expressed
computationally (verification)

• Testing: run it, and see with your own eyes

• Analysis: analyze it, use logic

• Both could be manual or automated

Pinjia He @ CUHK Shenzhen

Testing and Analysis

[Link]

Pinjia He @ CUHK Shenzhen

Testing
• Testing cannot prove that a
program does not have bugs.

• There could be infinite number of

possible input

• value ? getResult1() : getResult2()

Pinjia He @ CUHK Shenzhen

Analysis
• One kind of analysis: used logic to
divide the possible states of input
and their effect on the program's
behavior (symbolic execution)

• If we write a program to make it

automated, it is program analysis

• Manual: code reviews

Pinjia He @ CUHK Shenzhen

Testing Stages

Pinjia He @ CUHK Shenzhen

Creation of Test Harness

• Test driver
üApply tests to UUT, including setup & clean-up
• Test stub
üPartial, temporary implementation of a component
used by UUT
üMock a missing component back fake data

Pinjia He @ CUHK Shenzhen

Recommend Readings

[Link]

Pinjia He @ CUHK Shenzhen

Unit Testing
• Testing individual units

• Goal: Confirm units are correct & meet intended

functionalities

Pinjia He @ CUHK Shenzhen

Unit Testing Example

Pinjia He @ CUHK Shenzhen

Test Execution
• Execute the tests

Pinjia He @ CUHK Shenzhen

Eight Rules of Testing [M. Fowler]
1. Make sure all tests are 5. Better to write and run
fully automatic and incomplete tests than
check their own results not run complete tests
2. A test suite is a 6. Concentrate your tests
powerful bug detector on boundary conditions
that reduces the time to 7. Don’t forget to test
find bugs raised exceptions when
3. Run tests frequently – things are expected to
every test at least once go wrong
a day 8. Do not let the fear that
4. When you get a bug testing can’t catch all
report, start by writing bugs stop you from
a unit test to expose the writing tests that will
bug catch most bugs
Pinjia He @ CUHK Shenzhen
Testing Stages

Pinjia He @ CUHK Shenzhen

Integration Testing
• Test groups of subsystem & eventually the entire
system

• Goal: Test interfaces between subsystems

Pinjia He @ CUHK Shenzhen

System Testing
• Test the entire system

• Goal: Determine if the system meets the requirements,

both functional and non-functional

Pinjia He @ CUHK Shenzhen

System Testing Stages

Pinjia He @ CUHK Shenzhen

Functional Testing
• Goal: Test functionality of system
p System is treated as a black box

l Test cases are design from requirements

p Based on use cases
p Alternative source: user manual

l Test cases describe

p Input data
p Flow of events
p Results to check
Pinjia He @ CUHK Shenzhen
Acceptance Testing
• Goal: Demonstrate system meets customer requirements
• Performed by the client, not by the developer

• Alpha test
p Client uses the software at the developer’s site
p Software used in controlled setting --- developers ready to fix
bugs

l Beta test
p Conducted at client’s site (developer not present)
p Software gets a realistic workout in target environments

Pinjia He @ CUHK Shenzhen

Independent Testing
• Hard for programmers to accept they made a mistake
p Plus a vested interest in not finding mistakes
p Often stick to the data that makes the program work

l Designing and programming are constructive tasks

p Testers must seek to break the software

l Testing is done best by independent testers

Pinjia He @ CUHK Shenzhen

Testing Steps

Pinjia He @ CUHK Shenzhen

Testing Strategies
• Exhaustive testing: Check UUT for all possible inputs
p Not feasible, even for trivial programs

l Random testing: Select test data randomly & uniformly

l Functional testing: Use requirements to decide test cases

l Structural testing: Use design knowledge about system

structure, algorithms, data structures to determine test
cases that exercise a large portion of the code

Pinjia He @ CUHK Shenzhen

Testing Strategies

Pinjia He @ CUHK Shenzhen

Testing Steps

Pinjia He @ CUHK Shenzhen

Finding representative inputs

Pinjia He @ CUHK Shenzhen

Finding representative inputs
• Divide inputs into
equivalence classes
p Each possible input belongs
to one equivalence class
p Goal: some classes have
higher failure density

l Choose tests for each

equivalence class

Pinjia He @ CUHK Shenzhen

Finding representative inputs
• Divide inputs into
equivalence classes
p Each possible input belongs
to one equivalence class
p Goal: some classes have
higher failure density

l Choose tests for each

equivalence class

Pinjia He @ CUHK Shenzhen

Selecting representative values
• Once partitioned the inputs, need to select concrete
values for the tests for each equivalence class

• Input from a range of valid values

p Below, within, and above the range

• Input from a discrete set of values

p Valid and invalid discrete values
p Instances of each subclass

Pinjia He @ CUHK Shenzhen

Boundary Testing

• Many errors occur at boundaries of the input domain

p Overflows
p Comparisons (‘<’ instead of ‘<=’, etc.)
p Missing emptiness checks (e.g., collections)
p Wrong number of iterations

Pinjia He @ CUHK Shenzhen

Combinatorial Testing
• Many input values: equiv. classes + boundary testing

• Testing all combinations leads to combinatorial

explosion

• Reduce test cases to make effort feasible

p Random selection
p Semantic constraints
p Combinatorial selection

Pinjia He @ CUHK Shenzhen

Semantic Constraints

Pinjia He @ CUHK Shenzhen

Pairwise Combinatorial Testing
• Focus on possible combinations of pairs of inputs

• Example: Consider a method with 4 Boolean parameters

p Combinatorial testing requires 2! = 16 test cases
p Pairwise-combinations testing requires 5 test cases:

TTTT, TFFF, FTFF, FFTF, FFFT

l Can be generalized to k-tuples (k-way testing)

l For n parameters with d values per parameter, the
number of test cases grow logarithmically in n and
quadratic in d
l Results hold for large n & d, and for all k in k-way testing
Pinjia He @ CUHK Shenzhen
Functional Testing: Summary

Pinjia He @ CUHK Shenzhen