Introduction to Software Testing

CHAPTER 4
SOFTWARE ERROR
Content
□ What is a Software Error?
□ Finding, Reproducing and Analyzing a
Software Error
□ Reporting a Software Error
□ A Common Bug Life
□ Bug Management System

Lê Ngô Thục Vi Introduction to Software Testing 2

What is a Software Error
□ A software error is present when the program
does not do what its user reasonably expects it to
do.
□ It is fair and reasonable to report any deviation
from high quality as a software error.

The existence of software errors reflects an

impediment on the quality of the product, but
does not necessarily imply that the
developers. are incompetent
Lê Ngô Thục Vi Introduction to Software Testing 3
Common sources of error
□ Coding error: The program doesn’t do what the
programmer would expect it to do.
□ Design Issue: It’s doing what the programmer
intended, but a reasonable customer would be
confused or unhappy with it.
□ Requirements Issue: The program is well designed
and well implemented, but it won’t meet one of the
customer’s requirements.
□ Documentation or Specification/ Code Mismatch:
Sometimes the spec is right; sometimes the code is
right and the spec should be changed.
Lê Ngô Thục Vi Introduction to Software Testing 4
13 common types of Software error
□ 1/ User Interface
□ 2/ Error Handling
□ 3/ Boundary-Related
□ 4/ Calculation
□ 5/ Initial and Later States
□ 6/ Control Flow
□ 7/ Handling or Interpreting Data

Lê Ngô Thục Vi Introduction to Software Testing 5

13 common types of Software error
□ 8/ Race Conditions
□ 9/ Load Conditions
□ 10/ Hardware/Environment Compatibility
□ 11/ Source, Version, and ID Control
□ 12/Testing
□ 13/ Documentation

Lê Ngô Thục Vi Introduction to Software Testing 6

Content
□ What is a Software Error?
□ Finding, Reproducing and Analyzing
a Software Error
□ Reporting a Software Error
□ A Common Bug Life
□ Bug Management System

Lê Ngô Thục Vi Introduction to Software Testing 7

What to do in a bug finding process?
1. Reproduce the error
2. Analyze the error
3. Report the error

Lê Ngô Thục Vi Introduction to Software Testing 8

Reproducing a Software Error
□ Some bugs are always reproducible, but
some are just sometimes or even rarely.
□ Bugs don’t just miraculously happen and
then go away. If a bug happens
intermittently, it might be under some
certain conditions.

Lê Ngô Thục Vi Introduction to Software Testing 9

Reproducing a Software Error
□ We hypothesize the cause, then we try
to re-create the conditions that make the
error visible.
□ If the bug is non-reproducible, you
should always report it, but describe
your steps and observations precisely.
Programmers will often figure them out.

Lê Ngô Thục Vi Introduction to Software Testing 10

Why is a bug hard to reproduce?
□ Memory dependent
□ Memory corruption
□ Configuration dependent
□ Timing related
□ Initialization
□ Data flow dependent
□ Control flow dependent
□ Error condition dependent
□ Multi-threading dependent
□ Special cases: Algorithm, Dates..

Lê Ngô Thục Vi Introduction to Software Testing 11

Making an error reproducing
□ Write down everything you remember about what you
did the first time.
□ Note which things you are sure of and which are good
guesses.
□ Note what else you did before starting on the series of
steps that led to this bug.
□ Review similar problem reports you’ve come across
before.
□ Use tools such as capture/replay program, debugger,
debug-logger, videotape, or monitoring utilities
□ Talk to the programmer and/or read the code.
Lê Ngô Thục Vi Introduction to Software Testing 12
Analyzing a Software error
Why Analyze a Reproducible Bug?
□ Analyze bugs in order to:
■ Make your communication effective:
□ Make sure you are reporting what you think
you are reporting.
□ Make sure that questionable side issues are
thoroughly investigated.
□ Create accountability.

Lê Ngô Thục Vi Introduction to Software Testing 13

Analyzing a Software error
Why Analyze a Reproducible Bug?
□ Analyze bugs in order to:
■ Support the making of business decisions;
■ Avoid wasting the time of the programming
and management staff;
■ Find more bugs.

Lê Ngô Thục Vi Introduction to Software Testing 14

Analyzing a Reproducible error
□ Start by making sure the error is
reproducible.
1) Describe how to get the program into a
known state.
2) Specify an exact series of steps that expose
the problem.
3) Test your steps to make sure that you can
reproduce the problem if you do exactly (and
only) what it says in the bug report.

Lê Ngô Thục Vi Introduction to Software Testing 15

Content
□ What is a Software Error?
□ Finding, Reproducing and Analyzing a
Software Error
□ Reporting a Software Error
□ A Common Bug Life
□ Bug Management System

Lê Ngô Thục Vi Introduction to Software Testing 16

 How to report a software error

Bug reports are your primary work product.

Lê Ngô Thục Vi Introduction to Software Testing 17

A useful bug report
■ Written
■ Uniquely numbered (ID required)
■ Simple (non-compound - one bug per
report)
■ Understandable
■ Reproducible
■ Non-judgmental

Lê Ngô Thục Vi Introduction to Software Testing 18

Report content
□ Summary
□ Description
□ Steps to Reproduce – including expected behavior
and observed behavior
□ Reproducible
□ Severity
□ Frequency
□ Priority
□ Keyword (Functional Area)
□ Resolution
Lê Ngô Thục Vi Introduction to Software Testing 19
Bug summary
□ This one-line description of the problem is the most
important part of the report.
□ The ideal summary tells the reader what the bug is,
what caused the bug, what part of the program it’s
from and what its worst consequence is. It runs from 8
to 15 words long.
□ We use the following syntax for writing the problem
summary:
Symptom + Action + Operating Condition

Lê Ngô Thục Vi Introduction to Software Testing 20

Bug summary
Some good examples of Bug Summary:
1. Run-time error when submitting the Contact Us form
with first name of more than 256 characters.
2. The main dialog box is resizable
3. Help button does not bring up Help page.
4. Maximize button is still active while the dialog box is
maximized.

Lê Ngô Thục Vi Introduction to Software Testing 21

Bug summary
Some not-so-good examples of Bug Summary:
1. Software fails
2. Can't install
3. Back button does not work
4. My browser crashed. I think I was on [Link]. I
play golf with Bill Gates, so you better fix this
problem, or I'll report you to him. By the way, your
Back icon looks like a squashed rodent. Too ugly.
And my grandmother's home page is all messed up
in your browser.

Lê Ngô Thục Vi Introduction to Software Testing 22

Bug description & steps to reproduce
□ First, describe the problem. What is the bug? Don’t rely
on the summary to do this – a short line sometimes
cannot state all what you want to say.
□ Next, go through the steps that you use to recreate this
bug. Start from a known place (e.g. boot the program)
and then describe each step until you hit the bug.
□ Describe the erroneous behavior and if necessary,
explain what should have happened. (Why is this a bug?
Be clear.)

Lê Ngô Thục Vi Introduction to Software Testing 23

Reproducible
□ You may or may not have this on your form,
but you should always provide this
information.
■ Never say Yes unless you have recreated the bug
■ If you have tried and tried but you can not recreate
the bug, say No. Then explain what steps you tried
in your attempt to recreate it.
■ If the bug appears sporadically and you do not yet
know why, say “Sometimes” and explain.

Lê Ngô Thục Vi Introduction to Software Testing 24

Severity
□ You will have to rate the bug’s seriousness. Many
companies use a three-level rating:
■ 1 - Critical: This means fatal to the release
■ 2 - Serious: It’s a bad bug, but it doesn’t cause
data loss or a program crash.
■ 3 - Minor: It’s a bug, but it’s not a big deal.

□ Level rating depend on each company.

□ Many companies sort their summary reports by

severity, so you want to fill in this field thoughtfully.

Lê Ngô Thục Vi Introduction to Software Testing 25

Frequency
□ Frequency is usually graded by assessing the
following three characteristics:
■ How easy is it for the user to encounter the bug
■ How frequent would the user encounter the bug
■ How often the buggy feature is used

□ Many companies use a three-level rating:

■ 1 - Always
■ 2 - Often
■ 3 - Seldom

Lê Ngô Thục Vi Introduction to Software Testing 26

Priority
□ Priority rating is either automatically
generated by the bug tracking system by
assessing the Severity and the Frequency
ratings or assigned only by the project
manager.

Lê Ngô Thục Vi Introduction to Software Testing 27

Keyword (Functional Area)
□ You may have to categorize the bug according
to its functional area.
□ The tracking system should include a list of
the possible keywords.
□ It is important to categorize these bugs
consistently. Burying a bug in the wrong
category can lead to its never getting fixed.
□ If you’re creating the list of functional areas,
keep it short, perhaps 20 areas.

Lê Ngô Thục Vi Introduction to Software Testing 28

Resolution
□ The project manager has the privilege to assign most of the resolutions
in this field.
□ Common resolutions include:
■ New: The newly submitted bug
■ To Be Distributed: The bug is waiting to be distributed
■ To Be Fixed: The bug is being fixed.
■ QA Info Request: The bug needs more clarification from Tester.
■ Developer Info Request: The bug needs more clarification from
Developer.
■ Not Reproducible: The bug cannot be reproduced.
■ Fixed: The bug is fixed.
■ Not a Problem: The application works as it is supposed to.
■ Duplicate: The bug is just a repeat of another bug.
■ Deferred: The bug will be fixed in a later release.
■ Feature Limitation: There is some feature limitations that do not
allow to fix the bug.

Lê Ngô Thục Vi Introduction to Software Testing 29

Bug report simple template

Lê Ngô Thục Vi Introduction to Software Testing 30

Improve the bug report
□ But you may be able to improve the report in four
ways:

1) You might be able to simplify the report by:

❖ eliminating unnecessary or irrelevant steps.
❖ splitting it into two reports.
2) You might be able to strengthen the report by
❖ showing that it is more serious than it first
appears.
❖ showing that it is more general than it first
appears.

Lê Ngô Thục Vi Introduction to Software Testing 31

1. Eliminate unnecessary steps
❑ Look for critical steps
■ If you’ve found what looks like a critical step, try to
eliminate almost everything else from the bug
report.
■ Go directly from that step to the last one (or few)
that shows the bug.

Lê Ngô Thục Vi Introduction to Software Testing 32

2. Split the report in two
□ When you see two related problems, you might report
them together on the same report -> too long report
□ If this makes the report become confusing, write two
reports instead.
□ When you report related problems, it’s a courtesy to
cross-reference them. For example:
Related bug -- see Report # xxx

Lê Ngô Thục Vi Introduction to Software Testing 33

3. Show that it is more serious
□ Look for follow-up errors:
■ Keep using the program after you get this problem. Does
anything else happen?

□ Look for nastier variants:

■ Vary the conditions under which you got the bug.

□ Look for nastier configurations:

■ Sometimes a bug will show itself as more serious if you run
the program with less memory, a higher resolution output,
more graphics on a page, etc.

Lê Ngô Thục Vi Introduction to Software Testing 34

4. Show that it is more general
□ Look for alternative paths to the same
problem:
■ Sometimes the bug can happen in some
alternative path.

□ Look for configuration dependence:

■ Sometimes the bug happens because of some
hardware configuration.

Lê Ngô Thục Vi Introduction to Software Testing 35

Content
□ What is a Software Error?
□ Finding, Reproducing and Analyzing a
Software Error
□ Reporting a Software Error
□ A Common Bug Life
□ Bug Management System

Lê Ngô Thục Vi Introduction to Software Testing 36

Lê Ngô Thục Vi Introduction to Software Testing 37
Content
□ What is a Software Error?
□ Finding, Reproducing and Analyzing a
Software Error
□ Reporting a Software Error
□ A Common Bug Life
□ Bug Management System

Lê Ngô Thục Vi Introduction to Software Testing 38

Introduction
□ A bug tracking/management system is a
software application designed to help QA, QC,
testers and programmers keep track of
reported software bugs in their work
□ Many bug-tracking systems used by most
open source software projects, allow users to
enter bug reports directly.

Lê Ngô Thục Vi Introduction to Software Testing 39

Components
□ The major component is a database records
facts about known bugs:
■ The time a bug was reported
■ Bug’s severity
■ The erroneous program behavior
■ The details on how to reproduce the bug
■ The identity of the person who reported bug
■ The programmer who will fix bug

Lê Ngô Thục Vi Introduction to Software Testing 40

Components
□ Typical bug tracking systems support the
concept of the life cycle for a bug which is
tracked through status assigned to the bug:
■ Configure permissions based on status
■ Configure the bug statuses
■ Configure workflow of bug statuses
■ Move the bug to another status

Lê Ngô Thục Vi Introduction to Software Testing 41

Lê Ngô Thục Vi Introduction to Software Testing 42
Work at home
□ Read:
■ Textbook: chapter 5, 15, Appendix

Lê Ngô Thục Vi Introduction to Software Testing 43