Software Debugging Techniques

Stargate
UIC
nome e istituto reale verranno aggiunti in seguito
la bibliografia la sto sistemando ora.

Introduction
According to a very popular definition [1], debugging is a methodical process of finding and reducing
the number of bugs, or defects, in a computer program. Yet I am sure most people involved in spotting
and removing those defects would define it as an art rather then a method.
All bugs stem from a one basic premise: something you thought was right, was in fact wrong.
Due to this simple principle, truly bizarre bugs can defy logic, making debugging software challenging.
The typical behaviour of many inexperienced programmers is to freeze when unexpected problems arise.
Without a definite process to follow, solving problems seems impossible to them. The most obvious
reaction to such a situation is to make some random changes to the code, hoping to that it will start to
work again. The issue is simple: the programmers have no idea of how to approach debugging.
This lecture is an attempt to review some techniques and tools to assist in debugging for non-
experienced programmers. It contains both tips to solve problems and suggestions to prevent bugs to
manifest themselves. Finding your bug is a process of confirming what is working until you find some-
thing that is wrong. Therefore, do not expect to get an algorithm good for every situation: there is no
silver bullet for debugging. Experience and ingenuity are a part of the quest for bugs, but also disciplined
usage of tools.
The importance of a method of finding errors and fixing them during the life-cycle of a software
product cannot be stressed enough. Testing and debugging are fundamental parts of programmer’s ev-
eryday activity but some people still consider it an annoying option. When not carried out properly,
consequences can be dreadful. For example, in 1998, a crew member of the guided-missile cruiser USS
Yorktown mistakenly entered a zero for a data value, which resulted in a division by zero. The error
cascaded and eventually shut down the ship’s propulsion system. The ship was dead in the water for
several hours because a program didn’t check for valid input [2].
Bugs can also be very expensive. In 1999, the 125 million dollars Mars Climate Orbiter was
assumed lost by officials at NASA. The failure responsible for loss of the orbiter was attributed to a
failure of NASA’s system engineering process. The process did not specify the system of measurement
to be used on the project. As a result, one of the development teams used Imperial measurement while
the other used the metric system of measurement. When parameters from one module were passed to
another, during orbit navigation correction, no conversion was performed, resulting in the loss of the
craft [3].
These two famous bugs, as others in history of software [4], should make you understand the
importance of finding errors in software: it is not just an unavoidable part in the development cycle but
vital part of every software system’s lifespan.
The lecture starts with a general introduction to debugging containing some useful concepts for
programmers approaching this subject for the first time. It goes on with a detailed review of general
debugging techniques, not bound to any specific kind of software. Since C++ is the main language
commonly employed in particle physics nowadays, the final part is dedicated to analysing problems
related to the use of this programming language.
The lecture contains several examples and source files, written in C++. They can be compiled and
run and provide a starting point for personal experimentation. Although the examples refer very often to
a Unix-like operating system, the underlying concepts and techniques are platform independent.
 1 General Concepts About Debugging
After many days of brainstorming, designing and coding, you finally have a wonderful piece of code.
You compile it and you run it. Everything seems pretty straightforward but unfortunately it doesn’t work!
And now? Now the great fun starts! Time to dig into the wonderful world of debugging.
Despite being the realm of ingenuity and uncertainty, a debugging process can be divided into four
main steps:

1. Localising a bug;
2. Classifying a bug;
3. Understanding a bug;
4. Repairing a bug.

1.1 Localising a bug

A typical attitude of inexperienced programmers towards bugs is to consider their localisation an easy
task: they notice their code does not do what they expected, and they are led astray by their confidence
in knowing what their code should do. This confidence is completely deceptive because spotting a bug
can be very difficult. Remember the definition of bug I gave earlier: all bugs stem from the premise that
something you thought was right, was in fact wrong. Here a very simple example of a possible problem.

Listing 1: Bad naming convention causing an endless loop.

1 / / An e x a m p l e o f a p r o b l e m o f s c o p e
2 void c ( void ) ; / / f u n c t i o n p r o t o t y p e
3 int x = 1; / / global variable
4 i n t main ( )
5 {
6 i n t x = 5 ; / / l o c a l t o main
7 / / Some o t h e r c o d e
8 while ( x < 100)
9 c ( ) ; / / c ( ) uses global
10 / / Some o t h e r c o d e
11 return 0;
12 }
13 void c ( void )
14 {
15 / / Some o t h e r c o d e
16 x *= 1 0 ;
17 / / Some o t h e r c o d e
18 }

This program contains a typical endless loop. The main function consists of a loop calling the
function c() as long as the variable x is lower than 100. c() is supposed to increment the variable x,
defined globally. Unfortunately, due to a poor naming convention, a new declaration in the main scope
cause x not to be incremented as expected, producing an endless loop. This example also illustrates the
danger of giving to variables in different scopes the same names: despite language standards, this can be
a big source of troubles.
Noticing a bug implies testing. Testing should be performed with discipline and, when possible,
automatically, for example after each build of your code. In case of a test failure you have to see what
went wrong, so prepare your tests carefully. I will not talk about testing in this lecture, but I strongly
recommend to learn at least the basics of automatic software testing [6].

2
1.2 Classifying a bug
Despite the appearance, bugs have often a common background. This allows to attempt a quite coarse, but
sometimes useful, classification. The list is arranged in order of increasing difficulty (which fortunately
means in order of decreasing frequency).
Syntactical Errors should be easily caught by your compiler. I say "should" because compilers, beside
being very complicated, can be buggy themselves. In any case, always remember that quite often
the problem might not be at the exact position indicated by the compiler error message.
Build Errors derive from linking object files which were not rebuilt after a change in some source files.
These problems can easily be avoided by using tools to drive software building, like GNU Make.
Basic Semantic Errors comprise using uninitialised variables, dead code and problems with variable
types. A compiler can bring them to your attention, although it usually has to be asked to esplicitely
through flags (cp. 2.1).
Semantic Errors include using wrong variables or operators (e.g. & instead of && in C++). No tool
can catch these problems, because they are syntactically correct statements, although logically
wrong. You need a test case or a debugger (see par. 2.8) to spot them.
I must mention, among the others, a funny “physical” classification. Some programmers tend
to distinguish between Bohrbugs and Heisenbugs. Bohrbugs are deterministic: a particular input will
always manifest them with the same result. Heisenbugs are random: difficult to reproduce reliably, since
they seem to depend on environmental factors (e.g. a particular memory allocation, the way the operating
system schedules processes, the phase of the moon and so on). In C++ a Heisenbug is very often the
result of an error with pointers (cp. 3.3).

1.3 Understanding a bug

Make sure you fully understand a bug before attempting to fix it. Trying to fix a bug before understanding
it completely could end in provoking even more damage to the code, since the problem could change form
and manifest itself somewhere else, maybe randomly. Again, a typical example is memory corruption:
if you think your memory was corrupted during the execution of some algorithm, check all the data
involved in the algorithm before trying to change them. You can read more about memory corruption in
3.3.
The following check list is useful to assure a correct approach to the investigation:

– assure you found the real source of the problem and not only a symptom;
– check if you made similar mistakes (especially wrong assumptions) elsewhere in the code;
– verify you found just a programming error and not a more fundamental problem (e.g. an incorrect
algorithm).

1.4 Repairing a bug

The final step in the debugging process is bug fixing. Repairing a bug is more than modifying code.
Make sure you document your fix in the code and test it properly. More important, try to learn from
your mistakes. If the bug was something you didn’t see before, you could fill a small file with detailed
explanations about the way you discovered and corrected it. Again, a checklist can be a useful aid.
Several points are worth mentioning:

– how you noticed the bug, to help you in writing a test case;
– how you tracked it down, to give you a better insight on the approach to choose in similar circum-
stances;
– what type of bug you encountered;

3
 – if you encounter this bug often, to set up a strategy to prevent it from recurring;
– if your assumptions were unjustified; this is often the main reason why tracking a bug is so time
consuming.

What you have learnt from your investigation is valuable, therefore you should try to communicate
it with your colleagues in a concise but meaningful form.

2 General Debugging Techniques

As I said before, debugging is often the realm of ingenuity and uncertainty. Yet a number of tricks can
be adopted in your daily programming activity to ease your hunt for problems.

2.1 Exploiting Compiler Features

A good compiler can do some static analysis on your code. Static code analysis is the analysis of software
that is performed without actually executing programs built from that software. Static analysis can help
in detecting a number of basic semantic problems, e.g. type mismatch or dead code (code that will never
be executed).
I recommend having a look at the user manual of the compiler you employ, where all the features
should be documented. For gcc there are a number of options that affect what static analysis can be
performed. They are usually divided into two classes: warning options and optimisation flags. As far as
warning options are concerned, here is the list of the ones I consider as most useful:
Wall enables all the warnings about constructions that some users consider questionable and easy to
avoid. This also enables some language-specific warnings;
Wshadow warns whenever a local variable shadows another local variable, parameter or global variable
or whenever a built-in function is shadowed. With this flag, the bug shown in listing 1 could have
been easily avoided;
Wpointer-arith warns about anything that depends on the size of a function type or of void;
Wcast-qual warns whenever a pointer is cast so as to remove a type qualifier from the target type, for
example if a const char* is cast to an ordinary char*. Removing qualifier can be a painful
source of troubles;
Wcast-align warns whenever a pointer is cast such that the required alignment of the target is increased,
for example if a char * is cast to an int * on machines where integers can only be accessed at
two- or four-byte boundaries;
Wstrictprototype this option is valid only for programs written in C and warns if a function is declared
or defined without specifying the argument types.
I recommend to maintain the flag Wall active at all times, while the others are especially suitable when
compiling new code.
Compilers support also a number of optimisations. Some of these trigger the compiler to do
extensive code flow analysis, removing dead code. Nevertheless programmers must understand that
optimisation works, to some extent, against debugging. Optimisation implies a lot of code flow analysis,
which ends up in rearranging code statements. It means that once optimised, your code could be different
from what you originally wrote, making debugging virtually impossible. Therefore, optimisation flags
should be turned on only when the code appears to be reasonably bug free.
As far as gcc is concerned, optimisation level is identified by a number. For standard use, I do not
recommend to use an optimisation level higher then 2, unless you know what you are doing, since higher
levels could contain experimental optimisation which could generate bad code.
For more information about the various options supported by gcc, consult gcc manual [11].

4
 2.2 Reading The Right Documentation
This seems quite an obvious tip, but too often I see unexperienced programmers reading the wrong papers
looking for hints about the task they have to accomplish. Take the time to find at your fingertips relevant
documentation for your task, your tools, the libraries and the algorithms you employ. Obviously you
do not need to know everything, you just need to be aware of what documentation is relevant to your
purpose.
As far as documentation is concerned, the most important distinction is between tutorials and
references. A tutorial is a pedagogical paper, usually with plenty of examples. It doesn’t assume any
previous knowledge of the topic and its first aim is to convey ideas about the subject. Reference manuals,
on the contrary, are comprehensive and exhaustive descriptions, which allow you to find the answers to
your questions through indexes and cross-references.
In the world of programming, all these types of document are usually in electronic format. Make
sure that the reference documentation is up to date, accurate and corresponding to your problems and
tools: looking up in a wrong reference manual could end up in trying to use a feature that is not supported
by the current version of your tool, for example.

2.3 The Abused cout Debugging Technique

The cout technique takes its names from the C++ statement for printing on the standard output stream
(usually the terminal screen). It consists of adding print statements in the code to track the control
flow and data values during code execution. Although it is the favourite technique of all the newbies, it
is unbelievable how many experienced programmers still refuse to evolve and abandon this absolutely
time-wasting and very ad-hoc method.
Despite its popularity, this technique has strong disadvantages. First of all, it is very ad-hoc,
because code is temporary, to be removed as soon as the bug is fixed. A new bug means a new insertion,
making it a waste of time. In debugging as well as in coding, the professional should aim to find reusable
solutions whenever possible. Printing statements are not reusable, and so are deprecated. As we will see
shortly, there are more effective ways to track the control flow through messages. In addition, printing
statements clobber the normal output of the program, making it extremely confused. They also slow
the program down considerably: accessing to the outputting peripherals becomes a bottleneck. Finally,
often they do not help at all, because for performance reasons, output is usually buffered and, in case of
crash, the buffer gets lost and you miss exactly the important information you were looking for, possibly
causing you to start the debugging process in the wrong place.
If you consider using this kind of technique, please check out the use of assertion (par. 2.5) and
of a debugger (par. 2.8), much more effective and time saving. In some (very few) circumstances cout
debugging can be appropriate, although it can always be replaced by other techniques. If you want to use
it, here are some tips. To begin with, output must be produced on the standard error, because this channel
is unbuffered and it is less likely to miss the last information before a crash. Then, do not use printing
statements directly: define a macro around them (as illustrated in listing 2) so to switch debugging code
on and off easily. Finally, use debugging levels to manage the amount of debugging information. More
on debugging levels can be found in par. 2.4.1.

Listing 2: An example of cout technique - Declaration

1 # i f n d e f DEBUG_H
2 # d e f i n e DEBUG_H
3 # i n c l u d e < s t d a r g . h>
4 # i f d e f i n e d (NDEBUG) && d e f i n e d ( __GNUC__ )
5 / * g c c ’ s cpp h a s e x t e n s i o n s ; i t a l l o w s f o r macros w i t h a v a r i a b l e
6 number o f a r g u m e n t s . We u s e t h i s e x t e n s i o n h e r e t o p r e p r o c e s s
7 pmesg away . * /
8 # d e f i n e pmesg ( l e v e l , f o r m a t , a r g s . . . ) ( ( v o i d ) 0 )

5
 9 #else
10 v o i d pmesg ( i n t l e v e l , char * f o r m a t , . . . ) ;
11 / * p r i n t a message , i f i t i s c o n s i d e r e d s i g n i f i c a n t enough A d a p t e d
12 f r o m [K&R2 ] , p . 174 * /
13 # endif
14 # e n d i f / * DEBUG_H * /

Listing 3: An example of cout technique - Implementation

1 # i n c l u d e " debug . h "
2 # i n c l u d e < s t d i o . h>
3
4 e x t e r n i n t m s g l e v e l ; / * t h e h i g h e r , t h e more m e s s a g e s . . . */
5
6 # i f d e f i n e d (NDEBUG) && d e f i n e d ( __GNUC__ )
7 / * N o t h i n g . pmesg h a s b e e n " d e f i n e d away " i n debug . h a l r e a d y . * /
8 #else
9 v o i d pmesg ( i n t l e v e l , char * f o r m a t , . . . ) {
10 # i f d e f NDEBUG
11 / * Empty body , s o a good c o m p i l e r w i l l o p t i m i s e c a l l s
12 t o pmesg away * /
13 #else
14 v a_ l is t args ;
15
16 i f ( level >msglevel )
17 return ;
18
19 v a _ s t a r t ( args , format ) ;
20 v f p r i n t f ( s t d e r r , format , args ) ;
21 va_end ( a r g s ) ;
22 # e n d i f / * NDEBUG * /
23 # e n d i f / * NDEBUG && __GNUC__ * /
24 }

Here, msglevel is a global variable, which you have to define, that controls how much debugging
output is done. You can then use pmesg(100, "Foo is %l\n", foo) to print the value of foo in case
msglevel is set to 100 or more. Note that you can remove all this debugging code from your executable
by adding -DNDEBUG to the preprocessor flags (sec. 3.2): for GCC, the preprocessor will remove it, and
for other compilers pmesg will have an empty body, so that calls to it can be optimised away by the
compiler. This trick was taken from the file assert.h (sec. 2.5).

2.4 Logging
Logging takes the concept of printing messages, expressed in the previous paragraph, one step further.
Logging is a common aid to debugging. Everyone who has tried at least once to solve some system-
related problems (e.g. at machine start-up) knows how useful a log file can be. Logging means automat-
ically recording information messages or events in order to monitor the status of your program and to
diagnose problems. It is heavily used by daemons and services, exactly because their failure can affect
the correct operation of the whole system. Logging is a real solution to the cout technique. It can even
form the basis of software auditing, that is the evaluation of the product to ascertain its reliability.
A great example of a logging service is the Linux syslog program, provided by every distribution.
If you haven’t seen a log service before, I strongly recommend having a look at it. Studying the way
syslog works will provide you with a powerful example, not to mention the expertise to solve problems
with kernel, daemons and subsystems (like mail, news and web servers) on your own machine.

6
2.4.1 log4cpp C++ Logging
A way of setting up logging service in a C++ program is to employ a library called “Log for C++”
(Log4cpp for short). Log4cpp is a library of C++ classes for flexible logging to files, syslog, IDSA and
other destinations. According to the authors, it is modeled after the Log4j Java library, staying as close
to their API as is reasonable. For a further discussion about Log4cpp I recommend the article [12]. Here
I will give you a brief overview about this small but effective package.
Log4cpp has 3 main components:
– Layouts
– Appenders
– Categories
A layout class controls the appearance of the output messages. log4cpp provides the user with
some predefined layout classes, but of course you may derive your own classes from the basic class
Layout, to specify any style of output message you want.
An appender class writes the trace message out to some device. The messages have been formatted
by a layout object. Again, log4cpp provides the user with some standard classes to post messages to
standard output, a named file or a string buffer. The Appender class works closely with the Layout class,
and once again you may derive your own appender classes if you wish to log to a different channel: for
example a socket, a shared memory buffer or some sort of delayed write device.
A category class does the actual logging. The two main parts of a category are its appenders and
its priority. Priority controls which messages can be logged by a particular class. When a category object
is created, it begins with a default appender to standard output and a default priority of none. One or
more appenders can be added to the list of destinations for logging.
The priority of a category can be set to
1. NOTSET
2. DEBUG
3. INFO
4. NOTICE
5. WARN
6. ERROR
7. CRIT
8. ALERT
9. FATAL / EMERG
in ascending order of importance level. FATAL and EMERG are two names for the same highest level
of importance. Each message is logged to a category object. The category object has a priority level.
The message itself also has a priority level as it wends its way to the log. If the priority of the message
is greater than, or equal to, the priority of the category, then logging takes place, otherwise the message
is ignored. NOTSET is the lowest and if a category object is left with a NOTSET priority, it will accept
and log any message.
Messages can be given any of these priorities except NOTSET. Therefore if a category has been set
to level WARN, then messages with levels DEBUG, INFO and NOTICE will not be logged. Messages
set to WARN, ERROR, CRIT, ALERT, FATAL or EMERG will be logged.

2.4.2 Log4cpp Example

Here is a practical example to illustrate the usage of log4cpp. There are six initial steps to using a log4cpp
log:

7
 1. Instantiate an appender object that will append to a log file
log4cpp::Appender* app =
new log4cpp::FileAppender("FileAppender","/logs/[Link]");
2. Instantiate a layout object
log4cpp::Layout* layout = new log4cpp::BasicLayout();
3. Attach the layout object to the appender
app->setLayout(layout);
4. Instantiate a category object by calling the static function
log4cpp::Category main_cat = log4cpp::Category::getInstance("main_cat");
5. Attach the appender object to the category as an additional appender (in addition to the default
standard out appender), or set Additivity to false first and install the appender as the one and only
appender for that category
main_cat.setAppender(app);
6. Set a priority for the category
main_cat.setPriority(log4cpp::Priority::INFO);

You can now write a small test program to see the effect of the previous instructions. For example,
if you include the following statements in your program

main_cat.info("This is some info");

main_cat.debug("This debug message will fail to write");
main_cat.alert("All hands abandon ship");
main_cat.log(log4cpp::Priority::WARN, "This will be a logged warning");
main_cat.log(priority,"Importance depends on context");

you will get the following result

995871335 INFO main_cat : This is some info

995871335 PANIC main_cat : All hands abandon ship
995871335 WARN main_cat : This will be a logged warning
995871335 ALERT main_cat : Importance depends on context

As you can see, you can log a message by using the member function log() with a priority. The
message would not be logged if its priority is lower than the priority of the category. You can notice
how the debug message is not recorded because the category priority is set to INFO. You can find other
examples in the cited paper.

2.5 Defensive Programming

If you take a look at your code, you will see that in every part you make a lot of assumptions about other
parts. Assertions are expressions which should evaluate to be true at a specific point in your code. If
an assertion fails, you have found a problem. The problem could possibly be in the assertion, but more
likely it will be in the code. The important point to remember about assertions is that it make no sense to
execute a program after an assertion fails
Writing assertions in your code makes your assumptions explicit. In C/C++ you can include the
header file assert.h and write the expression you want to assert as macro argument, e.g. assert(var > 0)
Your program will be aborted when an assertion fails, and the failure reported by a message stating the
exact line of code and the exact file of the assertion.
Since assert is a macro, it can be easily removed from the final version of your code by compiling
it out. If you use gcc, you must use the preprocessor flag -DNDEBUG.

8
2.6 ACI Debugging Technique
This paragraph could be misunderstood for a joke. On the contrary it is serious and I invite you not
to underestimate the power of this technique with a funny name. Its name derives from the acronym
"Automobile Club d’Italia", an Italian organisation that helps with car trouble. If you are driving on a
motorway and your car gets stuck, all you have to do is to call ACI and a helpful mechanic will quickly
come to you to drag you out of troubles. What is all this to do with a debugging technique?
When you are in big troubles, and you don’t really see a way out, remember the following golden
rule: the best way to learn something is to teach it. This very simple principle is the key of the ACI
technique. In ACI debugging you must find a bystander and explain to him how your code works.
Believe it or not it is a successful technique, because calling a "mechanic" and illustrating the problem
to him forces you to rethink your assumption and explain what is really happening inside your code.
Finally this technique could be employed as a form of peer review.

2.7 Walking Through The Code

This technique is quite similar to the ACI technique, with the exception that it doesn’t rely on a bystander.
The recipe is quite simple as well. When you find yourself in complete darkness and you haven’t the
slightest idea of what is going wrong you must print your code, leave your terminal and go to the cafete-
ria. After choosing your favourite drink, possibly with caffeine and sugar, read your code and annotate
it carefully. Understanding what a program is doing without actually running it is a valuable skill a
programmer must develop.

2.8 The Debugger

When every other checking tool fails to detect the problem, then it is debugger’s turn. A debugger allows
to work through the code line-by-line to find out where and why it is going wrong. It allows you to work
interactively, control the execution of the program, stop it at various times, inspect variables, change
code flow whilst running.
In order to make use of a debugger, a program must be compiled with debugging information
inserted. This information is provided by debugging symbols included by the compiler in your binaries.
Debugging symbols describe where functions and variables are stored in memory. Don’t be afraid of
compiling your program with the debug flag on: an executable with debugging symbols can run as a
normal program, it is just slightly slower.
An important feature of debuggers is the possibility to set breakpoints. Breakpoints stop program
execution on demand: the program runs normally until it is about to execute the piece of code at the same
address of the breakpoint. At that point it drops back into the debugger for us to look at variables, or con-
tinue stepping through the code. Breakpoints are fundamental in interactive debugging, and accordingly
have many options associated with them. They can be set up on a specific line number, at the beginning
of a function, at a specific address, or conditionally (i.e. as soon as a condition is verified).
After stopping the program as a consequence of a breakpoint, a debugger can resume its execution.
There are several ways in which this can be done. The debugger can execute just the next program line
stepping over any function calls in the line. This way any call will be executed in one go, as if it were a
single instruction. Alternatively, the debugger can step into a function call, executing also its code line
by line. Obviously, the debugger can also go on running your program without performing any actions.
Another important feature that all decent debuggers must offer is the possibility to set watchpoints.
Watchpoints are particular type of breakpoints which stop the code whenever a variable changes, even
if the line doesn’t reference the variable explicitly by name. Instead, a watchpoint looks at the memory
address of the variable and alerts you when something is written to it.
In large programs, adding breakpoints for every iteration of a loop is prohibitive. It is not necessary

9
to step through each one in turn: you can employ a technique known as binary split. You must place a
breakpoint at the last line of the first half of the code, and run the code. If the problem has not manifested
itself, then the fault is likely to be within the second half. From here, you can repeat the procedure with
the area where the problem is supposed to be, reducing the area under test at each iteration, until you are
down to just one line, or a sufficiently small routine that can be stepped through line-by-line. A binary
split can limit the search area of a 1000 line program to just 10 steps!
Algorithm implementation errors are reasonably easy to track down with a debugger. You can just
step through looking for an invalid state or bad data (the last statement to execute either is itself wrong
or at least points you at the problem)

3 C/C++ Related Problems and Solutions

In the last part of this lecture, the focus is on problems arising when programming with C or C++
languages. C++ in now the most commonly used language in high energy physics, therefore it seems
quite appropriate to me to devote a large part of my review on debugging techniques to some common
problems that inexperienced programmers will have to deal with since day one of their activity.

3.1 C/C++ Build Process

Before examining the most common problem genereted by C/C++ programming, it is useful to recap the
steps involved in building and running a C/C++ program. C/C++ programs can be built incrementally, i.e.
the building process can be split in smaller steps. In a Unix environment, building is usually composed
of 5 steps:
Preprocessing During the preprocessing phase inclusions of header files are processed and macros are
expanded; the output of preprocessing is still pure C/C++ code.
Compiling Compilation is the translation of pure C/C++ code into assembly language [?].
Assembling The assembly code is translated into binary object code. The result is usually a file with a
.o extension.
Linking Linker’s task is to combines a number of object files and libraries to produce executables or
libraries.
Dynamic Loading libraries This last step consists of loading libraries (or library parts) required by a
dynamically linked executable prior to actual running it.

3.2 Preprocessor
The C/C++ preprocessor is the program that expands macros, declares dependencies and drives con-
ditional compilation. All the preprocessor operations are performed at textual level. This can make
tracking down missing declaration difficult. It could also lead to semantic problems. If you suspect a
preprocessing problem, let the preprocessor expand the file for examination. Since the output of the
preprocessor is just pure source code, debug can be done without any special tool: an editor is enough!
If you would like to try a real example in the Unix domain, you can use gcc with the option -E.
This option make gcc stop a after the preprocessing stage without running the compiler. The output is
preprocessed source code, which is sent to the standard output. You can redirect the output on a file to
examine it at your own pace.

3.3 Dynamic Storage Allocation

In C/C++ the programmer has can explicitly allocate and deallocate dynamic storage. (through mal-
loc/free or new/delete). If memory is (de)allocated incorrectly, it can cause problems at run time (e. g.
memory corruption, memory leak)

10
 Common errors are: trying to use memory that has not been allocated yet; accessing memory
already deallocated; deallocating memory twice.
When you have a memory problem, the best it can happen is a program crash! If the program
doesn’t crash, its behaviour becomes unpredictable, because memory corruption is like a trap: the pro-
gram could run normally as long as the program doesn’t fall on a corrupted memory region, and this
makes you believe that everything is fine. Sometimes memory problems don’t even appear during many
runs, and then, suddenly, you get a crash.
Fortunately, there are some tools you can usefully employ to check if your program has memory
problems. They can be classified into two categories:

– external libraries to be included and/or linked with you executables;

– executables which controls the execution of a program.

In the first category we can list, for examples, the libraries Memwatch and Electric Fence, while YAMD
and Valgrind falls in the second one. In this lecture we will discuss Electric Fence and Valgrind.

3.4 Electric Fence

Electric Fence is C library for malloc debugging which exploits the virtual memory hardware of the
system to check if and when a program exceeds the borders of a malloc buffer. At the borders of such
buffer, a red zone is added. When the program enters this zone, it is terminated immediately. The library
can also detect when the program tries to access memory already released.
Because Electric Fence uses the Virtual Memory hardware to detect errors, the program will be
stopped at the first instruction that causes a certain buffer to be exceeded. Therefore it becomes trivial to
identify the instruction that caused the error with a debugger. When memory errors are fixed, it is better
to recompile the program without the library.
Example âĂŞ Memory Error
1 i n t main ( i n t a r g c , char * a r g v [ ] )
2 {
3 double * h i s t o ;
4 h i s t o = ( double *) malloc ( s i z e o f ( double ) * 6 0 ) ) ;
5 f o r ( i n t i = 0 ; i < 1 0 0 ; i ++)
6 histo [ i ] = i * i ;
7 return 1;
8 }

An array of 60 elements is created. The program tries to fill it with 100 elements Compile the
program with: gcc -g -lefence -Wall -o memerror [Link]

3.5 Valgrind
Valgrind is a program which controls the execution of another one. The program can be, thus, compiled
without any special precaution. Valgrind checks every reading and writing operation on memory, in-
tercepting all calls to malloc/free new/delete. Valgrind detects problems like the usage of uninitialised
memory, reading from or writing to already freed memory and reading from or writing beyond the bor-
ders of allocated memory blocks.
Valgrind tracks every byte of the memory with nine status bits: one for the accessibility and the
other eight for the content, if valid. As a consequence, Valgrind can detect uninitialised areas and does
not report false errors on bitfield operations. Valgrind can debug almost all dynamically linked ELF x86
executables without any need for modification or recompilation.
Let’s see now some examples of Valgrind usage.

11
 Listing 4: Example âĂŞ Memory Error
1 i n t main ( i n t a r g c , char * a r g v [ ] )
2 {
3 d o u b l e * h i s t o = new d o u b l e [ 6 0 ] ;
4 f o r ( i n t i = 0 ; i < 1 0 0 ; i ++)
5 histo [ i ] = i * i ;
6 return 1;
7 }

In this first example, an array of 60 elements is created. The program tries to fill it with 100
elements, which obviously ends in writing outside the boundaries of the allocated memory region. If
you compile the program with the option -g to include debugging symbols, you can then run it under
valgrind with the command valgrind gdbattach=yes errorlimit=no ./memerror you should get
an output similar to the following one, which clearly spot the problem in the program.

==3252== Invalid write of size 8

==3252== at 0x80483DA: main ([Link])
==3252== by 0x4026F9B1: __libc_start_main (in /lib/[Link].6)
==3252== by 0x80482F0: ??? (start.S:102)
==3252== Address 0x410B2204 is 0 bytes after a block of size 480 alloc'd
==3252== at 0x4002ACB4: malloc (in /usr/lib/valgrind/vgskin_memcheck.so)
==3252== by 0x80483A8: main ([Link])
==3252== by 0x4026F9B1: __libc_start_main (in /lib/[Link].6)
==3252== by 0x80482F0: ??? (start.S:102)
==3252==
==3252== Attach to GDB ? [Return/N/n/Y/y/C/c]

The second example illustrates a typical error in initialisation

Listing 5: Forgetting the Initialisation

1 # include <iostream >
2 i n t main ( i n t a r g c , char * a r g v [ ] )
3 {
4 double k , l ;
5 double i n t e r v a l = a t o f ( argv [ 1 ] ) ;
6 i f ( i n t e r v a l == 0 . 1 ) { k = 3 . 1 4 ; }
7 i f ( i n t e r v a l == 0 . 2 ) { k = 2 . 7 1 ; }
8 l = 5 . 0 * exp ( k ) ;
9 s t d : : c o u t << " l = " << l << " \ n " ;
10 return 1;
11 }

The error doesn’t cause a crash. The user has to give an argument as an input. If the input value is
not equal to 0.1 or 0.2, the value is not initialised. We may get unexpected results.
If you run valgrind with valgrind gdbattach=yes errorlimit=no leakcheck=yes ./memerror
you can again spot the problem quite easily, since the execution is stopped after the first invalid access to
memory:

==3252== Invalid write of size 8

==3252== at 0x80483DA: main ([Link])
==3252== by 0x4026F9B1: __libc_start_main (in /lib/[Link].6)
==3252== by 0x80482F0: ??? (start.S:102)
==3252== Address 0x410B2204 is 0 bytes after a block of size 480 alloc'd
==3252== at 0x4002ACB4: malloc (in/usr/lib/valgrind/vgskin_memcheck.so)
==3252== by 0x80483A8: main ([Link])
==3252== by 0x4026F9B1: __libc_start_main (in /lib/[Link].6)

12
 ==3252== by 0x80482F0: ??? (start.S:102)
==3252== Attachto GDB ? [Return/N/n/Y/y/C/c]

The third and last example shows another typical error: returning a reference to a dynamically
allocated object.

Listing 6: Tracking Memory Leak

1 # include < string >
2 u s i n g namespace s t d ;
3 s t r i n g &x f o r m _ s t r i n g _ c o p y ( c o n s t s t r i n g &i n p u t ) ;
4 i n t main ( i n t a r g c , char * a r g v [ ] )
5 {
6 s t d : : s t r i n g o r i g i n a l ( " I am an a u t o m a t i c v a r i a b l e " ) ;
7 s t r i n g& s t r i n g r e f = x f o r m _ s t r i n g _ c o p y ( o r i g i n a l ) ;
8 }
9 s t r i n g & x f o r m _ s t r i n g _ c o p y ( c o n s t s t r i n g &i n p u t )
10 {
11 s t r i n g * x f o r m e d _ p = new s t r i n g ( " I w i l l p r o b a b l y be l e a k e d ! " ) ;
12 / / . . . maybe do some p r o c e s s i n g h e r e . . .
13 return * xformed_p ; / / C a l l e r s w i l l a l m o s t never f r e e t h i s o b j e c t .
14 }

The effect of a debugging session with valgrind is left as an exercise to the reader.

3.6 System Call Examination

A system call tracer is a program that allows you to examine problems at the boundary between your
code and the operating system. A user program cannot interact directly with the kernel of the operating
system. The program is actually executed in what is called user space. If a user space program wants to
access a hard disk, for example, it cannot do it directly but it must call an appropriate system function,
which is in charge of moving data between the program and the disk. If you want to know which function
is employed by your program, you have to use a system call tracer. The tracer shows what system calls
a process makes together with the passed parameters and the return value. Unfortunately a tracer cannot
tell you where a system call was made in your code, but only that the call took place. The exact place
has to be reconstructed. A good idea is to employ a system call tracer together with a logger. This is a
good example of how to combine different techniques to spot a problem

3.7 strace, the Linux System Tracer

The standard system call tracer in the Linux is strace. Strace is a powerful tool which shows all the
system calls issued by a user-space program. strace has several advantages. It displays the arguments to
the calls and returns values in symbolic (i.e human readable) form. Strace receives information from the
kernel and does not require the kernel to be built in any special way; therefore you can use it on every
machine.

3.8 Example
Let’s see a simple example of strace usage. This is a complete program; you can compile it with "g++
-o straceTest [Link]" and run it as it is. I recommend to study it carefully before starting to play
with it on your computer, trying to understand what the problem is only by visual examination. It is a
good exercise training yourself to spot problem without letting them manifest themselves during process
execution.
1 # i n c l u d e < i o s t r e a m > / / f o r I /O
2 # include < string > / / for s t r i n g s
3 # i n c l u d e < f s t r e a m > / / f o r f i l e I /O

13
 4 # include < cstdlib > / / for e x i t ( )
5 u s i n g namespace s t d ;
6 i n t main ( i n t a r g c , char * a r g v [ ] )
7 {
8 s t r i n g filename ;
9 s t r i n g basename ;
10 s t r i n g extname ;
11 s t r i n g tmpname ;
12 c o n s t s t r i n g s u f f i x ( " tmp " ) ;
13 / * f o r each commandline
14 a r g u m e n t ( w h i c h i s an o r d i n a r y C s t r i n g ) * /
15 f o r ( i n t i = 1 ; i < a r g c ; ++ i )
16 {
17 f i l e n a m e = a r g v [ i ] ; / / p r o c e s s a r g u m e n t a s f i l e name
18 s t r i n g : : s i z e _ t y p e i d x = f i l e n a m e . f i n d ( ’ . ’ ) ; / / s e a r c h p e r i o d i n name
19 i f ( i d x == s t r i n g : : n p o s )
20 {
21 / / f i l e name d o e s n o t c o n t a i n any p e r i o d
22 tmpname = f i l e n a m e ; / / HERE I S THE ERROR
23 / / tmpname = f i l e n a m e + ’ . ’ + s u f f i x ;
24 }
25 e l s e tmpname = f i l e n a m e ;
26 / / p r i n t f i l e name and t e m p o r a r y name
27 / / c o u t << f i l e n a m e << " => " << tmpname << e n d l ; / / USEFUL
28 }
29 i f s t r e a m f i l e ( tmpname . c _ s t r ( ) ) ;
30 if (! file )
31 {
32 c e r r << " Can ’ t open i n p u t f i l e \ " " << f i l e n a m e << " . tmp \ " \ n " ;
33 e x i t ( EXIT_FAILURE ) ;
34 }
35 char c ;
36 while ( f i l e . get ( c ) )
37 cout . put ( c ) ;
38 }

This simple program tries to access a text file with the (hardcoded) extension "tmp" inside the
current directory. The name of the file to be opened must be given as a command line parameter. The
program attaches the suffix to the name and opens the file. All you have to do to perform the exercise is
to create a text file with a suitable name (I used [Link]), using the command "ls > [Link]" or with an
editor of your choice.
If you now run the program you will get the following error message "Can’t open input file
"[Link]"". The program cannot find the input file, but that is pretty strange, because if you list the
content of your directory you will see the file right there. Obviously the program is affected by a bug.
As you can see from the source code the bug is in line ???. The programmer simply forgot to
attach the extension to the name.
In this program there is actually a second, logical, bug. In an attempt to communicate the failure
of file opening, the programmer inserted an error message (note the usage of the unbuffered cerr stream).
But the error message is misleading, because the programmer hardcoded the extension of the file name
in the message. It would have been wiser to print out just the name of the file to be read. In this way, the
problem could have been spotted immediately.
Let us see how to usefully employ strace to find the problem. Start strace with "strace -o [Link]
./straceTest list". The output of strace is written into the file [Link]. This is more convenient than
receiving the output on screen. If you scroll down the list of system calls, looking for the point where the
program attempts to open the file. If you already knew that the system call to open the file is the function
open, you can search for it in the file.

14
open("list", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "Can\'t open input file \"", 23) = 23
write(2, "list", 4) = 4
write(2, ".tmp\"\n", 6) = 6
exit_group(1) = ?

Examining the list of system call you can see immediately that the problem is with the name of
the file, because the parameter of the open function tells us the program is trying to open the file called
"list", not [Link]. but the file is not there, and you get the error return value -1.

Conclusions
Acknowledgements
I would like to thank very much J. H. M. Dassen and I. G. Sprinkhuizen-Kuyper for letting me use some
of their material on debugging techniques; P. F. Zema, my colleague in ATLAS, for useful technical
comments and discussions on Linux debugging; E. Castorina for a critical review of the lecture slides. I
would also like to thank F. Flukiger

References
[1] Wikipedia contributors. Debugging [Internet]. Wikipedia, The Free Encyclo-
pedia; 4 June 2007, 20:01 UTC [cited June 7, 2007]. Available from:
[Link]
[2] (reported in Scientific American, November 1998)
[3] [Link]
[4] For more famous bugs, take a look to Prof. G Santor’s site:
[Link]
[5] J.H.M. Dassen, I.G. SprinkhuizenKuyper, Debugging C and C++ code in a Unix environment,
Universiteit Leiden, Leiden, 1999
[6] Cite CSC
[7] T. Parr, Learn the essential of debugging, IBM developerWorks journal, Dec 2004
[8] S. Best, Mastering Linux debugging techniques, IBM developerWorks journal, Aug 2002
[9] S. Goodwin, The Pleasure Principle, Linux Magazine 31(2003) 64 69
[10] gdb User Manual
[11] gcc User Manual
[12] log4cpp article
[13] Valgrind User Manual
[14] F. Rooms, Some advanced techniques in C under Linux
[15] W. Mauerer, Visual Debugging with ddd, The Linux Gazette, Jan 2001
[16] M. Budlong, Logging and Tracing in C++ Simplified, Sun Developers Technical Articles, 2001
[17] S. Goodwin, D. Wilson, Walking Upright, Linux Magazine 27 (2003) 76 80
[18] J. World, Using Log4c, online at [Link]
[19] [Link]
[20] [Link]

15