SOFTWARE TESTING

(ITS 64704)
Static Testing
Topics

Fundamentals

Reviews

Static Analysis

Control and Data Flow Analysis

Metrics
Software Quality Assurance and the Static Test
 Static vs Dynamic Test

• In development process, different product (artefacts) are created,

e.g
• Models
• Formal/ Informal text
• Program code
• Programs are static descriptions of dynamic processes
Static vs Dynamic Test

Static test Dynamic test

• Testing of a component/system at • Testing that involves the execution
requirement or implementation of software of the component or
level, without execution of any system
software • Find the failures
• e.g code analysis, review • Documentation relating to the test
• Find defects which are causes of objects is created in a construction
failures phase and used as a test basis
• Test objects will be verified against (validation)
a document of a prior construction
phase, via a static analysis
 Static Test

• Analysis of a test objects (mostly documents) via intensive

observation
• Objective: Detection of faults/defects in an artefact

Deviations from standards, requirement defects, design defects,

insufficient maintainability and incorrect interface specifications

Check the deviation from project test plans

Results of the analysis can be further used to optimize the

development process
Static Test

• Basic idea: Defect prevention!!

• Defects and incidents are detected as soon as possible
• Before they have an effect in the further software development
and possibly require extensive correction and improvements
 Manual vs Automated Testing

• Observation of the test object manually or via tool support

Reviews

• Manual testing by one or more people

• Human analysis and thinking capabilities are used to evaluate work products
• Any software work product can be reviewed (e.g. requirement specifications,
design specifications, code, test plans, test specifications, test cases, test
scripts or user guides)

Static analysis

• Automated testing by means of tool support against predefined rules (e.g.

coding rules)
• –Used only for documents with a formal structure (e.g. program code, UML
diagrams)
 Manual Testing Technique

• Non-tool-supported static testing use human analysis and thinking

capabilities to test and evaluate complex work products
• This is performed via intensive reading and consideration of the
documents to be tested
• Different approaches exist to statically test the documents.
• They differ in
• the intensity of the analysis
• the resources needed (people, time)
• the desired goals
Topics

Fundamentals

Reviews

Static Analysis

Control and Data Flow Analysis

Metrics
Reviews

Roles

Static Test Reviews Types

Fundamental
activities
Reviews

• In general, a review is
• an overview of something
• a survey
• a look back over past events
• “Review” is a term used to describe a particular way of testing
software artefacts
• Often, reviews are the only possibility to test the semantics of
such artefacts efficiently
• The term “review“ is also a general term for a number of specific
types of review.
Reviews

Roles

Static Test Reviews Types

Fundamental
activities
Reviews: Fundamental Activities

• With a formal
review, there
are 6 basic
steps to
execute
Reviews: Planning

• The management decides which documents

should be reviewed
• The estimated expenses for the separate reviews
is to be considered in the project planning
• The entry criteria for the execution of the review
are confirmed (especially with more formal
reviews)
• The exit criteria for the completion of the review
are defined (especially with more formal reviews)
• Furthermore, testing criteria are confirmed
Reviews: Planning

• The manager chooses professionally qualified

members and puts together a review team with
specific roles
• The manager assures in cooperation with the
author of the test object, that it has a
“reviewable” state, i.e., the document is to a
certain degree finalized and has reached a
satisfactory maturity
• Re-examination if the entry criteria for the
execution of the review are satisfied
• If a Kick-Off meeting takes place, time and venue
have to be fixed
Reviews: Kick-Off

• All people who are involved in the review are

provided with the necessary information
• Written invitation or spontaneous first
meeting of the review team to inform about
the reason and the objectives of the review.
• If the involved people are not fully
knowledgeable of the document to be
reviewed, it can be briefly introduced and its
relevance to the application area can be
presented.
Reviews: Individual Preparation

• Each reviewer studies the document intensively

and uses the data provided
• Use of checklists and other documents to support
the task is recommended
• Any potential defects, questions or comments
should be noted
• The review team prepares themselves for the
review meeting (examination)
Reviews: Examination

• Conducted by the moderator

• Generally of fixed duration (max. 2 hours)
• If needed, a further examination can be called for
(on the next day at the earliest)
• Objective: Evaluation of the document to be
reviewed with regard to the adherence and
implementation of relevant rules and guidelines
• The evaluation of the specific findings and the
overall conclusion should be agreed upon by all
reviewers
Reviews: Examination

• The moderator has the right to cancel or

discontinue an examination if one or more
experts (reviewers) are absent or are not
appropriately prepared
• The review document is to be scrutinized and
not the author!
• Reviewers must pay attention to expressions
and wording
• The author should not try to defend defects
found
Reviews: Examination

• To ensure this, reviewers should not “attack“ the

author in a way that forces a defensive response
• A justification the author‘s decision will be
viewed partly as appropriate and helpful
• The moderator shall not take a reviewer role at
the same time
• No general matters of style and grammar (outside
of the guidelines) are to be discussed
Reviews: Examination

• Development and discussion of solutions is not

the task of the review team
• Every reviewer must have the opportunity to
present his/her comments appropriately
• The consensus of the reviewer about a comment
is to be continuously recorded by the scribe
• Defects are not to be recorded as instructions to
the author
• Instructions in the form of additional, specific
suggestions for correction might be viewed to a
certain degree as useful and helpful for the
improvement of the quality
Reviews: Examination

• Prioritizing of defects found. For example:

•Critical Defect
• Test object cannot be used until corrected.
• The defect must be corrected before the next release

Major Defect
• Usability of the test object is limited.
• The defect should be corrected before the next release

•Minor Defect
• A small issue, e.g., spelling mistakes or flaws in expression which do not affect
usability

Good
• Free of flaws
• No further changes to be done
Reviews: Examination

• Review team gives recommendations about the

adoption of the test object:
• Accept without changes
• Accept with changes – no further review needed
• Reject– further review or other test required
• The review report contains a list of all defects, which
have been discussed in the examination and, in addition,
to that
• Information about the test object and the data used
• People involved and their roles
• A short summary of the important points
• Result of the examination with the recommendation
of the review team
• At the end, all members of the examination approve and
sign the review report
Reviews: Report
(Example)
Reviews: Rework

• Correcting the discovered defects and recording

the new status
• Generally done by the author
• Depending on the ease of understanding of the
defects, it might be useful to discuss with the
reviewer again
 Reviews: Follow up

• Ensuring that the review has been executed

according to the guideline
• Checking that defects have been removed by
means of rework, gathering metrics
• Checking on exit criteria (for more formal review
types)
• Manager decides if the recommendation of the
reviewers will be followed or makes a different
decision, for which he/she then bears the sole
responsibility
• If the result of the first review wasn't acceptable,
and another review is to be done, the second
review should be done according to the described
process (although often shortened)
Reviews

Roles

Static Test Reviews Types

Fundamental
activities
Roles in the Review Process

Manager

• Decides on the execution of reviews

• Chooses the testing objects and the documents to be used
• Provides the necessary resources and allocates the review team
• Determines if the review objectives have been met
• Decides about the next steps after the examination

Moderator

• Leads the review and is the person upon whom the success of the review
rests
• Often responsible for the planning activities
• Has to mediate between the various points of views and pays attention to
any subtle comments
• Is not to be prejudiced and is not permitted to express his/her own opinion
regarding the testing object
Roles in the Review Process

Author

• Writer of the document which is to be reviewed

• If there are several authors, one responsible writer should be named, who will take
over the overall role
• It is important, that the author does not consider the expressed views as a criticism
of his own personality, but that it is exclusively about the improvement of the quality
of the reviewed document

Reviewer (Inspector)

• Several (maximum five) experts, who take part in the examination after the necessary
preparation
• Ensure that parts of document, which are found to be ‚good‘, are also being labeled
as such
• Will clearly mark defective parts of the review object so that their removal is possible
• Reviewers should be chosen to represent different perspectives and roles in the
review process and can take part in all the examinations
Roles in the Review Process

Scribe

• Documents all the issues, problems, and open points that

were identified during the meeting
• Has to formulate concisely and precisely and write legibly
• Writes up the discussions taking place at the meeting so that
they can‘t be corrupted
• The roles of scribe and moderator should not be taken by one
and the same person
• For practical reasons, it might be useful to let the author to
also be the scribe. The author knows exactly what to note
down and with what level of precision. This ensures that
sufficient information is available to make the changes later.
Reviews

Roles

Static Test Reviews Types

Fundamental
activities
Reviews: Types according to the examined
object

Informal Review

• Informal review during a developmental activity

• Mostly a single review with no formal process
• Can be optionally documented
• Varies in usefulness depending on the reviewers
• Inexpensive way to get some benefit
Reviews: Types according to the examined
object

Walkthrough

• Main purposes: Learning, gaining understanding, finding defects

• Definition: A step-by-step presentation by the author of a document in order to
gather information and to establish a common understanding of its content
• The preparation has the least complexity. It can be partly even waived.
• During the examination, the author often takes over the role of the moderator
( but not the role of the scribe).
• The author presents the review object to the reviewers
• Since the author conducts the examination, he/she can influence its course
strongly. This can be a disadvantage if the author does not permit discussion of
critical points of the review object with the required intensity or doesn‘t even
permit the discussion at all.
Reviews: Types according to the examined
object

Technical Review
• Main purposes:
• Discussing, decision making, evaluating alternatives, finding defects, solving
technical problems
• Checking conformity of the review object to specifications, adherence to
regulations, suitability for the intended purpose.
• Definitions:
• A peer group discussion activity that focuses on achieving consensus on the
technical approach to be taken
• •Can be done as a "Peer Review" without the participation of the management,
and vary in practice from informal to very formal
• In case of high formality, entry-criteria and exit-criteria may be defined
• It is not the responsibility of the technical review team to decide about the
consequences of the review results
Reviews: Types according to the examined
object
Technical Review
• Optional items
• –Using checklists
• –Producing a review report or a list of findings
• –Participation of the management
• Substantial time and effort needed in preparation phase
• –Reviewers test the object according to fixed testing criteria
• –Reviewer comments about each testing criteria are to be recorded in writing
and handed to the moderator in advance
• During the examination
• –Ideally led by a trained moderator (not the author)
• –Only relevant comments should be discussed
• –Scribe documents all comments and produces a final documentation of the
verdict
• The moderator prioritizes reviewer comments
• Review result must be agreed by consensus by the review team
Reviews: Types according to the examined
object
Inspection
• Main purpose: Finding defects (or even causes for defects)
• Using checklists
• Producing a review report or a list of findings
• Participation of the management
• Substantial time and effort needed in preparation phase
• Reviewers test the object according to fixed testing criteria
• Reviewer comments about each testing criteria are to be recorded in writing
and handed to the moderator in advance
• During the examination
• Ideally led by a trained moderator (not the author)
• Only relevant comments should be discussed
• Scribe documents all comments and produces a final documentation of the
verdict
• The moderator prioritizes reviewer comments
Benefits of Reviews

• Correction of defects costs resources – if defects can be

recognized and corrected at an early stage, it leads to an
increased development productivity as less resources have to be
used for defect recognition and correction, and these resources
are therefore available for development
• It leads often to shorter development timescales
• If defects are detected and corrected in an early stage, it leads to
a reduction of the effort needed for dynamic test, as less defects
are left in the test object
• Fewer defects in a running system
• Mutual learning and possible process improvements are further
positive effects
Reviews: Risk Factors

Not enough or not enough staff with the required training

staff available

Conduct training or engage appropriate staff from external

companies

Not enough incorrect estimation when planning resources

time

Possibly, a less extensive type of review can solve the problem

Poor Choose different reviewers

preparation

If reviewers are not convinced of the importance of the review and

its high level of effectiveness regarding quality improvements to the
examined documents, this is to be emphasized with adequate
numeric data
Reviews: Risk Factors

Non-existent or Finding defects? Improving the process? Blaming people?

confused
objectives
Confirm the objectives which are to be achieved with this review
when planning it (i.e. prior to conducting the review!)

Lacking or Check in advance, whether all needed documents are available and
inadequate in the depth needed.
documentation
Only if this is the case can the review be done

No The review process is not going to be successful if the required

management resources are not made available and the achieved results are not
support used for the improvement of the process
Topics

Fundamentals

Reviews

Static Analysis

Control and Data Flow Analysis

Metrics
Static Analysis

Objective: To find defects or error-prone areas in a document

This type of testing does not include the execution of the test object
(program code).

Example:
• Spell-checker programs as a kind of static analyser, that detects
defects in documents (spelling and to a certain degree grammar
mistakes), and thereby plays its part in quality improvement
• Compiler carries out an analysis of program code, by checking
against syntax violations of the programming language
• Specialised tools which check use of programming standards
Static Analysis & Tool Support

• Static analysis is only meaningful with tool support

• The document which is to be analysed must adhere to a formal
structure (e.g. programming language syntax) to enable analysis
to be performed by a tool
• Other documents, that adhere to a formal structure, are for
example:
• Technical specifications
• Software architecture
• Software designs
Elements of a tool based static analysis
Static Analysis & Reviews

• They are closely related to each other

• If a static analysis is being conducted prior to the review of a
formal document, a number of defects and deviations can be
identified and the amount of aspects to be considered in a
review is substantially lower
• As static analysis is conducted with tool support, the
complexity is substantially less than with a review
• •Therefore:

Static
Review
Analysis
Static Analysis of Program Code

• All compilers conduct a static analysis of the program code, by

checking the adherence of the programming language syntax
• Most of the compilers also provide additional information, which
can be determined via static analysis
• Examples of defects or error-prone situations, which can be
detected by the static analysis of program code are:

Convention
Syntax Security
and standard
violations vulnerabilities
deviations

Control flow Data flow

anomalies anomalies
Syntax Testing

Syntax • Syntax violations of the programming language will be reported as

violations warnings or defects
• Many compilers conduct further checks, such as:
Convention • Usage of separate elements of the program (variables,
and standard functions, ...)
deviations • Testing of properly formatted usage of data and variables with
strongly typed programming languages (C, C++, Java)
Security • Detection of undeclared variables
vulnerabilities
• The results are normally
presented as a list
Control flow • The results don't always directly
anomalies indicate defects. Further checks
are necessary

Data flow
anomalies
Adherence to conventions and standards

Syntax
violations • The adherence of programming or design conventions by
means of appropriate analysers can be checked
Convention
• Needs less time and almost no staff resources
and standard • Furthermore, there is often another benefit given: If the
deviations programmers or the software designers are aware, that
the program code or the design of a tool is being checked
Security against the adherence to guidelines, their willingness to
vulnerabilities
implement it, is substantially higher than without an
automated testing
Control flow • But: Static analysis tools may produce a large number of
anomalies warning messages and hints, which need to be well-managed
to allow the most effective use of the tool
Data flow
anomalies
Discovering of security vulnerabilities

Syntax
violations • Because of the usage of certain error-prone
programming structures and the absence of
Convention
and standard
necessary reviews, security vulnerabilities can
deviations appear
• Example:
Security
vulnerabilities
• No interception of input-buffer overflow
• No testing of adherence to data limits at the
Control flow
entry
anomalies • Analysis tools can discover these defects, as they
are mostly of a certain “structure“, which can be
Data flow traced and analysed by tools
anomalies
Topics

Fundamentals

Reviews

Static Analysis

Control and Data Flow Analysis

Metrics
Control and Data Flow Analysis

• Searching for anomalies in the program code

• Not all defects and irregularities can be identified with static
analysis: Some defects can only be identified as a failure while the
program is running, and not before
• If, for example, the divisor in a division is a variable, the
variable could take the value zero, which might lead to a
failure
• Statically this defect is usually not detectable

Anomalies: Irregularities, which might be a defect which lead

to a failure, but do not have to lead to one.
Control Flow

Control Flow: an abstract representation of all possible sequences

which can be executed by a program

• Sequence of statements executed while the program is running

• Doesn‘t have to be in the sequence of the statements in the
program code
• Is being determined by “controlling” statements
• Conditional branches
• Loops
• •The logical value of conditions, e.g., in IF-statements in program
code, governs the flow of control
• Loops lead back to prior statements, leading to either repeated
execution of the code within the loop or exiting the loop.
Control Flow Graph: Elements
Directed graph
Nodes
• Statement of the program
• Linear sequences of statements (block, segment) can be presented as a
single node
• If the first statement of the sequence is executed, all further statements
of the sequence will also be executed

Edges
• Possible execution sequences of the statements, represented by the
nodes
• Nodes with several outgoing edges represent the statements that
control the flow
• IF-statements and loop statements: Two outgoing edges
• CASE-statements: several outgoing edges

Others
• There is exactly one start node at the beginning of a program, or head of
the method and exactly one end node at the end of the program or
method
Example of a Control Flow Graph
Data Flow Anomalies

• The usage of every single variable is analysed

• Three usages or conditions of variables can be differentiated

Undefined (u)
• the variable has no defined value (e.g. at the start of the program, when it
is de-allocated, or when it leaves its area of validity)

Defined (d)
• the variable has been allocated a value

Referenced (r)
• the value of the variable is being read or used
Data Flow Anomalies

• Three types of data flow anomaly are defined

ur-Anomaly
• An undefined value (u) of a variable is read (r)

du-Anomaly
• The variable contains a value (d), but which is made undefined (u),
before it has been used
dd-Anomaly
• The variable is assigned a value (d) for a second time before the first
value has been used
Example: Data Flow Anomalies
Topics

Fundamentals

Reviews

Static Analysis

Control and Data Flow Analysis

Metrics
Measurements and measuring in Software
Projects

Software Metrics (Software Measurement) is a branch of

information technology, in which software process and
product qualities as well as their correlation to each
other are quantified and measured
Why measure in Software Projects?

Understanding
• Measuring helps us to understand what happened in the development of a product,
by highlighting explicitly the characteristics and quantifying them.

Control
• Projects can be controlled with the help of measurement and, based on earlier
projects, a wealth of experience can be built up which puts the processes
( methods, techniques...) into relation with the achieved results.
• This knowledge can be used to make adjustments to future projects, so that desired
objectives can be achieved.

Improve
• The correlations (based on extracted experiences) between processes and
achieved goals gradually develop to generally accepted principles. These are
strengthened the more often they are confirmed in successful projects.
• In subsequent projects, the use of these principles can be planned from the
beginning. Time and resources can be better planned.
What are Software Metrics?
Types of Measurement

Product measurements for software systems

• Size, functionality, complexity, usability, reliability, safety, maintainability, portability and
adaptability
Project or resource measurements for people,hardware and software
• Amount of developers, % overhead, price, rate of achievement, memory capacity, speed,
precision and benefit
Process measurements for software development and maintenance
• Duration, complexity, costs, issues, change requests, requirements, amount of releases, gaps
between releases
Static measurement
• Measures a characteristic at a certain fixed time

Dynamic measurement
• Measures the development of the characteristics over time
McCabe’s Cyclomatic Complexity

• Software quality metric that quantifies the complexity of a software program.

• Complexity is inferred by measuring the number of linearly independent paths through
the program.
• The higher the number the more complex the code.

The Significance of the McCabe Number

• Programs with high McCabe numbers (e.g. > 10) are likely to be difficult to understand
and therefore have a higher probability of containing defects.
• The cyclomatic complexity number also indicates the number of test cases that would
have to be written to execute all paths in a program. (white box testing: branch coverage
and full coverage)
Calculating McCabe’s Cyclomatic Complexity

Cyclomatic complexity is derived from the control flow graph of a

program as follows:

Cyclomatic complexity (CC) = E - N + 2P

Where:
P = number of disconnected parts of the flow graph (e.g. a calling
program and a subroutine)
E = number of edges (transfers of control)
N = number of nodes (sequential group of statements containing only
one transfer of control)
Calculating McCabe’s Cyclomatic Complexity

Example:
Cyclomatic complexity (CC) = E - N + 2P
* Sometimes CC is also known as V(G)
Calculating McCabe’s Cyclomatic Complexity
Example:

(CC) = E - N + 2P
Calculating McCabe’s Cyclomatic Complexity
Example:

(CC) = E - N + 2P
= 7 - 7 + 2(1)
=2
Calculating McCabe’s Cyclomatic Complexity
Example:

(CC) = E - N + 2P
= 7 - 7 + 2(1)
=2

2: V (G)= ∏ + 1

Predicate nodes
= nodes which has 2 outgoing
edges
Calculating McCabe’s Cyclomatic Complexity
Example:

(CC) = E - N + 2P
= 7 - 7 + 2(1)
=2

2: V (G)= ∏ + 1

Predicate nodes
= nodes which has 2 outgoing
edges
Calculating McCabe’s Cyclomatic Complexity
Example:

(CC) = E - N + 2P
= 7 - 7 + 2(1)
=2

2: V (G)= ∏ + 1
=1+1
=2
Calculating McCabe’s Cyclomatic Complexity
Example:

(CC) = E - N + 2P
= 7 - 7 + 2(1)
=2

2: V (G)= ∏ + 1
=1+1
=2

3: V (G)= No. of regions

Regions = closed edges

Calculating McCabe’s Cyclomatic Complexity
Example:

(CC) = E - N + 2P
= 7 - 7 + 2(1)
=2

2: V (G)= ∏ + 1
=1+1
=2

3: V (G)= No. of regions

Calculating McCabe’s Cyclomatic Complexity
Example:

(CC) = E - N + 2P
= 7 - 7 + 2(1)
=2

2: V (G)= ∏ + 1
=1+1
=2

3: V (G)= No. of regions

=2
Using McCabe’s Cyclomatic Complexity in
Software Testing

STEP 1
• Construction of graph with nodes and edges from the code (Control
Flow Graph)

STEP 2
• Identification of independent path

STEP 3
• McCabe Cyclomatic Complexity Calculation

STEP 4
• Design of test cases

Once the basic set is formed, TEST CASES should be written to execute
all the paths.
Calculating McCabe’s Cyclomatic Complexity
Example:

(CC) = E - N + 2P
= 7 - 7 + 2(1)
=2

2: V (G)= ∏ + 1
=1+1
=2

3: V (G)= No. of regions

=2
3 METHODS OF Calculating McCabe’s
Cyclomatic Complexity

T Condi Input/ Test data Expected

C tion result
1 Path 1 A=10, B=10, C=9 Print A,B,C
2 Path 2 A = 10, B=9, C= 10 Print A,B,C

Sample of test cases derived from

McCabe’s Calculation
McCabe’s Cyclomatic Complexity
Metrics in Quality Assurance

Objective: Quantitative statements regarding the naturally abstract

product software
• Measurement numbers or metrics measure e.g., quality
characteristics
 To evaluate measured values towards the satisfaction of
specified requirements
• Metrics always only deliver selective statements regarding the
examined aspects
 Calculated metrics become only significant if they are
compared with numbers of other examined programs (parts)
 The interpretation of metrics are useful only in relatively
stable, repeating processes
Uses of Cyclomatic Complexity

• Helps developers and testers to determine independent path

executions
• Developers can assure that all the paths have been tested at least
once
• Helps us to focus more on the uncovered paths
• Improve code coverage in Software Engineering
• Evaluate the risk associated with the application or program
• Using these metrics early in the cycle reduces more risk of the
program
Benefit of Static Analysis

Early detection of defects prior to the test execution

Early warning of critical aspects in code or design, via calculation of metrics, e.g.,
high degree of complexity

Identification of potential defects, which cannot be detected effectively and

efficiently with dynamic tests

Detecting of dependencies and inconsistencies in software models, for example,

links that are redundant or violate the architecture

Improved readability, changeability and maintainability of code and design

Prevention of defects, if learning from experience takes place, and lessons are
applied in further development
Among defects detectable with static analysis

• Referencing of a variable with no defined values

• Inconsistent interface with modules and components
• Variables, that are never used
• Unreachable (dead) code
• Violation of coding conventions/rules
• Security vulnerabilities
• Syntax violations of code and software models
• ...
THANK YOU