Software

Reliability
Software Reliability, Failures
and Faults
The reliability of a software system is a measure of how well it
provides the services expected by its users, expressed in terms of
software failures.
A software failure is an execution event where the software
behaves in an unexpected or undesirable way.
A software fault is an erroneous portion of a software system
which may cause failures to occur if it is run in a particular state, or
with particular inputs.
Kinds of failures
Programming for Reliability
Fault avoidance:
development techniques to reduce the number of faults in a system

Fault tolerance:
developing programs that will operate despite the presence of faults
Fault Avoidance
Fault avoidance depends on:
1. A precise system specification (preferably formal)
2. Software design based on information hiding and
encapsulation
3. Extensive validation reviews during the development process
4. An organizational quality philosophy to drive the software
process
5. Planned system testing to expose faults and assess reliability
Common Sources of
Software Faults
Several features of programming languages and systems are common sources of faults in
software systems:
Goto statements and other unstructured programming constructs make
programs hard to understand, reason about and modify.
◦ Use structured programming constructs

Floating point numbers are inherently imprecise and may lead to invalid
comparisons.
◦ Fixed point numbers are safer for exact comparisons

Pointers are dangerous because of aliasing, and the risk of corrupting memory
◦ Pointer usage should be confined to abstract data type implementations
Common Sources of
Software Faults….
Parallelism is dangerous because timing differences can affect overall program behavior in
hard-to-predict ways.
◦ Minimize inter-process dependencies

Recursion can lead to convoluted logic, and may exhaust (stack) memory.
◦ Use recursion in a disciplined way, within a controlled scope

Interrupts force transfer of control independent of the current context, and may cause a
critical operation to be terminated.
◦ Minimize the use of interrupts; prefer disciplined exceptions
 Fault Tolerance
A fault-tolerant system must carry out four activities:

1. Failure detection: detect that the system has reached a particular

state or will result in a system failure
2. Damage assessment: detect which parts of the system state have
been affected by the failure
3. Fault recovery: restore the state to a known, “safe” state (either by
correcting the damaged state, or backing up to a previous, safe
state)
4. Fault repair: modify the system so the fault does not recur (!)
Approaches to Fault
Tolerance
N-version Programming:
Multiple versions of the software system are implemented independently by different teams.

The final system:

runs all the versions in parallel,
compares their results using a voting system, and
rejects inconsistent outputs.
(At least three versions should be available!)
Approaches to Fault
Tolerance ...
Recovery Blocks:

A finer-grained approach in which a program unit contains a test to check for failure, and
alternative code to back up and try in case of failure.
alternatives are executed in sequence, not in parallel
the failure test is independent (not by voting)
Defensive Programming
Failure detection:
Use the type system to ensure that variables do not get assigned invalid values.
Use assertions to detect failures and raise exceptions. Explicitly state and check all invariants
for abstract data types, and pre- and post-conditions of procedures as assertions. Use
exception handlers to recover from failures.
Use damage assessment procedures, where appropriate, to assess what parts of the state
have been affected, before attempting to fix the damage.

Fault recovery:
Backward recovery: backup to a previous, consistent state
Forward recovery: make use of redundant information to reconstruct a consistent state from
corrupted data
Software Reliability Models
 What is Reliability?
Reliability is applied whenever we expect something to behave in a certain way.
Reliability is one of the metrics that are used to measure quality.
It is a user-oriented quality factor relating to system operation.
Intuitively, if the users of a system rarely experience failure, the system is considered to be more
◦

reliable than one that fails more often.

A system without faults is considered to be highly reliable.

Constructing a correct system is a difficult task.
◦

Even an incorrect system may be considered to be reliable if the frequency of failure is “acceptable.”
◦

Key concepts in discussing reliability:

Fault, Failure, Time
◦

Three kinds of time intervals: MTTR (Mean Time to Recover), MTTF (Mean Time to Failure), MTBF
◦

(Mean Time Between Failure)

13
What is Reliability?
Failure
A failure is said to occur if the observable outcome of a program execution is
◦

different from the expected outcome.

Fault
The adjudged cause of failure is called a fault.
◦ ◦

Example: A failure may be caused by a defective block of code.

Time
Time is a key concept in the formulation of reliability. If the time gap between
◦

two successive failures is short, we say that the system is less reliable.
Two forms of time are considered.
◦

Execution time () and Calendar time (t)

◦

14
What is Reliability?
Two ways to measure reliability
Counting failures in periodic intervals
◦

Observe the trend of cumulative failure count - µ().

◦

Failure intensity
◦

Observe the trend of number of failures per unit time – λ().

◦

µ()
This denotes the total number of failures observed until execution time  from the beginning
◦

of system execution.

λ()
This denotes the number of failures observed per unit time after  time units of executing the
◦

system from the beginning. This is also called the failure intensity at time .

Relationship between λ() and µ() λ() = dµ()/d

15
 Definitions of Software
Reliability
First definition
Software reliability is defined as the probability of failure-free operation of a software
◦

system for a specified time in a specified environment.

Key elements of the above definition
◦

Probability of failure-free operation

◦

Length of time of failure-free operation

◦

A given execution environment

◦

Example
◦

The probability that a PC in a store is up and running for eight hours without crash is 0.99.
◦

Second definition
Failure intensity is a measure of the reliability of a software system operating in a given
◦

environment.
Example: An air traffic control system fails once in two years.
◦

Comparing the two

The first puts emphasis on MTTF, whereas the second on count.
◦

16
Factors Influencing Software
Reliability
A user’s perception of the reliability of a software depends upon two categories of
information.
The number of faults present in the software.
◦

The ways users operate the system.

◦

This is known as the operational profile.

◦

The fault count in a system is influenced by the following.

Size and complexity of code
◦

Characteristics of the development process used

◦

Education, experience, and training of development personnel

◦

Operational environment
◦

17
Applications of Software
Reliability
Comparison of software engineering technologies
What is the cost of adopting a technology?
◦

What is the return from the technology -- in terms of cost and quality?
◦

Measuring the progress of system testing

Key question: How testing has been done?
◦

The failure intensity measure tells us about the present quality of the system: high intensity means more tests
◦

are to be performed.

Controlling the system in operation

The amount of change to a software for maintenance affects its reliability. Thus the amount of change to be
◦

effected in one go is determined by how much reliability we are ready to potentially lose.

Better insight into software development processes

Quantification of quality gives us a better insight into the development processes.
◦

18
Reliability Models
Main idea
We develop mathematical models for λ() and µ().
◦

Basic assumptions in developing a reliability model

Faults in the program are independent.
◦ ◦ ◦ ◦ ◦

Execution time between failures is large w.r.t. instruction execution time.

Potential test space covers its use space.
The set of inputs per test run is randomly chosen.
The fault causing a failure is immediately fixed or else its re-occurrence is not
counted again.

19
 Reliability Models
Intuitive idea
As we observe another system failure and the corresponding fault is fixed, there will be fewer
◦

number of faults remaining in the system and the failure intensity will be smaller with each fault
fixed.
In other words, as the cumulative failure count increases, the failure intensity decreases.
◦

Two decrement processes

Decrement process 1
◦

The decrease in failure intensity after observing a failure and fixing the corresponding fault is constant.
◦

This gives us the Basic model.

◦

Decrement process 2
◦

The decrease in failure intensity after observing a failure and fixing the corresponding fault is smaller than the previous
◦

decrease.
This gives us the Logarithmic model.
◦

20
 Reliability Models

cumulative failures µ.
Fig: Failure intensity λ as a function of
Parameters of the models
λ 0: The initial failure intensity observed
◦

at the beginning of system testing.

v 0: The total number of system failures

◦

that we expect to observe over infinite

time starting from the beginning of
system testing.

: A parameter representing non-linear

◦

drop in failure intensity in the Logarithmic

model.

21
Reliability Models

Fig: Failure intensity λ as a function of execution time 

(λ 0= 9 failures/unit time, v 0= 500 failures,  = 0.0075).
Basic model
Assumption:
λ(µ) = λ 0(1 - µ/v 0)
dµ()/d = λ 0(1 - µ()/v 0)
µ() = λ 0(1 - µ/v 0)
λ() = λ 0.e-λ0 /v0

Logarithmic model
Assumption:
λ(µ) = λ 0e-µ
dµ()/d = λ 0e-µ()
µ() = ln(λ 0 + 1)/
λ() = λ 0/(λ 022 + 1)
Fig: Cumulative failure µ as a function of
execution time  (λ 0= 9 failures/unit
time, v 0= 500 failures,  = 0.0075).

23
 Reliability Models
Example
Assume that a software system is undergoing system level testing. The initial failure intensity
(λ0 ) of the system was 25 failures/CPU hours, and the current failure intensity is 5
failures/CPU hour. It has been decided by the project manager that the system will be
released only after the system reaches a reliability level of at most 0.001 failures/CPU hour.
From their experience the management team estimates that the system will experience a total
of 1200 failures over infinite time. Calculate the additional length of system testing
required before the system can be released.
The system will experience a total of 1200 failures over infinite time (v 0). Thus, we use the
◦

Basic model.
λ cand λ rare the current failure intensity and the failure intensity at the time of release.
◦

Assume that current failure intensity has been achieved after executing the system for  c
◦

hours.
Let λ rbe achieved after testing the system for a total of  rhours.
◦

24
(Example continued)

( r-  )c denotes the additional execution time required to achieve λ .r

◦

We can write λ cand λ ras follows.

λ c= λ 0.e-λ0 c/v0
λ r= λ 0.e-λ0 r/v0
λ c/ λ r= (λ 0.e-λ0 c/v0 )/(λ 0.e-λ0 r/v0 )
= e(r - c) λ0/v0
ln(λ c/ λ )r = ( r-  )c λ 0/v 0

( r-  )c = (v 0/ λ 0)ln(λ c/ λ )r
= (1200/25)ln(5/0.001)
= 408.825 hours
It is required to test the system for more time so that the CPU runs for another 408.825 hours to
achieve the reliability level of 0.001 failures/hour.
25
Software Reliability Models
▪ Basic Execution Time Model

 (  )  0  1   (1)

 V0 

Fig: Failure intensity  as a function of µ for basic

model
 Software Reliability

d  (2)
d  V 0
0

Fig.7.14: Relationship between  & µ for basic model

 Software Reliability
For a derivation of this relationship, equation 1 can be written as:

d (    ( )
 0
) d   V0 
1 
The above equation can be solved for  ( ) and result in :


   0 
 ( )  V 1  exp V
0 
 (3)
  0

 Software Reliability
The failure intensity as a function of execution timeis shown in figure given below

 ( )  0    0 
 V 
exp  0 

Fig.7.15: Failure intensity versus execution time for basic

model
 Software Reliability
▪ Derived quantities

Fig.7.16: Additional failures required to be experienced to reach the objective

51
 Software Reliability

Fig.7.17: Additional time required to reach the

objective
This can be derived in mathematical form as:
V  P 
  Ln
0

 F 
0


Example- 7.1

Assume that a program will experience 200 failures in infinite time. It has now experienced 100.
The initial failure intensity was 20 failures/CPU hr.

(i) Determine the current failure intensity.

(ii) Find the decrement of failure intensity per failure.

(iii) Calculate the failures experienced and failure intensity after 20 and 100 CPU hrs. of execution.

(iv) Compute additional failuresand additional execution time required to reach the failure
intensity objective of 5 failures/CPU hr.

Use the basic execution time model for the above mentioned calculations.
Solution
Here Total failures experienced (Vo) = 200 failures

Mean failures experienced () = 100 failures

Initial failure intensity (0) = 20 failures/CPU hr.
(i) Current failure intensity:

 (  )  0  1  
 V0 

 
100
 201   20(1 0.5)  10 failures/CPU
200 
hr
(ii) Decrement of failure intensity per failure can be calculated as:
20
d     0.1/ CPU hr.

d V0 200
0
(iii) (a) Failures experienced & failure intensity after 20 CPU
hr:

   0 
0 
 ( )  V 1  exp V 
 
 0 

  20 20  
 2001
 exp 200    200(1 exp(1
2))
 200(1 0.1353)  173failures
  ( )  0 exp   0 
 V0 

  20 20

 20 exp  200   20 exp(2)  2.71 failures / CPU hr

(b) Failures experienced & failure intensity after 100 CPU hr:

   0 
0  1  exp
 ( )  V 
  V 
 0 
   20 100  
 2001 exp 200    200 failures (almost)
 
 ( )  0    0 
 V 
 0 
exp
  20 100
 20 exp  200   0.000908 failures / CPU hr

(iv) Additional failures   required to reach the failure intensity objective of 5
failures/CPU hr.
 200
 V  P  F    (10  5)  50
 00  failures
 20 


Additional execution time required to reach failure intensity objective of 5 failures/CPU
hr.

 V   0 
  
 0   F 
 Ln  P

 
200 10
 20 Ln   6.93 CPU
hr.  5
Logarithmic Poisson Execution Time Model
Failure Intensity
 (  )  0 exp( )

Fig.7.18: Relationship
between
d     0 d 
exp(  ) 
d d

Fig.7.19: Relationship
 1
 ( )  Ln(0 
1)
 ( )  0 /(0
1)   Present failure
1  P 
 intensity
FP  Failure intensity
  Ln F  objective

  (4)
 1  1  1 
   F  P 
Example- 7.2

Assume that the initial failure intensity is 20 failures/CPU hr. The failure intensity decay
parameter is 0.02/failures. We have experienced 100 failures up to this time.

(i) Determine the current failure intensity.

(ii) Calculate the decrement of failure intensity per failure.

(iii) Find the failures experienced and failure intensity after 20 and 100 CPU hrs. of execution.

(iv) Compute the additional failures and additional execution time required to reach the failure intensity
objective of 2 failures/CPU hr.

Use Logarithmic Poisson execution time model for the above mentioned calculations.
Solution

0  20 failures/CPU hr.
  100 failures
  0.02 / failures
(i) Current failure intensity:

 (  )  0 exp( )

= 20 exp (-0.02 x 100)

= 2.7 failures/CPU hr.

(ii) Decrement of failure intensity per failure can be calculated as:
d
 θλ
d
= -0.02 x 2.7 = -0.054/CPU hr.

(iii) (a) Failures experienced & failure intensity

after 20 CPU hr:
 ( )  1 Ln0 

1
1
 Ln(20  0.02  20 1)  109 failures
0.02
  ( )  0 / 0 1
 (20) /(20 .02  20 1)  2.22 failures / CPU hr.

(b) Failures experienced & failure intensity after 100 CPU hr:

 ( )  1 Ln0 

1 1
 Ln (20  0.02 100 1)  186 failures
0.02

 ( )  0 / 0 1

 (20) /(20 .02 100 1)  0.4878 failures /

CPU hr.
(iv) Additional failures   required to reach the failure intensity objective of 2 failures/CPU
hr.
1 P 1  2.7 
 Ln Ln   15
   2 failures

 F 0.02
  1 
1 1 1 1 1
         6.5 CPU
  F P  0.02  2 2.7 
hr.
Example- 7.3
The following parameters for Basic and Logarithmic Poisson models are given:
(a) Determine the addition failures and additional execution time required to reach the failure
intensity objective of 5 failures/CPU hr. for both models.

(b) Repeat this for an objective function of 0.5 failure/CPU hr. Assume that we start with the
initial failure intensity only.

Basic execution time model Logarithmic Poisson execution time model

  10 failures/CPU hr
o
  30 failures/CPU hr
o

V  100 failures
o
  0.25 / failure
 Solution
(a) (i) Basic execution time model

V
  0 (P   F )
0

100
 (10  5)  50 failures
10
P (Present failure intensity) in this case is same as 0 (initial failure
intensity).

V  P   
100 10
Now,    0 Ln  10 Ln   6.93 CPU hr.
5
 F 
0


 (ii) Logarithmic execution time model
1  P 
 Ln
 
 F 
  
1 30
 0.025 Ln   71.67 Failures
 5 
 1 1  1 
   F  P 
 
 
1 1 1
 0.025 Ln    6.66 CPU hr.
 5 30 
Logarithmic model has calculated more failures in almost some duration of execution time
initially.
(b) Failure intensity objective F  = 0.5 failures/CPU hr.

(i) Basic execution time model

V0   P   F V0
   P 
Ln
 0  0 
  F 

100  
 (10  0.5)  95 failures 100 10
10  Ln   30
10 0.05 
CPU /hr 
(ii) Logarithmic execution time model

1 
 P 
θ Ln 
  F 
1 
30

 0.025 Ln   164 failures
0.5 

 1 1  1 
 θ  F P 
 
 
1 1 1
 
    78.66 CPU
0.025 0.5 30 
/hr
▪ Calendar Time Component
The calendar time component is based on a debugging process model. This
model takes into account:

1. resources used in operating the program for a given execution

time and processing an associated quantity of failure.
2. resources quantities available, and

3. the degree to which a resource can be utilized (due to

bottlenecks) during the period in which it is limiting.

Table 7.7 will help in visualizing these different aspects of the resources, and
the parameters that result.
Resource usage
Usage parameters Planned parameters
requirements per
Resource CPU hr Failure Quantities Utilisation
available
Failure identification
personnel
I µI PI 1
Failure correction
personnel
0 µf Pf Pf
Computer time c µc Pc Pc

Fig. : Calendar time component resources and parameters

Hence, to be more precise, we have

X C   c    c (for computer time)


Xf f (for failure correction)


X I   I    I (for failure identification)

dxT / d   r  r

Calendar time to execution time relationship

dt / d  (1/ Pr pr )dxT / d

dt / d  (r  r  ) / Pr pr
Fig.7.20: Instantaneous calendar time to execution time
ratio
Fig.7.21: Calendar time to execution time ratio for different limiting resources
Example- 7.4

A team run test cases for 10 CPU hrs and identifies

25 failures. The effort required per hour of execution
time is 5 person hr. Each failure requires 2 hr. on an
average to verify and determine its nature. Calculate
the failure identification effort required.
 Solution
As we know, resource usage is:

X r   r  r

Here   25 failures
θ r  15 person r  2
hr. hrs./failure
  10 CPU hrs.
Hence, Xr = 5 (10) + 2 (25)

= 50 + 50 = 100 person hr.

 Example- 7.5
Initial failure intensity (0 ) for a given software is 20 failures/CPU hr. The failure intensity
objective (F ) of 1 failure/CPU hr. is to be achieved. Assume the following resource
usage parameters. Resource Usage Per hour Per failure
Failure identification effort 2 Person hr. 1 Person hr.
Failure Correction effort 0 5 Person hr.
Computer time 1.5 CPU hr. 1 CPU hr.

(a)What resources must be expended to achieve the reliability improvement? Use the
logarithmic Poisson execution time model with a failure intensity decay parameter of
0.025/failure.

(b)If failure intensity objective is cut to half, what is the effect on requirement of resources?
Solution

(a)   1 Ln 
P
  F 


 
1 20
 0.025 Ln   119
failures  1 

 1  1 
 1  F P 
 
1 1 1  1
   
0.025
1 0.05  38 CPU
0.025  1 20 
hrs.
Hence X1  1  θ1

= 1 (119) + 2 (38) = 195 Person hrs.

XF  F 
= 5 (119) = 595 Person hrs.

X C  c   θc 
= 1 (119) + (1.5) (38) = 176 CPU hr.
 (b) F  0.5 failures/CPU
hr.  
1 20
 0.025 Ln   148
failures 1  1 0.5 1 
 
  
 78 CPU hr.
0.025  0.5 20 
So, XI = 1 (148) + 2 (78) = 304 Person hrs.

XF = 5 (148) = 740 Person hrs.

XC = 1 (148) + (1.5)(78) = 265 CPU hrs.

Hence, if we cut failure intensity objective to half, resources
requirements are not doubled but they are some what less. Note
that  is approximately doubled but increases logarithmically.
Thus, the resources increase will be between a logarithmic
increase and a linear increase for changes in failure intensity
objective.
Example- 7.6

A program is expected to have 500 faults. It is also assumed that one

fault may lead to one failure only. The initial failure intensity was 2
failures/CPU hr. The program was to be released with a failure intensity
objective of 5 failures/100 CPU hr. Calculated the number of failure
experienced before release.
Solution

The number of failure experienced during testing can be

calculated using the equation mentioned below:
V0
  P  F
 0

Here V0  500 because one fault leads to one
failure
0  2 failures/CPU hr.
F  5 failures/100 CPU hr.
 0.05 failures/CPU hr.
 500
So   2 
0.05
2
= 487 failures
Hence 13 faults are expected to remain at the release instant of
the software.
▪ ISO 9000
The SEI capability maturity model initiative is an attempt to improve software quality by
improving the process by which software is developed.
ISO-9000 series of standards is a set of document dealing with quality systems that
can be used for quality assurance purposes. ISO-9000 series is not just software
standard. It is a series of five related standards that are applicable to a wide
variety of industrial activities, including design/ development, production, installation,
and servicing. Within the ISO 9000 Series, standard ISO 9001 for quality system is
the standard that is most applicable to software development.
▪ Mapping ISO 9001 to the CMM
1. Management responsibility 10. Inspection, measuring and test
equipment
2. Quality system
11. Inspection and test status
3. Contract review
12. Control of nonconforming product
4. Design control
13. Corrective action
5. Document control
14. Product identification and
6. Purchasing traceability
7. Purchaser-supplied product 15. Quality records
8. Process control

9. Inspection and testing

15. Handling, storage, packaging and delivery
16. Internal quality audits

17. Training
18. Servicing
19. Statistical techniques
▪ Contrasting ISO 9001 and the CMM

There is a strong correlation between ISO 9001 and the CMM, although some
issues in ISO 9001 are not covered in the CMM, and some issues in the CMM are not
addressed in ISO 9001.

The biggest difference, however, between these two documents is the emphasis of
the CMM on continuous process improvement.

The biggest similarity is that for both the CMM and ISO 9001, the bottom line is
“Say what you do; do what you say”.