University of Information Technology
Lecture: MSc. Nguyen Thi Thanh Truc
Email: trucntt@[Link]
Error, Fault, Failure
Software Requirements
Testing
Validation & Verification
Error, Fault, Failure
Software Requirements
Testing
Validation & Verification
• Definition
• Cost of defects
• Defect Life Cycle
• Severity and Priority
• Error
– Error is a human action producting an incorrect
result.
– When programmers make errors they introduce
faults to program code.
– Errors are not just accidents or mistakes that can
be solved by “just being more careful".
– Errors are not an act of incompetence.
– Errors are inevitable in a complex activitiy.
• Fault
– A manifestation of human error in software.
– Also known as a defect or bug.
– In US, fault are more commonly known as defects.
– Bugs are a universally accepted name for faults.
– We normally think of programmers when we talk
about faults and human error.
– However, faults can also be created in documents by:
• managers working on project plans
• analysts working on requirements
• designers working on designs or
• testers working on test plans too.
• Fault
– Human error causes faults in any project
deliverable (requirements, design or code).
– However only faults in software cause software to
fail.
– Faults are static
– Software faults are static (immobile)
– When we test software, it is easy to believe that
the faults in the software move.
• Fault
– Once injected into the software, they will remain
there until exposed by a test and fixed.
– They are discovered either by inspecting code
(reading the code) or by inferring theiry existence
from software failures (by executing the software)
– When the software fails we can infer the existence
of a fault.
• Failures
– A deviation of the software from its expected
delivery or service
– A failure occurs when software does the “wrong"
thing
– Most of the time software does the right thing
– Software faults cause software failures when the
program is executed with a set of inputs the
expose the fault.
A developer makes an
ERROR
… and injects a FAULT into the
software
… and the fault causes software
to FAIL
BRAINTEASER
Error, Fault, Failure
Software Requirements
Testing
Validation & Verification
• There are 3 types:
– Explicit Requirements: requirements written in
documents --> functional requirements
– Implicit Requirements: requirements that users
are going to expect that were not captured
explicitly. Examples: performance, usability and
security. non - functional requirements
– Latent Requirements: things that delight
customers.
Error, Fault, Failure
Software Requirements
Testing
Validation & Verification
• Validation & Verification
• Independent Validation & Verfication
• Capability Maturity Model (CMM)
• CMM Levels
• Done by third party organization not involved
in the development of the product.
• Third party can be another internal team or
external (other company)
developer independent tester
Understands the system Must learn about the system,
but, will test "gently" but, will attempt to break it
and, is driven by quality
• Independent
– Technical: IV&V prioritizes its own efforts
– Managerial: Independent reporting route to Program Management
– Financial: Budget is allocated by program and controlled at high level such that
IV&V effectiveness is not compromised
• Verification (Are we building the product right?)
– The process of determining whether or not the products of a given phase of the
software development cycle fulfill the requirements established during the
previous phase
– Is internally complete, consistent and correct enough to support next phase
• Validation (Are we building the right product?)
– The process of evaluating software throughout its development process to ensure
compliance with software requirements. This process ensures:
• Expected behavior when subjected to anticipated events
• No unexpected behavior when subjected to unanticipated events
• System performs to the customer’s expectations under all operational conditions
Software IV&V is a systems engineering process employing rigorous
methodologies for evaluating the correctness and quality of the
software product throughout the software life cycle
Adapted to characteristics of the target program
Requirements Phase
•System Reqts Design Phase
Analysis
•S/W Reqts •Design Analysis Code Phase
Analysis •Interface Analysis
•Test Program •Code Analysis Test Phase
•Interface Analysis •Test Program
•Process Analysis Analysis •Test Program
•Supportability Analysis
•Technical Reviews •Supportability Analysis
& Audits Analysis •Independent Test
•Process Analysis Analysis
•Process Analysis •Supportability
•Technical Reviews Analysis
& Audits •Technical Reviews
& Audits •Technical Reviews
& Audits
Verify
Verify
Verify
Validate
Catastrophic/Critical/High Risk Functions List
Traceability Analysis
I ssues Tracking
M etrics Assessment
Loading Analysis
Change I mpact Analysis
Special Studies
• IV&V Process provides tools and analysis procedures appropriate to each
phase of the software development life cycle:
– Formulation Phase:
• Is development process sound, repeatable, and managed?
– Requirements Phase:
• Verify that system and software requirements are correct, complete, traceable and
testable
• Analyze system-level requirements: Are test plans and acceptance criteria sufficient to
validate system requirements and operational needs?
• Are testing methods sufficient to verify and validate software requirements?
• Are the correct software development, management, and support processes in place?
– Design Phase:
• Does the design support the requirements?
• Are test plans and test environments sufficient to verify and validate software and
operational requirements?
• Does the design have any characteristics that will cause it to fail under operational
scenarios? What solutions are appropriate?
• Typical IV&V functions by Software life-cycle phase
(cont.):
– Coding Phase:
• Does the code reflect the design?
• Is the code correct?
• Verify that test cases trace to and cover software requirements and operational needs
• Verify that software test cases, expected results, and evaluation criteria fully meet testing
objectives
• Analyze selected code unit test plans and results to verify full coverage of logic paths, range of
input conditions, error handling, etc.
– Test Phase:
• Analyze correct dispositioning of software test anomalies
• Validate software test results versus acceptance criteria
• Verify tracing and successful completion of all software test objectives
– Operational Phase:
• Verify that regression tests are sufficient to identify adverse impacts of changes
• IV&V identifies deficiencies in program’s test
planning
• Program changes their procedures to address
deficiencies vice IV&V independently test
• IV&V may independently test highly critical
software using an IV&V testbed
– Whitebox
– Stress
– Endurance
– Limit
• Developer motivated to show software works
• IV&V attempts to break software
Integrates IV&V into program Reflects IV&V in program mgmt plan
Provides IV&V funding Agrees to data transfer plan
Resolves Exception issues Reflects agreement in subcontracts
Defines IV&V scope
Program and objectives
Planning Requirements
agreement
IV&V proposal
CARA results
information
Design
Developer Code
Test
IV&V Assessment Planning Execute
Normal
Introduce issues at lowest level
Allow developer time to respond
IV&V in phase
Issue resolved
Exception with development
Introduce issues at lowest level
Allow developer time to respond
If no resolution, take issue to program
• Scope is determined so as to minimize the risk within the
Program’s IV&V budget. Effort is based on:
– Criticality and risk of system functions performed/managed by
software
– Budget limitations
Program’s
IV&V budget
Specifications Program goals Dev plans/schedules
Criticality Analysis Estimate
resource
Yes IV&V
and Acceptable?
requirements Plan
Risk Assessment
No
Revise scope:
- Breadth vs depth
- Exceptions
Technical Management
•Better software/system •Better Visibility into
Performance Development
•Higher Confidence in •Better Decision Criteria
Software Reliability
•Second Source Technical
•Compliance between Alternative
Specs & Code
•Reduced maintenance cost
•Criteria for Program
Acceptance •Reduced Frequency of
Operational Change