Software Quality Assurance SE422

Lecture 1: Introduction to SW Testing

Outline

▪ Why do we Test Software?

Example of SW catastrophic failures
−
▪ Whats is Software Testing?
▪ What is the Goal of Software Testing?
▪ History of SW Testing
▪ Evolution of SW Testing

2
Why do we Test Software?

▪ Think about your own

experience.
▪ Where do you see or use
software?
▪ What kind of SW failures you
encountered or suffered from?
▪ What could go wrong if
software is not tested
(appropriately, adequately,
or not test at all)?

3
Some Software Failures-I
▪ 2024: A fault in the CrowdStrike’s software update crashed machines running Microsoft’s
Windows operating systems, resulting in an outage and massive domino effect on many
companies, including major global banks, airlines, and healthcare and energy companies.
▪ 2023: Flights from coast to coast were grounded and unable to depart due to a software
–
malfunction (FAA outage).
▪ 2023: Military helicopter crash caused by failure to apply a software patch –
–
▪ 2022: Tesla recalled nearly 12K vehicles due to battery controller failures.
▪ 2022: Millions of web server vulnerability due to a defect in the Log4j software
–
▪ 2021: Log4j did not sanitize its input, allowing malicious attackers to execute code remotely
on any targeted computer.
▪ –2020: More than 100 flights to and from London’s Heathrow airport disruption due to issues
with departure boards and check-in systems.
▪ –2020: A number of Hyundai and Kia recalls due to software park system malfunction.
▪ –2020: Microsoft Azure experienced a six-hour outage due to issues with the building
automation control system that caused a cooling system failure.

4
 Some Software Failures-II
▪ 2020: Google Cloud service disruptions affecting Gmail, Google Classroom, Nest, YouTube
due to storage issues with Google’s authentication system and email configuration update
▪ 2020: AWS suffered over 6 hours due to an operating system configuration issue .
–
▪ 2019: Facebook, Instagram, WhatsApp 14 hours downtime due to Facebook News Feed issue
in routine maintenance.
▪ 2019: Boeing 737 Max crashed due to aggressive software flight overrides.
–
▪ 2018: Hawaii Emergency Management Agency sent out a false missile alert due to no visible
–
alterations between live alert and testing environments.
▪ 2018: Pedestrian in Arizona was killed by an Uber car due to its self-driving software failure.
–
▪ 2018: Google shut down Google+ due to the undetected fault that was present for more
–
than two years, causing nearly 500,000 users’ data to be compromised.
▪ 2018: TSB system upgrade causes months of online banking disruption.
–
▪ 2017: Cloudflare’s major software fault led to customer sensitive data leakage.
–
▪ 2017: 606 recorded software failures, impacting 3.7 billion people, 314 companies, $1.7
–
trillion in financial losses.
5
 Some Software Failures-III
▪ 2016: Nissan recalled 4 millions cars from the market due to software failure in the airbag sensory
detectors.
▪ 2016: Info lost due to the browser back button while using TurboTax software.
–
▪ 2015: Bloomberg’s trading terminal failures forced the British government to postpone $4.4
–
billion debt sale.
▪ 2014: Dropbox’s outage was due to a fault in a maintenance script.
–
▪ 2012: Faults in a new Knight Capital’s trading software causes $440 millions.
–
▪ 2007: Symantec concluded that most security vulnerabilities are due to faulty software.
–
▪ 2003: Northeast blackout due to the alarm system in the energy management system failure,
–
affecting 40 million people in 8 US states, 10 million people in Ontario, Canada.
▪ 1999: NASA’s Mars lander crashed due to a unit integration fault.
–
▪ 1997: Ariane 5 explosion: Exception-handling bug forced self-destruct on maiden flight (64-bit to
–
16-bit conversion).
▪ –1986: 3 patients were killed by Therac-25 radiation machine due to poor testing of its safety-
critical software.
6
Spectacular & Catastrophic SW Failure: Ariane 5

▪ 10 years and $7 billion to produce

▪ < 1 min to explode
▪ the error came from a piece of the software
▪ that was not needed during the crash
▪ programmers thought that this particular value
would never become large enough to cause trouble
▪ removed the test present in Ariane 4 software
▪ • 1 bug = 1 crash

7
Cost of late SW Testing
▪ Testing is the most time consuming and expensive part of software development
▪ Not testing is even more expensive
▪ Having too little testing effort early increases the testing cost
▪ Planning for testing after development is prohibitively expensive
▪ A tester’s goal is to eliminate faults as early as possible

8
So, why do we test software?

▪ Find sources of problems and get them fixed

▪ Check intraoperability and interoperability
▪ Help in decision making (release/no-release)
▪ Minimize technical support costs
▪ Assess conformance and compliance
▪ Minimize safety-related lawsuit risk
▪ Determine safe scenarios for use of the product
➔ Increase confidence for anyone who is affected through
some forms of evidence

9
Whats is Software Testing?
▪ IEEE defines software testing as:
− A process of analyzing a software item to detect the differences
between existing and required conditions (that is defects/errors/bugs)
and to evaluate the features of the software item.
− Note that we need:
⚫ to know the required conditions
⚫to be able to observe the existing conditions
▪ Testing focuses on:
− behavioral (what the program does)
− and structural (how the program is) aspects
10
Test Case
▪ Test Case: test case has an identity and is associated with a program behavior. A test case also
has a set of inputs and a list of expected outputs. Test case consists of:
− test input value(s)
− expected result(s)
▪ Test Case Structure:
− Identifier – persistent unique identifier of the test case
− Setup/environment/preconditions
− How to perform the test – including input data
− Expected output data – including how to evaluate it
− Purpose – what this test case is supposed to show
− Link to requirements/user story/use case/design/model
− Related test cases
− Author, date, revision history
11
Program Behavior

12
Program Behavior & Test

13
What is the Goal of Software Testing?
▪ Some common answers:
− Do we want to isolate and fix bugs in the program?
− Do we want to demonstrate that the program works?
− Do we want to demonstrate that the program doesn’t work?
− Do we want to reduce the risk involved in using the program?
− Do we want to have a methodology for producing better quality
software?

14
Definitions: Error, Fault
▪ Error: An error refers to a human mistake or action that introduces incorrectness into the
[Link] people make mistakes while coding, we call these mistakes bugs.
− Errors may occur during the development or operational phase and is often caused by
misinterpretation, omission, or misunderstanding of requirements. For example, a
developer might misinterpret a requirement and write incorrect code. Errors are the
root cause of faults (defects) in the software.
▪ Fault: a fault is the result of an error. More precisely, a fault is the representation of an
error, where representation is the mode of expression, such as narrative text, data flow
diagrams, hierarchy charts, source code, etc.
− Two types of faults: faults of commission and faults of omission.
⚫ Fault of commission occurs when we enter something into a representation that
is incorrect.
⚫ Fault of omission occurs when we fail to enter correct information.
− Faults of omission are more difficult to detect and resolve.
15
Definitions: Failure, Incident
▪ Failure: a failure occurs when a fault executes.
− failures only occur in an executable representation, which is
usually taken to be source code, or more precisely, loaded object
when we run the software.
▪ Incident: an incident is the symptom associated with a failure that
alerts the user to the occurrence of a failure. When a failure occurs,
it may or may not be readily apparent to the user (or customer or
tester).

16
 Error => Fault => Failure => Incident
Test: testing is obviously concerned with errors, faults, failures, and
incidents. A test is the act of exercising software with test cases. A test
has two distinct goals: to find failures or to demonstrate correct
execution.

17
Software Testing LifeCycle

18
Who Tests Software?
▪ Software testing is not just for testers.
▪ As a developer:
− It is also your responsibility to ensure the quality of your
product.
− Tests are the tool to help you with that responsibility.
− If you design tests properly, you can test your code in an
effective and systematic way.

19
Limitations of Software Testing?
▪ Testing cannot prove correctness
▪ Testing can demonstrate the presence of failures
▪ Testing cannot prove the absence of failures
▪ Testing by itself (alone) doesn’t make software better
− Testing must be combined with fault resolution to improve
software

20
History of SW Testing

[image: [Link] 21
Testing in 21st Century
⚫ More safety critical, real-time software
⚫ Embedded software is ubiquitous … check your pockets
⚫ Enterprise applications means bigger programs, more users
⚫ Paradoxically, free software increases our expectations !
⚫ Security is now all about software faults
⚫ Secure software is reliable software
⚫ The web offers a new deployment platform
− Very competitive and very available to more users
− Web apps are distributed
− Web apps must be highly reliable
⚫ Artificial Intelligence
⚫ Software testing becomes more important
⚫ We need reliable software. Testing is one way to assess reliability and thus improve
quality of software

22
Comments & Questions

23