Software Quality

Engineering
Lecture 07

Instructor: Shimza Butt

2 Software Testing
 Software Testing is a method to check whether the actual
software product matches expected requirements and to ensure
that software product is Defect free.
 It involves execution of software/system components using manual or
automated tools to evaluate one or more properties of interest.
 The purpose of software testing is to identify errors, gaps or missing
requirements in contrast to actual requirements.
 Testing is the process of executing a program with the aim of finding
errors. To make our software perform well it should be error-free. If
testing is done successfully, it will remove all the errors from the
software.
 Software testing can be stated as the process of verifying and validating
that software or application is bug-free, meets the technical
requirements as guided by its design and development, and meets the
user requirements effectively and efficiently with handling all the
exceptional and boundary cases.
3 Software Testing (Cont..)

 Software testing can be divided into two steps:

1. Verification: it refers to the set of tasks that ensure that
software correctly implements a specific function.
 2. Validation: it refers to a different set of tasks that ensure that
the software that has been built is traceable to customer
requirements.
 Verification: “Are we building the product, right?”
Validation: “Are we building the right product?”
Types of Testing (Dynamic)

 Black Box Testing

 White Box Testing
 Gray Box Testing
5 Black Box Testing

 In science and engineering, a black box is a device, system or object

which can be viewed solely in terms of its input, output and transfer
characteristics without any knowledge of its internal workings, that is,
its implementation is "opaque" (black).
 Also known as functional testing. A software testing technique whereby
the internal workings of the item being tested are not known by the
tester.
 For example, in a black box test on a software design the tester only
knows the inputs and what the expected outcomes should be and not
how the program arrives at those outputs.
 The tester does not ever examine the programming code and does not
need any further knowledge of the program other than its specifications.
6 Black Box Testing

• Tester can be non- • Chances of having

technical. repetition of tests that
• This testing is most are already done by
likely to find those bugs programmer.
as the user would find. • It is difficult to identify
• Testing helps to identify all possible inputs in a
the contradiction in limited testing time.
functional specifications.
• Test cases can be
designed as soon as the
functional specifications
are complete.
7 White-Box Testing
 White-box testing (also known as clear box
testing, glass box testing, transparent box
testing, and structural testing) is a method of
testing software that tests internal structures or
workings of an application, as opposed to its
functionality (i.e. black-box testing).
 The connotations of “clear box” and “glass box”
appropriately indicate that you have full visibility
of the internal workings of the software product,
specifically, the logic and the structure of the
code.
 In white-box testing an internal perspective of
the system, as well as programming skills, are
required and used to design test cases. The
tester chooses inputs to exercise paths through
the code and determine the appropriate outputs.
8 White-Box Testing

• As the knowledge of • As knowledge of code

internal coding structure and internal structure is
is prerequisite, it a prerequisite, a skilled
becomes very easy to tester is needed to carry
find out which type of out this type of testing..
input/data can help in
testing the application
effectively.
• The other advantage of
white box testing is that
it helps in optimizing the
code
9 Gray Box Testing

 Gray box testing is a software testing technique that uses a combination

of black box testing and white box testing. Gray box testing is not black
box testing, because the tester does know some of the internal workings
of the software under test.
 In gray box testing, the tester applies a limited number of test cases to
the internal workings of the software under test. In the remaining part of
the gray box testing, one takes a black box approach in applying inputs
to the software under test and observing the outputs.
 This is particularly important when conducting integration testing
between two modules of code written by two different developers,
where only the interfaces are exposed for testing.
10 Dynamic Testing
 black-box/functional testing
 verifies the correct handling of the external functions provided by the software or
whether the observed behavior conforms to user expectations or product
specifications
 The emphasis is on reducing the chances of encountering functional problems by
target customers.

 white/clear-box/structural testing
 verifies the correct implementation of internal units, structures and relations among
them
 When white box testing is performed ,failures related to internal implementations
can be observed, leading to corresponding faults being detected and removed.
11 Techniques of Testing
 Black Box Testing (BBT)
 White Box Testing (WBT)
 Software Testing can be majorly classified into two categories:
1. Black Box Testing is a software testing method in which the internal
structure/ design/ implementation of the item being tested is not known to
the tester
2. White Box Testing is a software testing method in which the internal
structure/ design/ implementation of the item being tested is known to the
tester.
12 QA vs. Testing
 Black Box Testing Methods
 Equivalence Class Partitioning  State Transition Testing 
Testing Fuzzy Testing
 Installation Testing
 Boundary Value Analysis/  Comparison Testing
Testing  Security Testing  End-to-End Testing
 Omission Testing  Integration Testing  Localization Testing
 Null Case Testing  Compatibility Testing  Globalization Testing
 Volume Testing  Configuration Testing  Error Handling/ Guessing
 Load Testing  Documentation Testing Testing
 Stress Testing  Smoke Testing  Sandwich Testing
 Performance Testing  Sanity Testing  Decision Table Testing
 Resource Testing  Usability Testing  Graph-Based Testing
 Requirements/Specification  Exploratory Testing
Testing
 Button Press Testing
14 Black Box Testing
 The primary source of black-box testing is a specification of requirements
that are stated by the customer.
 It is a software testing technique that examines the functionality of the
software without knowing its internal structure or coding.
 It does not require programming knowledge of the software.
 All test cases are designed by considering the input and output of a
particular function.
 In this testing, the test engineer analyzes the software against
requirements, identifies the defects or bugs, and sends it back to the
development team.
 In this method, the tester selects a function and gives input value to
examine its functionality, and checks whether the function is giving the
expected output or not. If the function produces the correct output, then it
is passed in testing, otherwise failed.
15 Black Box Testing (Cont..)
 Black box testing is less exhaustive than White Box and Grey Box testing
methods.
 It is the least time-consuming process among all the testing processes.
 The main objective of implementing black box testing is to specify the business
needs or the customer's requirements.
 In other words, we can say that black box testing is a process of checking the
functionality of an application as per the customer's requirement.
 The black box is a powerful technique to check the application under test from the
user’s perspective.
17 Black Box Testing Example
 A simple login screen of software or a web application will be tested for
seamless user login. The login screen has two fields, username and
password as an input and the output will be to enable access to the
system.
 A black box testing will not consider the specifications of the code, and it
will test the valid username and password to login to the right account.
 This form of testing technique will check the input and output.
• A user logged in when inputs a present username and correct password
• A user receives an error message when enters username and incorrect
password
 The black box testing is also known as an opaque, closed box, function-
centric testing. It emphasizes on the behavior of the software. Black box
testing checks scenarios where the system can break.
 For example, a user might enter the password in the wrong format, and
a user might not receive an error message on entering an incorrect
password.
18 When we do Black Box Testing?

• Unlike traditional white box testing, black box testing is beneficial for
testing software usability.
• The overall functionality of the system under test
• Black box testing gives you a broader picture of the software.
• This testing approach sees an application from a user’s perspective.
• To test the software as a whole system rather than different modules.
 Equivalence Class Partitioning Testing
19
 Equivalence Partitioning is a black-box testing method that divides the
input domain of a program into classes of data from which test cases
can be derived
 An ideal test case single handedly uncovers a class of errors e.g
incorrect processing of all character data that might otherwise require
many cases to be executed before the general error is observed.
 Equivalence Partitioning strives to define the test case that uncovers
classes of errors, there by reducing the total number of test cases that
must be developed.
 An equivalence class represents a set of valid or invalid states for input
conditions.
20
Equivalence partitioning
 This test case designing techniques checks the input and output by dividing
the input into equivalent classes. The data must be tested at least once to
ensure maximum test coverage of data. It is the exhaustive form of testing,
which also reduces the redundancy of inputs.
 For example: Taking inputs for a test case data for the example mentioned
above will have three classes from which one data will be tested.
 Valid class: 1 to 100 (any number), Invalid class: -1 (checking the lowest of
lowest), Invalid class: 101(highest of highest).
 It is often seen that many type of inputs work similarly so instead of giving
all of them separately, they can be grouped together and test only one input
of each group.
 The idea is to partition the input domain of the system into a number of
equivalence classes such that each member of class works in a similar way,
i.e., if a test case in one class results in some error, other members of class
would also result into same error.
21 Equivalence partitioning (Cont..)
 The technique involves two steps:
1. Identification of equivalence class – Partition any input domain into minimum
two sets: valid values and invalid values. For example, if the valid range is 0 to
100 then select one valid input like 49 and one invalid like 104.
2. Generating test cases –
(i) To each valid and invalid class of input assign unique identification number.
(ii) Write test case covering all valid and invalid test case considering that no two
invalid inputs mask each other.
3. To calculate the square root of a number, the equivalence classes will be:
(a) Valid inputs:
1. Whole number which is a perfect square- output will be an integer.
2. Whole number which is not a perfect square- output will be decimal number.
3. Positive decimals
4. (b) Invalid inputs:
1. Negative numbers (integer or decimal).
2. Characters other that numbers like “a”,”!”,”;”,etc.
22
Equivalence Class Partitioning Testing
 Equivalence classes can be defined according to the following guidelines;
 If an input condition specifies a range one valid and two invalid
equivalence classes are defined.
 If an input condition specifies a specific value one valid and two invalid
equivalence classes are defined.
 If an input condition specifies a member of a set one valid and one invalid
equivalence class are defined.
 If an input condition is Boolean, one valid and one invalid class are
defined.
23 Boundary Value Testing
(Specific case of Equivalence Class Partitioning Testing)
 Boundary value analysis leads to a selection of test cases that
exercise bounding values. This technique is developed because a
great number of errors tend to occur at the boundary of input
domain rather than at the center.
 Tests program response to extreme input or output values in
each equivalence class.
 Guideline for BVA are following;
 If an input condition specifies a range bounded by values a and b,
test cases should be designed with values a and b and just above
and below a and b.
24 Boundary Value Analysis
 It is the widely used black-box testing, which is also the basis for
equivalence testing. Boundary value analysis tests the software with test
cases with extreme values of test data. BVA is used to identify the
flaws or errors that arise due to the limits of input data.
 Boundaries are very good places for errors to occur. Hence if test cases
are designed for boundary values of input domain then the efficiency of
testing improves and probability of finding errors also increase.
 For example: Taking inputs for a test case data for an age section should
accept a valid data of anything between 1-100. According to BVP analysis,
the software will be tested against four test data as -1, 1, 100, and 101 to
check the system’s response using the boundary values.
25 Fuzzy Testing
 Fuzzy testing or fuzzing is a software testing technique, often automated or
semi-automated, that involves providing invalid or unexpected data to the
inputs of a computer program. The program is then monitored for exceptions
such as crashes or failing built-in code assertions or for finding potential
memory leaks.
 The term first originates from a class project at the University of Wisconsin
1988 although similar techniques have been used in the field of quality
assurance, where they are referred to as robustness testing or negative testing.
26 Omission Testing
 Omission Testing (also called Missing Case Testing):
 Exposes defects caused by inputting cases (scenarios) the developer
forgot to handle or did not anticipate
 A study by Sherman on a released Microsoft product reported that 30% of
client reported defects were caused by missing cases.
 Other studies show that an average of 22 to 54% of all client reported defects
are caused by missing cases.
 Omission Testing in software testing refers to the process of
identifying and verifying whether any required functionalities,
components, or requirements are missing or left out in the software
system. It is a type of negative testing aimed at uncovering defects
due to incomplete implementation or oversight.
 Null Case Testing
27
Null Testing: (a specific case of Omission Testing, but triggers
defects extremely often)
 Exposes defects triggered by no data or missing data.
 Often triggers defects because developers create programs
to act upon data, they don’t think of the case where the
project may not contain specific data types
 Example: X, Y coordinate missing for drawing various
shapes in Graphics editor.
 Example: Blank file names
28 Comparison Testing
 There are some situations in which the reliability of software is
absolutely critical. In such applications, redundant hardware and
software are often used to minimize the possibility of error.
 When redundant software is developed, separate software engineering
teams develop independent versions of an application using the same
specification.
 In such situations, each version can be tested with the same test data to
ensure that all provide identical output
 Then all versions are executed in parallel with real-time comparison of
results to ensure consistency
 These independent versions form the basis of black box testing
technique called comparison testing or back-to-back testing
29 Comparison Testing
 If the output from each version is the same, it is assumed that
all implementations are correct
 If the output is different, each of the applications is
investigated to determine if the defect in one or more versions
is responsible for the difference
 Comparison testing is not foolproof; if the specification from
which all versions have been developed is in error, all versions
will likely reflect the error.
 In addition, if each of the independent versions produces
identical but incorrect results, comparison testing will fail to
detect the error.
Where It’s Used?

• During system migration or platform transitions.

• When rewriting an application in a new language or framework.
• When replacing third-party components.
• In regression testing without formal expected outputs.
31 End to End Testing
 End-to-end testing is a methodology used to test whether the
flow of an application is performing as designed from start to
finish.
 End-to-end testing involves ensuring that that integrated
components of an application function as expected.
 This is basically exercising an entire “workflow”. Although
System Testing is similar, in System Testing you do not have to
complete the entire workflow, but may exercise individual
interfaces one at a time.
 For example, a simplified end-to-end testing of an email
application might involve logging in to the application, getting
into the inbox, opening and closing the mail box, composing,
forwarding or replying to email, checking in the sent items and
logging out of the application.
32 SQA Team
 Unlike System Testing, End-to-End Testing not only validates the
software system under test but also checks it’s integration with
external interfaces. Hence, the name “End-to-End”.
 End to End Testing is usually executed after functional and system
testing. It uses actual production like data and test environment to
simulate real-time settings. End-to-End testing is also called Chain
Testing
Aspect System Testing End-to-End Testing
Tests the entire
Tests the complete system
Definition workflow, from start to
as a whole.
finish.
Focuses on verifying if the Covers the real-world
Scope entire app meets usage, including external
requirements. systems.
Simulate full flow: User
Test composing, sending,
Example (Email logs in → sends email →
receiving emails within the
App) receiver gets it → logs
app.
out.
Involves external
Mostly internal systems like email
Dependencies
components. servers, databases, 3rd-
party APIs.
Ensure business
Validate the system as a
Goal processes work across
whole is working correctly.
multiple systems.
Type Black Box Testing Black Box Testing
Often controlled and Uses realistic or
Test Data
limited to system modules. production-like data.
High – tests across
Medium – tests the system
Complexity systems, browsers,
internally.
networks, etc.
34

Globalization Testing
 Globalization Testing is the software testing process for checking if
the software can work properly in various culture/locale settings using
every type of international input.
 Globalization Testing ensures your app is ready for international
use.
35 Localizing Testing
 Localization is the process of customizing a software application that was
originally designed for a domestic market so that it can be released in foreign
markets.
 This process involves translating all native language strings to the target
language and customizing the GUI so that it is appropriate for the target
market
 Localization testing checks how well the build has been translated into a
particular target language (e.g., Japanese product for Japanese user).
 We should invite the local staff to help with our localization testing by
checking the quality of translation as well.
 Common bugs found from this testing
 Cannot display the correct format
 Functionality is broken
36 Error Handling Testing
 Error-handling testing determines the ability of the application system to
properly process incorrect transactions.
 Specific objectives of error-handling testing include:
 Determine that all reasonably expected error conditions are recognizable by the
application system.
 Determine that the accountability for processing errors has been assigned
 Determine that the procedures provide a high probability that the error will be
properly corrected.
 Error-handling Test examples:
 Produce a representative set of transactions containing errors and enter them into
the system to determine whether the application can identify the problems.
 Enter improper master data, such as prices or employee pay rates, to determine if
errors that will occur repetitively are subjected to greater scrutiny (inspection/
analysis) than those causing single error results.
Aspect Error Handling Testing Fuzz Testing
Involves feeding the system with
Verifies how a system responds to known random, unexpected, or invalid data
Definition
invalid or error conditions. to discover unknown vulnerabilities or
crashes.
To uncover unknown bugs, crashes,
To ensure the system handles predictable
Purpose or security flaws by overwhelming the
errors gracefully.
system with faulty data.
Planned and controlled testing using Uncontrolled/randomized input
Approach
specific invalid inputs or fault scenarios. generation to stress-test the system.
Validates error messages, exception Focuses on breaking the system or
Focus
handling, and system recovery. identifying unhandled failures.
Carefully designed (e.g., blank fields, wrong Random strings, malformed inputs,
Test Data
credentials, null values). corrupted files, etc.
Manual test cases or tools for structured Fuzzers like AFL, Peach Fuzzer, Burp
Tools Used
validation. Suite, etc.
Who Performs Security testers, penetration testers,
QA testers, developers.
It? advanced testers.
Type of Both black box and white box depending Primarily black box (no need to know
Testing on level of access. internal structure).
Primary Use Ensure user-facing and internal errors Identify hidden bugs, memory leaks,
Case are handled properly. and security flaws.
38 State Transition Testing
 This testing technique uses the inputs, outputs, and the state of
the system during the testing phase. It checks the software against
the sequence of transitions or events among the test data.
 Based on the type of software that is tested, it checks for the
behavioral changes of a system in a particular state or another
state while maintaining the same inputs.
 For example, A login page will let you input your username and
password for three attempts. Each incorrect password will send the
user to the login page. After the third attempt, the user will be sent
to an error page. This state transition method considers the various
states of the system and the inputs to pass only the right sequence
of testing.
39 Decision Table Testing
 This approach creates test cases based on various possibilities. It
considers multiple test cases in a decision table format where each
condition is checked and fulfilled, to pass the test and provide accurate
output. It is preferred in case of various input combinations and multiple
possibilities.
 For example, A food delivery application will check various payment
modes as input to place the order — decision making based on the table.
 Case1: If the end-user has a card, then the system will not check for cash
or coupon and will take action to place the order.
 Case2: If the end-user has a coupon will not be checked for a card or cash
and action will be taken.
 Case3: if the end-user has cash, the action will be taken.
 Case4: If the end-user doesn’t have anything, then action will not be
taken.
40 Steps to perform BBT
• The first step to black-box testing is to understand the requirement
specifications of the application under test. An accurate and precise
SRS document should be there.
• The next step is to evaluate the set of valid inputs and test scenarios to test the
software. The goal is to save time and get good test coverage.
• Prepare the test cases to cover a maximum range of inputs.
• The test cases are run in the system to generate output, which is validated with
the expected outcome to mark pass or fail.
• The failed steps are marked and sent to the development team to fix them.
• Retest the system using various testing techniques to verify its recurring nature
or to pass it.
 The black box testing can be easily used to check and validate the entire
software development life cycle. It can be used at various stages such as unit,
integration, acceptance, system, and regression to evaluate the product.
41 Benefits of BBT
 Black box testing helps to find the gaps in functionality, usability, and other
features.
 This form of testing gives an overview of software performance and its output.
 It improves software quality and reduces the time to market.
 This form of testing mitigates the risk of software failures at the user’s end.
 The tester doesn’t need any technical knowledge to test the system. It is
essential to understand the user’s perspective.
 Testing is performed after development, and both the activities are
independent of each other.
 It works for a more extensive coverage which is usually missed out by testers
as they fail to see the bigger picture of the software.
 Test cases can be generated before development and right after specification.