Testing conventional/web/mobile applications
Pressman R S, Bruce [Link], “Software engineering - A
Practitioner’s Approach”, Eighth Edition, Tata McGraw-Hill, 2014
Chapter 23, 25, 26
Topics Covered
Topic Text Book Reference
Testing conventional applications 23.1, 23.2, 23.3, 23.6
Testing Web Apps 25.1, 25.2, 25.3, 25.4, 25.7, 25.8, 25.9,
Testing Mobile Apps 26.1, 26.2, 26.3
2
Testing fundamentals
• The goal of testing is to find errors, and a good test is one
that has a high probability of finding an error
• Therefore, you should design and implement a computer-
based system or a product with “testability” in mind
• At the same time, the tests themselves must exhibit a set
of characteristics that achieve the goal of finding the most
errors with a minimum of effort
• Testability - “ Software testability is simply how easily [a
computer program] can be tested.”
Dr Ganesh Neelakanta Iyer 3
Testability Characteristics
“The better it works, the more efficiently it can be
Testability Operability tested.”
“What you see is what you test.”
Observability
“The better we can control the software, the more the
Controllability testing can be automated and optimized.”
“By controlling the scope of testing, we can more quickly
Decomposability isolate problems and perform smarter retesting.”
“The less there is to test, the more quickly we can test
Simplicity it.”
“The fewer the changes, the fewer the disruptions to
Stability testing.”
“The more information we have, the smarter we will
Understandability test.”
Dr Ganesh Neelakanta Iyer 4
Test characteristics
• A good test has a high probability of finding an error
• A good test is not redundant
• A good test should be “best of breed”
• A good test should be neither too simple nor too complex
Dr Ganesh Neelakanta Iyer 5
Internal and external views of testing
Black-box testing
• alludes to tests that are conducted
at the software interface.
• A black-box test examines some
White-box testing
fundamental aspect of a system • of software is predicated on close
with little regard for the internal examination of procedural detail
logical structure of the software • Logical paths through the software and
collaborations between components are
tested by exercising specific sets of
conditions and/or loops
Dr Ganesh Neelakanta Iyer 6
White box testing
• White-box testing, sometimes called glass-box testing or structural
testing, is a test-case design philosophy that uses the control
structure described as part of component-level design to derive test
cases
• Using white-box testing methods, you can derive test cases that
– (1) guarantee that all independent paths within a module have been
exercised at least once,
– (2) exercise all logical decisions on their true and false sides,
– (3) execute all loops at their boundaries and within their operational
bounds, and
– (4) exercise internal data structures to ensure their validity.
Dr Ganesh Neelakanta Iyer 7
Black box testing
• Black-box testing , also called behavioral testing or functional
testing, focuses on the functional requirements of the
software
• That is, black-box testing techniques enable you to derive
sets of input conditions that will fully exercise all functional
• requirements for a program
• Black-box testing is not an alternative to white-box techniques
• Rather, it is a complementary approach that is likely to
uncover a different class of errors than white-box methods.
Dr Ganesh Neelakanta Iyer 8
Black box testing
• Black-box testing attempts to find errors in the following
categories:
– (1) incorrect or missing functions,
– (2) interface errors,
– (3) errors in data structures or external database access,
– (4) behavior or performance errors, and
– (5) initialization and termination errors.
• Unlike white-box testing, which is performed early in the
testing process, blackbox testing tends to be applied during
later stages of testing
Dr Ganesh Neelakanta Iyer 9
Graph-Based Testing Methods
• The first step in black-box testing is to
understand the objects that are
modeled in software and the
relationships that connect these
objects
• Once this has been accomplished, the
next step is to define a series of tests
that verify “all objects have the
expected relationship to one another”
Dr Ganesh Neelakanta Iyer 10
Graph-Based Testing Methods
• You can then derive test cases by
traversing the graph and covering
each of the relationships shown
• These test cases are designed in an
attempt to find errors in any of the
relationships
Dr Ganesh Neelakanta Iyer 11
Discussion
Model any of the scenarios in your project and derive
graph-based tests for the same.
Dr Ganesh Neelakanta Iyer 12
Equivalence Partitioning
• Equivalence partitioning is a black-box testing method
that divides the input domain of a program into classes of
data from which test cases can be derived
• An ideal test case single-handedly uncovers a class of
errors (e.g., incorrect processing of all character data) that
might otherwise require many test cases to be executed
before the general error is observed
• An equivalence class represents a set of valid or invalid
states for input conditions
Dr Ganesh Neelakanta Iyer 13
Boundary value analysis
• A greater number of errors occurs at the boundaries of the
input domain rather than in the “center.”
• It is for this reason that boundary value analysis (BVA) has
been developed as a testing technique
• Boundary value analysis leads to a selection of test cases
that exercise bounding values
• Boundary value analysis is a test-case design technique that
complements equivalence partitioning
• Rather than selecting any element of an equivalence class,
BVA leads to the selection of test cases at the “edges” of the
class
Dr Ganesh Neelakanta Iyer 14
Summary
• The primary objective for test-case design is to derive a set of tests
that have the highest likelihood for uncovering errors in software
• To accomplish this objective, two different categories of test-case
design techniques are used: white-box testing and black-box testing
• White-box tests focus on the program control structure. Test cases
are derived to ensure that all statements in the program have been
executed at least once during testing and that all logical conditions
have been exercised\
• Black-box tests are designed to validate functional requirements
without regard to the internal workings of a program
Dr Ganesh Neelakanta Iyer 15
Chapter 25, 25.1, 25.2, 25.3, 25.4, 25.7, 25.8, 25.9
TESTING WEB APPLICATIONS
Dr Ganesh Neelakanta Iyer 16
Dimensions of Quality
Content Function Structure Usability Navigability
Performance Compatibility Interoperability Security
Dr Ganesh Neelakanta Iyer 17
Errors within a WebApp Environment
1. Because many types of WebApp tests uncover problems that are first
evidenced on the client side, you often see a symptom of the error, not
the error itself
2. Because a WebApp is implemented in a number of different
configurations and within different environments, it may be difficult or
impossible to reproduce an error outside the environment in which the
error was originally encountered
3. Although some errors are the result of incorrect design or improper
HTML (or other programming language) coding, many errors can be
traced to the WebApp configuration
4. Because WebApps reside within a client-server architecture, errors can
be difficult to trace across three architectural layers: the client, the server,
or the network itself
5. Some errors are due to the static operating environment (i.e., the specific
configuration in which testing is conducted), while others are attributable
to the dynamic operating environment
Dr Ganesh Neelakanta Iyer 18
Testing Strategy
• The content model for the WebApp is reviewed to uncover errors
• The interface model is reviewed to ensure that all use cases can be accommodated
• The design model for the WebApp is reviewed to uncover navigation errors
• The user interface is tested to uncover errors in presentation and/or navigation
mechanics
• Selected functional components are unit tested
• Navigation throughout the architecture is tested
• The WebApp is implemented in a variety of different environmental configurations
and is tested for compatibility with each configuration
• Security tests are conducted in an attempt to exploit vulnerabilities in the WebApp or
within its environment
• Performance tests are conducted
• The WebApp is tested by a controlled and monitored population of end users; the
results of their interaction with the system are evaluated for content and navigation
errors, usability concerns, compatibility concerns, and WebApp security, reliability,
and performance.
Dr Ganesh Neelakanta Iyer 19
Test Planning
• A WebApp test plan identifies
– the task set 2 to be applied as testing commences,
– the work products to be produced as each testing task is
executed, and
– the manner in which the results of testing are evaluated,
recorded, and reused when regression testing is conducted
Dr Ganesh Neelakanta Iyer 20
The testing Process
Dr Ganesh Neelakanta Iyer 21
Content Testing
• Content testing has three important objectives:
– to uncover syntactic errors (e.g., typos, grammar mistakes) in
text-based documents, graphical representations, and other
media;
– to uncover semantic errors (i.e., errors in the accuracy or
completeness of information) in any content object presented
as navigation occurs, and
– to find errors in the organization or structure of content that is
presented to the end user
Dr Ganesh Neelakanta Iyer 22
Content Testing
Database Testing
• WebApps interface with sophisticated
database management systems and build
dynamic content objects that are created
in real time using the data acquired from a
database
• For example, a financial services WebApp
can produce complex text-based, tabular,
and graphical information about a specific
equity (e.g., a stock or mutual fund)
• The composite content object that
presents this information is created
dynamically after the user has made a
request for information about a specific
equity
Dr Ganesh Neelakanta Iyer 23
Content Testing
Database Testing
1. an equities database is queried,
2. relevant data are extracted from
the database,
3. the extracted data must be
organized as a content object, and
4. this content object (representing
customized information requested
by an end user) is transmitted to
the client environment for display
Dr Ganesh Neelakanta Iyer 24
Content Testing
Database Testing – Complicating factors
• The original client-side request for information is rarely
presented in the form [e.g. SQL] that can be input to a
database management system (DBMS)
– Therefore, tests should be designed to uncover errors made in
translating the user’s request into a form that can be processed by
these DBMS
• The database may be remote to the server that houses the
WebApp
– Therefore, tests that uncover errors in communication between the
WebApp and the remote database must be developed
Dr Ganesh Neelakanta Iyer 25
Content Testing
Database Testing – Complicating factors
• Raw data acquired from the database must be transmitted to
the WebApp server and properly formatted for subsequent
transmittal to the client
– Therefore, tests that demonstrate the validity of the raw data
received by the WebApp server must be developed, and additional
tests that demonstrate the validity of the transformations applied to
the raw data to create valid content objects must also be created
• The dynamic content object(s) must be transmitted to the
client in a form that can be displayed to the end user
– Therefore, a series of tests must be designed to (1) uncover errors
in the content object format and (2) test compatibility with different
client environment configurations
Dr Ganesh Neelakanta Iyer 26
User Interface Testing
• Verification and validation of a WebApp user interface occurs at three
distinct points
• During requirements analysis, the interface model is reviewed to ensure
that it conforms to stakeholder requirements and to other elements of the
requirements model
• During design the interface design model is reviewed to ensure that
generic quality criteria established for all user interfaces have been
achieved and that application-specific interface design issues have been
properly addressed
• During testing, the focus shifts to the execution of application-specific
aspects of user interaction as they are manifested by interface syntax and
semantics. In addition, testing provides a final assessment of usability
Dr Ganesh Neelakanta Iyer 27
Interface Testing Strategy
• Interface testing exercises interaction mechanisms and
validates aesthetic aspects of the user interface
• The overall strategy for interface testing is to
– Uncover errors related to specific interface mechanisms (e.g.,
errors in the proper execution of a menu link or the way data
are entered in a form) and
– Uncover errors in the way the interface implements the
semantics of navigation, WebApp functionality, or content
display
Dr Ganesh Neelakanta Iyer 28
Testing Interface Mechanisms
Client side
Links Forms Dynamic HTML
scripting
Application
Streaming specific
Pop up windows Cookies
content interface
mechanisms
Dr Ganesh Neelakanta Iyer 29
Usability Testing
• Usability testing can occur at a variety of different levels of
abstraction:
– the usability of a specific interface mechanism (e.g., a form)
can be assessed,
– the usability of a complete Web page (encompassing interface
mechanisms, data objects, and related functions) can be
evaluated, or
– the usability of the complete WebApp can be considered
Dr Ganesh Neelakanta Iyer 30
Usability Testing
Usability categories
Interactivity Layout Readability Aesthetics
Display
Time sensitivity Personalization Accessibility
characteristics
Dr Ganesh Neelakanta Iyer 31
Usability Testing
Qualitative assessment of usability
• Figure illustrates possible
set of assessment
“grades” that can be
selected by users
• These grades are applied
to each feature
individually, to a complete
Web page, or to the
WebApp as a whole
Dr Ganesh Neelakanta Iyer 32
Compatibility Tests
• Different computers, display devices, operating systems, browsers, and
network connection speeds can have a significant influence on WebApp
operation
• Each computing configuration can result in differences in client-side
processing speeds, display resolution, and connection speeds
• Operating system vagaries may cause WebApp processing issues
• Different browsers sometimes produce slightly different results, regardless
of the degree of HTML standardization within the WebApp
• Required plug-ins may or may not be readily available for a particular
configuration
• Compatibility testing strives to uncover these problems before the
WebApp goes online
Dr Ganesh Neelakanta Iyer 33
Compatibility Tests
• The first step in compatibility testing is to define a set of “commonly
encountered” client-side computing configurations and their variants
• In essence, a tree structure is created, identifying each computing
platform, typical display devices, the operating systems supported
on the platform, the browsers available, likely Internet connection
speeds, and similar information
• Next, a series of compatibility validation tests are derived, often
adapted from existing interface tests, navigation tests, performance
tests, and security tests
• The intent of these tests is to uncover errors or execution problems
that can be traced to configuration differences
Dr Ganesh Neelakanta Iyer 34
Discussion
Compatibility is an important quality dimension. What must
be tested to ensure that compatibility exists for a WebApp?
Dr Ganesh Neelakanta Iyer 35
25.7
CONFIGURATION TESTING
Dr Ganesh Neelakanta Iyer 36
Configuration Testing
• Configuration variability and instability are important factors that make
WebApp testing a challenge
• Hardware, OS, browsers, storage capacity etc are difficult to predict for
each user
• The result can be a client-side environment that is prone to errors that are
both subtle and significant
• One user’s impression of the WebApp and the manner in which she
interacts with it can differ significantly from another user’s experience, if
both users are not working within the same client-side configuration
• The job of configuration testing is not to exercise every possible client-side
configuration
• Rather, it is to test a set of probable client-side and server-side
configurations to ensure that the user experience will be the same on all
of them and to isolate errors that may be specific to a particular
configuration Dr Ganesh Neelakanta Iyer 37
Discussion
Which errors tend to be more serious—client-side errors or
server-side errors? Why?
Dr Ganesh Neelakanta Iyer 38
25.8
SECURITY TESTING
Dr Ganesh Neelakanta Iyer 39
Security Testing
• WebApp security is a complex subject that must be fully
understood before effective security testing can be
accomplished
• WebApps and the client-side and server-side environments in
which they are housed represent an attractive target for
external hackers, disgruntled employees, dishonest
competitors, and anyone else who wishes to steal sensitive
information, maliciously modify content, degrade
performance, disable functionality, or embarrass a person,
organization, or business
Dr Ganesh Neelakanta Iyer 40
Security Testing
• Security tests are designed to probe vulnerabilities of the client-side
environment, the network communications that occur as data are passed
from client to server and back again, and the server-side environment
• Each of these domains can be attacked, and it is the job of the security
tester to uncover weaknesses that can be exploited by those with the
intent to do so
• On the client side, vulnerabilities can often be traced to preexisting bugs
in browsers, e-mail programs, or communication software
• On the server side, vulnerabilities include denial-of-service attacks and
malicious scripts that can be passed along to the client side or used to
disable server operations
• In addition, server-side databases can be accessed without authorization
(data theft)
Dr Ganesh Neelakanta Iyer 41
OWASP Top 10 attacks
Not from Text book
But extremely useful to understand
[Link]
Dr Ganesh Neelakanta Iyer 42
OWASP Top 10 attacks
Three of them from security testing perspective
Injection Broken Access Control Cross-Site Scripting
• Injection flaws, such as SQL • Improperly configured or • Cross-site scripting (XSS)
injection, LDAP injection, missing restrictions on flaws give attackers the
and CRLF injection, occur authenticated users allow capability to inject client-side
when an attacker sends them to access unauthorized scripts into the application,
untrusted data to an functionality or data, such as for example, to redirect
interpreter that is executed accessing other users’ users to malicious websites.
as a command without accounts, viewing sensitive • Developer training
proper authorization. documents, and modifying complements security
• Application security data and access rights. testing to help programmers
testing can easily detect • Penetration testing is prevent cross-site scripting
injection flaws. Developers essential for detecting non- with best coding best
should use parameterized functional access controls; practices, such as encoding
queries when coding to other testing methods only data and input validation.
prevent injection flaws detect where access
controls are missing
Dr Ganesh Neelakanta Iyer 43
25.9
PERFORMANCE TESTING
Dr Ganesh Neelakanta Iyer 44
Performance Testing
• Performance testing is used to uncover performance problems that
can result from a lack of server-side resources, inappropriate
network bandwidth, inadequate database capabilities, faulty or weak
operating system capabilities, poorly designed WebApp functionality,
and other hardware or software issues that can lead to degraded
client-server performance
• The intent is twofold:
– to understand how the system responds as loading (i.e., number of users,
number of transactions, or overall data volume), and
– to collect metrics that will lead to design modifications to improve
performance
Dr Ganesh Neelakanta Iyer 45
Performance Testing
Questions to be answered
• Does the server response time degrade to a point where it is noticeable
and unacceptable?
• At what point (in terms of users, transactions, or data loading) does
performance become unacceptable?
• What system components are responsible for performance degradation?
• What is the average response time for users under a variety of loading
conditions?
• Does performance degradation have an impact on system security?
• Is WebApp reliability or accuracy affected as the load on the system
grows?
• What happens when loads that are greater than maximum server capacity
are applied?
• Does performance degradation have an impact on company revenues?
Dr Ganesh Neelakanta Iyer 46
Load Testing
• The intent of load testing is to determine how the WebApp
and its server-side environment will respond to various
loading conditions
• As testing proceeds, permutations to the following
variables define a set of test conditions:
– N, number of concurrent users
– T, number of online transactions per unit of time
– D, data load processed by the server per transaction
Dr Ganesh Neelakanta Iyer 47
Load Testing
• The intent of load testing is to determine how the WebApp
and its server-side environment will respond to various
loading conditions
• As testing proceeds, permutations to the following variables
define a set of test conditions:
– N, number of concurrent users
– T, number of online transactions per unit of time
– D, data load processed by the server per transaction
• You should examine these measures to determine whether a
precipitous decrease in performance can be traced to a
specific combination of N, T, and D
Dr Ganesh Neelakanta Iyer 48
Load Testing
• Load testing can also be used to assess recommended connection
speeds for users of the WebApp
• Overall throughput, P, is computed in the following manner:
P=N*T*D
• As an example, consider a popular sports news site. At a given moment,
20,000 concurrent users submit a request (a transaction, T ) once every 2
minutes on average. Each transaction requires the WebApp to download
a new article that averages 3K bytes in length. Therefore, throughput can
be calculated as:
P = [20,000 * 0.5 * 3Kb]/60 5 500 Kbytes/sec
= 4 megabits per second
• The network connection for the server would therefore have to support
this data rate and should be tested to ensure that it does
Dr Ganesh Neelakanta Iyer 49
Stress Testing
• Stress testing is a continuation of load testing, but in this instance
the variables, N, T, and D are forced to meet and then exceed
operational limits
• The intent of these tests is to answer each of the following
questions:
– Does the system degrade “gently,” or does the server shut down as
capacity is exceeded?
– Does server software generate “server not available” messages? More
generally, are users aware that they cannot reach the server?
– Does the server queue resource requests and empty the queue once
capacity demands diminish?
– Are transactions lost as capacity is exceeded?
– …..
Dr Ganesh Neelakanta Iyer 50
Summary
• The goal of WebApp testing is to exercise each of the
many dimensions of WebApp quality with the intent of
finding errors or uncovering issues that may lead to
quality failures
• Testing focuses on content, function, structure, usability,
navigability, performance, compatibility, interoperability,
capacity, and security
Dr Ganesh Neelakanta Iyer 51
Chapter 26, 26.1, 26.2, 26.3
TESTING MOBILE APPLICATIONS
Dr Ganesh Neelakanta Iyer 52
Mobile App Testing
• There are several important questions to ask when
creating a MobileApp testing strategy
– Do you have to build a fully functional prototype before you test
with users?
– Should you test with the user’s device or provide a device for
testing?
– What devices and user groups should you include in testing?
– What are the benefits/drawbacks of lab testing versus remote
testing?
Dr Ganesh Neelakanta Iyer 53
Testing Guidelines
• Understand the network and device landscape before testing to identify
bottlenecks
• Conduct tests in uncontrolled real-world test conditions
• Select the right automation test tool
• Use the Weighted Device Platform Matrix method to identify the most
critical hardware/platform combination to test
• Check the end-to-end functional flow in all possible platforms at least once
• Conduct performance testing, GUI testing, and compatibility testing using
actual devices
• Measure performance only in realistic conditions of wireless traffic and
user load
Dr Ganesh Neelakanta Iyer 54
MobileApp Testing - Checklist
Unit and
Conceptual Ux Stability Connectivity
System
Device
Performance Security Certification
Compatibility
Dr Ganesh Neelakanta Iyer 55
Building a test matrix
• A weighted device platform
matrix (WDPM) helps ensure
that test coverage includes each
combination of mobile device
and context variables
• The WDPM can also be used to
help prioritize the device/context
combinations so that the most
important are tested first
Dr Ganesh Neelakanta Iyer 56
Building a test matrix
1. list the important operating system
variants as the matrix column labels
2. list the targeted devices as the matrix
row labels
3. assign a ranking (e.g., 0 to 10) to
indicate the relative importance of each
operating system and each device, and
4. compute the product of each pair of
rankings and enter each product as the
cell entry in the matrix
Dr Ganesh Neelakanta Iyer 57
Stress Testing
• Stress testing for mobile apps attempts to find errors that will occur
under extreme
• operating conditions
• In addition, it provides a mechanism for determining whether the
MobileApp will degrade gracefully without compromising security
• Among the many actions that might create extreme conditions are:
– running several mobile apps on the same device,
– infecting system software with viruses or malware,
– attempting to take over a device and use it to spread spam,
– forcing the mobile app to process inordinately large numbers of
transactions, and
– storing inordinately large quantities of data on the device
Dr Ganesh Neelakanta Iyer 58
Testing in a Production Environment
• Many MobileApp developers advocate testing-in-the-wild , or testing
in the users’ native environments with the production release
versions of the MobileApp resources
• Some of the characteristics of in-the-wild testing include adverse and
unpredictable environments, outdated browsers and plug-ins, unique
hardware, and imperfect connectivity
• Creating test environments in-house is an expensive and error-prone
process
• Cloud-based testing can offer a standardized infrastructure and
preconfigured software images, freeing the MobileApp team from the
need to worry about finding servers or purchasing their own licenses
for software and testing tools
Dr Ganesh Neelakanta Iyer 59
CONSIDERING THE SPECTRUM OF
USER INTERACTION
Voice Input
Gesture
and
Testing
Recognition
Alerts and
Virtual Key
Extraordinary
Board Input
Conditions
Dr Ganesh Neelakanta Iyer 60
Summary
• The goal of MobileApp testing is to exercise each of the many
dimensions of MobileApp quality with the intent of finding
errors or uncovering issues that may lead to quality failures
• Testing focuses on quality elements such as content, function,
structure, usability, use of context, navigability, performance,
power management, compatibility, interoperability, capacity,
and security
• It incorporates reviews and usability assessments that occur
as the MobileApp is designed, and tests that are conducted
once the MobileApp has been implemented and deployed on
an actual device
Dr Ganesh Neelakanta Iyer 61