PROJECT

MANAGEMENT(PM)

Module 4:
Part 1: Project Management in the
Testing Phase

[Link]
 Syllabus Module 4:
Project Management in the Testing and
Maintenance Phase
• Project Management in the Testing phase: Introduction, what is testing? what
are the activities that makeup testing? test scheduling and types of tests, people
issue in testing, management structures for testing in global teams, metrics for
testing phase.
• Project Management in the Maintenance Phase: Introduction, Activities during
Maintenance Phase, management issues during Maintenance Phase,
Configuration management during Maintenance Phase, skill sets for people in
the maintenance phase.
2
Project Management in the
Testing Phase
 What is Testing?

❖ The eventual goal of a software project is to have zero defects in the end product that
goes out to customers.

❖ The defects in the final product normally creep in by means of defects from various
phases. For example,
• Defects can come from improper identification of requirements.
• Incorrect translation of requirements into design.
• Through erroneous coding.

❖ Mechanisms like design reviews can catch the defects in the initial phases.

4
The main goal of any software project is to deliver a product to the customer that is
free of defects (errors/bugs).
But in reality, defects creep in during different stages of software development:
[Link] Phase – If requirements are not properly understood or documented,
the software may not solve the customer’s actual problem.
Example: Customer asks for "monthly reports" but the requirement is wrongly noted as
"weekly reports."
[Link] Phase – Even if requirements are correct, they can be wrongly converted
into design.
Example: Requirement says “fast response time,” but the design chosen makes the
system slow.

5
Coding Phase – Mistakes in programming can also introduce defects.
Example: Wrong formulas, missing conditions, or syntax errors.

6
 What is Testing?

Testing refers to activities that are carried out to ensure that the final
software product meets the requirements that the product is intended
to satisfy.
Attributes of Testing are:
1. Testing is done on a product that has already been built.

2. Testing is done to detect the presence of defects in the product, not to prove that the product
is defect free.

7
 Attributes of Testing are:

3. Tests are divided into physical units called the test scripts (or test cases).

• Tests are written as scripts (test cases).

• Each script defines an expected result and produces an actual result when
executed.

• If both match, the test passes; if not, it fails.

8
Example
•Test Script: Enter valid username and password into the
login page.
•Expected Result: User should be redirected to the
homepage.
•Actual Result: User is redirected to the homepage.
•Outcome: Pass (Expected = Actual).

9
 Attributes of Testing are:

4. When a test fails, both the product and the expected results are
reviewed to find the cause.

• If the product is wrong, it is fixed and re-tested.

• If the expected results were incorrect, they are corrected.

• In both cases, tests are re-run and updated to confirm success.

10
Test Case: Add two numbers (5 + 3).
•Expected Result: 9
•Actual Result: 8
•Outcome: Fail

Analysis:
•If the program logic is wrong → fix the code, re-run → now result is 9 (Pass).
•If the expected result was wrongly set as 9 but should actually be 8 → update
expected result → test becomes Pass.
Either the product is corrected, or the expected result is updated, and the test is re-
run until it succeeds.

11
12
 ACTIVITIES THAT MAKE UP TESTING

Test Test Test Test Test

Test Design
Specification Development Registration Execution Maintenance

13
 ACTIVITIES THAT MAKE UP TESTING
Test Specification

➢ In this phase, what needs to be tested is finalized and documented.

➢ This can also be viewed as "requirements specifications" for the testing activity.

➢ Some of the questions that get answered during this phase are:
1. Which hardware configurations should be used for testing the product?
2. Are there any cut-off configurations under which the product must be tested?
3. Which software environments should the product be tested under?
4. What are the most common scenarios that need to be tested?
5. What are the criteria for test completion?

14
 ACTIVITIES THAT MAKE UP TESTING
Test Specification

1. Which hardware configurations should be used for testing the product?

❖ The hardware platforms required for testing are identified and documented.

❖ This helps estimate the cost of hardware and the time needed for testing.

❖ More platforms mean higher effort, cost, and time for both test development and
execution.

15
 ACTIVITIES THAT MAKE UP TESTING
Test Specification

2. Are there any cut-off configurations under which the product

must be tested?
❖ Some minimum hardware configurations (e.g., 256 MB RAM) are required for the
product to run.

❖ These should be identified during requirements and revisited during test design to
ensure proper testing on them.

16
 ACTIVITIES THAT MAKE UP TESTING
Test Specification

3. Which software environments should the product be tested

under?
❖ The product must be tested in different software environments (OS, compilers,
databases, etc.)

❖ since customer environments may differ from development. Documenting these

environments ensures accurate testing and faster problem resolution.

17
 ACTIVITIES THAT MAKE UP TESTING
Test Specification

4. What are the most common scenarios that need to be tested?

❖ It’s not possible to test every possible user scenario.

❖ Instead, a careful selection of key scenarios is made,
documented, and finalized after discussions with typical users.

18
 ACTIVITIES THAT MAKE UP TESTING
Test Specification

5. What are the criteria for test completion?

❖ During the test specification stage, the criteria for test completion should
also be documented.

❖ Test completion criteria (e.g., 95% code coverage, 80% routine coverage, or market
deadlines) should be defined during test specification.

❖ These criteria help plan and estimate testing resources.

19
 ACTIVITIES THAT MAKE UP TESTING
Test Design

• In the test specification stage, we identify what to test at a high level

(scenarios).
In the test design stage, we go into details of how to test:
• Define test cases: Break down each scenario into detailed steps with
expected outputs.
• Decide execution method: Choose whether tests will be automated
(using tools like Selenium) or done manually.
• Set structure and standards: Use consistent formats, naming
conventions, and templates for test case writing or automated scripts.

20
 ACTIVITIES THAT MAKE UP TESTING
Test Design

• In the test specification stage, we only decide what to test (high level).
In the test design stage, we add details like exact test cases and expected
results.

Example
• High-level scenario (specification): “System should support a maximum of 100
concurrent users.”
• Test design details:
• Test Case 1: Try with 50 users → Expected output: System works smoothly.
• Test Case 2: Try with 100 users → Expected output: System still works, no crash.
• Test Case 3: Try with 120 users → Expected output: System rejects extra users with an
error message.

21
 ACTIVITIES THAT MAKE UP TESTING
Test Design

2) Are we planning to use a tool for test automation or are the tests going to be run
manually?
• Once the tests are developed, they have to be run multiple times, once each time a new version
of the product is built.

• This makes test execution repetitive and labor-intensive.

• Using a test automation tool eases the problem. Automation tools provide mechanisms to
capture or "record" tests and the outputs, and then play the tests to verify that the produced
output and expected outputs match.

• Decision to use a test automation tool has to be taken during the test design stage itself as it has
ramifications on the cost, effort, and the technology to be used for test development.
22
 ACTIVITIES THAT MAKE UP TESTING
Test Design

3. What kind of structure or standards are to be used for development of tests?

• Test automation tools come with their set of recommended standards and structures.

➢ Coding standards → Define how automated test scripts are [Link]:

Always use meaningful names like Login_TC_01 instead of Test1.

➢ Documentation standards → Record what each test does, its purpose, and any
updates. Example: A test case document explains “Login test checks valid/invalid
credentials.”

➢ Structure standards → Organize files in a proper directory [Link]: Store

tests in folders like /login/, /checkout/, /payment/
23
 ACTIVITIES THAT MAKE UP TESTING
Test Development

❖ This step is similar to the coding part of software development.

❖ During this activity, tests are actually written.
❖ Specific activities differ depending on whether an automated tool is
to be used or whether testing is to be done manually.

24
 ACTIVITIES THAT MAKE UP TESTING

Test Automation
❖ One way to lessen the problem associated with manual testing is
the use of Test Automation Tools.

➢ Automation testing is a type of testing in which we take the help

of tools (automation) to perform the testing. It is faster than
manual testing because it is done with some automation tools.
There is no chance of any human errors.

25
 ACTIVITIES THAT MAKE UP TESTING
Test Automation
❖ Such tools provide mechanisms to
• Capture or record the user inputs.
• Invoke the right parts of the product (or the environment) in an automated
way, based on the user inputs.
• Specify responses the product should produce.
• Compare actual results and expected results.
❖ Automation requires manual effort to create initial testing scripts.
❖ Test automation tool can run the tests automatically with minimal human
intervention.
❖ Allows the execution of repetitive tasks and regression tests.
26
 ACTIVITIES THAT MAKE UP TESTING
Test Automation
When to Perform Automation Testing?
➢ When we need to run repetitive tasks: Automated tests are the best option in
scenarios where there is a requirement to run repetitive tests.
• For example, case in which regression tests must be executed periodically to make sure
that the newly added code does not disrupt the existing functionality of the software.

➢ When human resources are scarce: Automated tests are viable and the best
option to get tests executed within deadlines when there are only a limited
number of dedicated testers.

27
 ACTIVITIES THAT MAKE UP TESTING
Test Automation
Benefits of Automation Testing
➢ Finds more bugs: Automation testing helps to find more bugs and defects in the
software.
➢ Reduce time for regression tests: Automated tests are suitable for regression tests as
the tests can be executed in a repetitive manner periodically.
➢ The process can be recorded: This is one of the benefits of using automation tests as
these tests can be recorded and thus allows to reuse of the tests.
➢ No fatigue: As automation, tests are executed using software tools so there is no
fatigue or tiring factor as in manual testing.
➢ Increased test coverage: Automation tests help to increase the test coverage as using
the tool for testing helps to make sure that not even the smallest unit is left for testing.

28
 ACTIVITIES THAT MAKE UP TESTING
Test Automation

Challenges in Automation Testing :

1. Complexity of test automation increases with an increase in user
interaction and also with the increase in "GUI-ness" of the application.
• Capturing user inputs and expected screen outputs becomes tricky.
• Input/ output undergoes minor changes from one release to another, hence
verification of validity of observed (actual) outputs becomes difficult.
• Test automation tools can overcome this problem by capturing inputs and
outputs from logical screen instead of physical screen.

29
 ACTIVITIES THAT MAKE UP TESTING
Test Automation

Challenges in Automation Testing :

2. Complexity of test automation increases with an increase in the number of platforms on
which the product is supposed to run.
• When a product runs on many platforms, test automation becomes more complex.
• Even if the logic of the app is the same (e.g., login works the same way), the GUI look-and-
feel (buttons, layouts, controls) changes across platforms.
• Because of this, you can’t always reuse the same automated expected results everywhere
(what passes on Windows may fail on Android just due to UI differences).
• To solve this, teams use a generic testing framework that abstracts platform differences and
allows the same test logic to be reused.

30
 ACTIVITIES THAT MAKE UP TESTING
Test Automation

Challenges in Automation Testing :

• Many tests have some environment dependency in capturing expected Many tests
depend on the environment (system date, time, machine ID, session ID, terminal ID,
etc.).
• If these values are included in the expected results, they will always differ from run to
run.
• A naïve test comparison would flag these differences as errors, even though the
application is working correctly.
• To avoid this, the test comparison tool/framework should be intelligent enough to
ignore environment-dependent fields while still checking the important [Link].
31
 ACTIVITIES THAT MAKE UP TESTING
Test Automation

Test writing comprises of the following steps:

1. Training the test engineers on the automated testing tools to be used.
• Most testing tools have special languages in which the tests have to be
represented.
• Each tool has specific way of capturing inputs and outputs of a test and validating
the results.

2. Coding the tests in the language that the tool would understand, that would result
in test scripts.

32
 ACTIVITIES THAT MAKE UP TESTING
Test Automation

Test writing comprises of the following steps:

3. Running the test under reference conditions. This step ensures that -
• The test runs.
• The documented expected output is consistent with what was originally expected.
4. Capturing the inputs and outputs in this run.

• Ensures that this matches with the standard expected behavior for future runs.

5. Registering (baselining) the test script/the input output capture into the
Configuration Management System.

33
 ACTIVITIES THAT MAKE UP TESTING

Test Automation

❖ At this stage, the test scripts are available and the standard expected
inputs and outputs are calibrated to the actual expected
inputs/outputs.

34
 ACTIVITIES THAT MAKE UP TESTING
Test Automation
Limitations of Automation Testing
➢ Difficult to inspect visual elements: It is difficult to get insight into the visual
elements like color, font size, font type, button sizes, etc. as there is no human
intervention.

➢ High cost: Automation tests have a high cost of implementation as tools are
required for testing, thus adding the cost to the project budget.

➢ Test maintenance is costly: In automation tests, test maintenance is costly.

35
 ACTIVITIES THAT MAKE UP TESTING
Test Automation

Limitations of Automation Testing

➢ Not false proof: Automation tests also have some limitations and mistakes in

automated tests can lead to errors and omissions.

➢ Trained employees required: For conducting automated tests, trained

employees with knowledge of programming languages and testing knowledge

are required.

36
 ACTIVITIES THAT MAKE UP TESTING
Manual Testing

Manual testing is a type of testing in which we do not use any tools or

automation to perform the testing.
❖ Testers make test cases for the code, tests the software and give the final
report about that software.
❖ Manual testing is time-consuming testing because humans do it and there is a
chance of human errors.
❖ Manual testing is conducted to discover bugs in the developed software
application.
37
 ACTIVITIES THAT MAKE UP TESTING
Manual Testing

❖ The tester checks all the essential features of the application.

❖ The tester executes test cases and generates test reports without any
help from the automation tools.

❖ It is conducted by the experienced tester to accomplish the testing

process.

38
 ACTIVITIES THAT MAKE UP TESTING
Manual Testing

❖ In manual testing, test writing refers to writing detailed instructions on

what to test and how to test it.

❖ Test development in this case requires very clear and unambiguous

documentation.

❖ An example of how to document a manual test case is given in the next

slide. It covers how to carry out a deadlock detection test properly while
two users access a table in an Inventory application.
39
40
41
 ACTIVITIES THAT MAKE UP TESTING
Manual Testing

Some salient points that should be kept in mind while documenting tests
for manual execution are:
1. Do not assume the tester will have an in-depth knowledge of the application.
• The tester may not know the details of the application he is testing.
• For example, the statement, "check if the item has reached its re-order quantity
level" may be very obvious to the developer of an Inventory application but may
not make sense to someone who is new to the application domain and is testing
the application.

42
 ACTIVITIES THAT MAKE UP TESTING
Manual Testing

2. Do not assume that the tester is proficient with the details and syntax of the
underlying software

• e.g. if an application is built on top of a relational DBMS supporting SQL, it may not
be sufficient to tell the tester, "subtract quantity ordered from quantity on hand
for the product“.

• You would have to give an explicit SQL syntax which the tester would have to key
in.

43
 ACTIVITIES THAT MAKE UP TESTING
Manual Testing

3. To be specific and detail oriented

❖ When the instructions are documented, they should follow the specific syntax
and the punctuation that the tester should type in.

❖ When a GUI application is being tested, the instructions should go into details of :
• Where the mouse should be placed?
• What kind of cursor the tester would see?
• What he should do (mouse click, double click, tabbing across choices, etc.)?
• What would be the result of such an action?

44
 ACTIVITIES THAT MAKE UP TESTING
Manual Testing

4. Do not assume that all the environment is set up and is ready for testing
• Test may require some set-up to be done.

• E.g. Suppose the required table (PART) has to be created and populated with
necessary rows (ITEM 100 and 200).

• It is the test developer's responsibility to ensure that specific instructions for doing
the set-up are provided.

• It is a good idea to drop the existing environment and recreate it to remove any
undesirable side effects of data left over by the previous test runs.
45
 ACTIVITIES THAT MAKE UP TESTING
Manual Testing
5. Run the test on an environment distinct from the one in which the test
development was carried out before registering/baselining the test.

• When a test developer develops the tests, he is usually running in his test
environment which may not match the environment in which the tests are
supposed to run.

• There may be specific environment variables set up in the test developer's

environment which may not exist in the live test environment. Before baselining
the tests, it is important to validate them in a separate environment.

46
 ACTIVITIES THAT MAKE UP TESTING
Manual Testing
6. Do have someone else run the tests by just following your test documentation

• The person running the tests routinely may not have access to the test developer while
running the tests. Hence it is useful to have someone else test out the test
documentation.

7. Do clean up after your test is over

• Eventually, all the tests have to be run one after the other to test out the complete
product functionality.

• Every test should be followed by clean up i.e. drop the unnecessary data, unset any
environment variables etc.
47
ACTIVITIES THAT MAKE UP TESTING
Manual Testing

48
 Manual Testing And Automated Testing

Conclusion
➢ Both manual testing and automated testing play crucial roles in ensuring
software quality.

➢ While manual testing excels in exploratory testing and usability testing,

automated testing shines in regression, performance, and load testing.

➢ The choice between them depends on factors such as project requirements,

timeline, budget, and the nature of the application being tested.

49
 ACTIVITIES THAT MAKE UP TESTING
Test Registration

1. The tests have to be run multiple times for each build of the product.

2. Whenever a product it tested, it is necessary that the correct versions of the tests be
run.

3. Hence tests need to go through proper configuration management.

4. “Tests", refers to test specification, test scripts, test documentation, test inputs that
are captured and the expected test outputs that are produced.

50
1.
Example:
Imagine you’re developing a mobile banking app.
•Build 1: You add the “Login” feature. You test it.
•Build 2: You add the “Money Transfer” feature. You must test both Login and Money Transfer again.
•Build 3: You add “Check Balance”. You again test all three

2. Example:
Build 1: Login button is labeled “Submit”.
Build 2: It’s renamed to “Sign In”.
→ The old test script looking for the “Submit” button will fail.
→ You must update the test script and run the new version for Build 2.
3. Example:
In your testing repository, you might have:
/tests/v1.0/login_test.py
/tests/v2.0/login_test.py
You know Build v1.0 uses the first script, and Build v2.0 uses the second one.

51
 •Test specification: Describes what should be tested and how.
•Test scripts: The code or automation steps to perform the test.
•Test documentation: Explains the test plan, steps, and results.
•Test inputs: Data used for testing.
•Expected outputs: What result should appear if the software works correctly.

Example (for Login feature):

Component Example
Test specification Verify user login with correct and incorrect passwords
Test script Selenium script to enter username/password and click login
Test documentation Test case ID TC001 – “Login Functionality”
Test input username: user123, password: pass@123
Expected output User successfully logs into dashboard

52
 ACTIVITIES THAT MAKE UP TESTING
Test Execution
❖ Test Execution refers to the actual execution of tests against a given build of the product. This
step is repeated several times, usually every time the product is built.

❖ Objectives of test execution are to detect any old defects that have re-surfaced and to report
any new defects.

❖ If automated tools are used for executing the tests, such test execution can be piggybacked
on the activity of building the product (CI/CD).

❖ If the test execution is manual, then it can be triggered by a formal communication to the
testing team upon successful completion of a given version of the product.

53
Test Execution refers to the actual execution of tests against a given build of the
product.”
This is the stage where tests are actually run on the software build (version) to check
if it works correctly.
Each “build” of the product is a version that includes new code or fixes.
The goal is to see whether the software behaves as expected and whether any bugs
appear.
Example:
Imagine your team is developing an online shopping app.
After the developers finish a new build (say Build 3.2), the testing team:
runs all the test cases (like login, add to cart, payment, etc.)
checks if everything still works properly.
That activity is called Test Execution.
54
•Every time a new build is created (for bug fixes, new features, or improvements),
tests must be run again to ensure:
• old bugs haven’t come back, and
• new bugs haven’t appeared.

Example:
•Build 3.0 – Tests run → all passed.
•Build 3.1 – New feature added → Tests run again.
•Build 3.2 – Bug fix done → Tests run again.
→ This repetition ensures product stability.

55
“Objectives of test execution are to detect any old defects that have re-surfaced
and to report any new defects.”
•Sometimes, when developers fix something, an old bug might reappear (called
regression).
•Also, new bugs may be introduced due to new code changes.
•So the testing team’s aim is to find both.

Example:
•Earlier bug: “Payment page crashes” — fixed in Build 3.0.
•But in Build 3.2, after UI changes, the same crash happens again.
→ This is an old defect that re-surfaced.
•If a new issue occurs (e.g., product image not loading), it’s a new defect.
56
Automated test execution
“If automated tools are used for executing the tests, such test execution can be piggybacked on the
activity of building the product (CI/CD).”
•Automated testing tools (like Selenium, Jenkins, or JUnit) can run tests automatically each time
developers build the product.
•This is part of CI/CD (Continuous Integration / Continuous Deployment) pipelines.
•It means as soon as the new build is ready, tests start running automatically — no human trigger
needed.
Example:
In Jenkins pipeline:
Step 1: Build application
Step 2: Run automated test suite
Step 3: Generate test report
If the build passes all tests, it automatically moves to deployment.

57
Manual test execution
“If the test execution is manual, then it can be triggered by a formal
communication to the testing team upon successful completion of a given version
of the product.”
•When tests are not automated, testers run them manually.
•The development team informs the testing team — usually through an email,
ticket, or project management tool — that the build is ready for testing.
•Only then do testers begin manual test execution.
Example:
•Developer sends a message:
•“Build 3.2 is successfully deployed to the test server. Please start testing.”
•The QA team receives it and begins manual testing following test cases.
58
 CI/CD

Typical Flow in CI/CD

❖ Code Commit → Developer pushes code to the repository.
❖ Build Triggered → CI tool (like Jenkins, GitHub Actions, or Azure DevOps) starts
building the product.
❖ Automated Tests Run → Unit tests, integration tests, and sometimes UI tests are
executed.
❖ Results Evaluated → If tests pass, the build proceeds to deployment or further
stages. If not, it's flagged for review.
59
❑ Code Commit
•The developer writes or updates code and then pushes it to the shared repository (like GitHub or
GitLab).
•This saves the latest version of the code so others can access it.
Example: A developer commits new login feature code to GitHub.

❑ Build Triggered
•Once the code is pushed, a CI (Continuous Integration) tool such as Jenkins, GitHub Actions, or
Azure DevOps automatically starts the build process.
•The build means compiling the code, packaging files, and preparing it for testing.
Example: Jenkins detects the new commit and starts building the latest version of the app.

60
3️⃣ Automated Tests Run
•After building, automated tests (like unit tests, integration tests, or UI tests) run
automatically.
•These check if the new code works correctly and doesn’t break existing features.
Example: The system runs tests to ensure the login feature works and doesn’t affect
the signup process.

4️⃣ Results Evaluated

•The CI tool analyzes test results:
• If all tests pass → the build moves to deployment or the next phase.
• If any test fails → the build is flagged for review, and developers fix the issues.
Example: If all login tests pass, the app is deployed. If one test fails, the build is stopped, and the
developer gets a report.

61
 ACTIVITIES THAT MAKE UP TESTING
Test Maintenance

Why is Test Maintenance required?

❖ Over time, a product evolves with its new versions. New features are introduced.

❖ Some features keep getting enhanced while some other become obsolete.

❖ Also, testing technology itself can evolve with more automation and tools.

❖ Because of all these changes, the tests and/or the expected results need to be
maintained.

62
 ACTIVITIES THAT MAKE UP TESTING
Test Maintenance

❖ The existing tests may need to be changed to reflect changes in product behavior.
Example:
Earlier, when a user entered the wrong password, the app showed:
• “Invalid Password.”

After an update, the same app now shows:

• “Incorrect password. Please try again.”

63
 ACTIVITIES THAT MAKE UP TESTING
Test Maintenance
❖ Existing tests may also need to be scrapped as some of the features may be desupported in newer
releases.
• Example
Suppose you have an online shopping app.
• Old version: Had a feature called “Pay by Cheque”.
• New version: The company removes that feature and now supports only “Pay by Card” or
“UPI”.
Old test case:
• “Verify that payment through cheque works correctly.”
This test is now irrelevant, because the feature no longer exists in the app.
→ So, the test should be scrapped (deleted).
64
 ACTIVITIES THAT MAKE UP TESTING
Test Maintenance

❖ Any new product version would have new features. It is important that as new features
get added to the product, new tests for those features also get added to the test suites.

Example
Old version of app: Only allows users to view products.
New version: Adds a feature to add products to wishlist.
New test case needed:
“Verify that a user can successfully add a product to the wishlist.”

65
 Similarities between Code Development
& Test Development
❖ Requirements: Product requirements get finalized first; similarly, what needs

to be tested gets finalized first.

❖ Development: Test development is a means of translating the test design

into executable tests, just as Program development is a means of

representing the design/ algorithms in a programming language, eventually

complied into an executable code.

66
 Similarities between Code Development
& Test Development
❖ Configuration Management: Test code, just like the program code, needs to go

through proper configuration management.

❖ Debugging: Tests get debugged just as code gets debugged. Debugging a test involves

ensuring that the tests are coded and executed in the right way to produce the right

expected results.

❖ Maintenance: Tests need to be maintained and kept current with changing

requirements, just like code is kept current with changes in requirements/design.

67
 Is Testing a separate and Distinct Phase of
Product Development?
❖ The intent of any product development activity is to avoid product defects and
in case product defects do creep in, to minimize the delay between the point of
injection and the point of correction of the defect.
❖ Testing is a means of uncovering the defects in the final product.
❖ It is important that the defects in the products be detected reasonably close to
the point of injection and not at the end of the development phase.
❖ Testing is an activity that should run parallel with development so that defects
can be detected and corrected early in the cycle.
68
Example
•While developing a shopping app, a developer accidentally writes
code that allows users to checkout without entering a shipping address.
•This is a defect injected during coding.
•During testing, the QA team finds this issue quickly and reports it.
•The developer fixes it immediately.

❖ Early detection prevents this defect from affecting users or delaying

the product release.

69
 Test Scheduling and Types of Tests

❖ The kind of testing that needs to be done depends upon the level or position that a
product is in.

❖ There are different types of tests that are done at different points of product
development life cycle.

❖ Each of these test types may require different skill sets, different profiles, and the
responsibility of the test activities would rest with different people.

70
•A developer accidentally creates a defect in the
shopping app (checkout works without a shipping
address).
•This defect is injected during coding.
•The QA team finds it during testing and reports it.
•The developer fixes it immediately.

71
 Test Scheduling and Types of Tests

Acceptance White box

testing testing

Different types of
Regression Black box
testing testing
testing done during
Product Life Cycle Installation Integration
testing testing

System
testing

72
 White Box Testing

❖ This testing method examines the internal structure, design, and

implementation of a software system.

❖ It requires knowledge of the code and is often performed by developers

to verify the logic, control flow, and data flow within the application.

73
 White Box Testing
Why and how is this type of testing is done?
❖ A software product accomplishes different functionality or options by
executing different paths through the code.
❖ In white box testing, the program is run with various test data that
exercise as many of the various paths in the source program as possible,
thereby testing as much of the functionality as possible.
❖ White box testing requires knowledge of source code of the product.
Either the developer (or someone elşe who can understand the source
code) looks at it and identifies the various paths through the program
code and designs test cases to cover as many of these paths as possible.

74
 White Box Testing
Why and how is this type of testing is done?

Some means of widening the coverage of different paths through the program are:
❖ Tests that exercise the multiple conditions in a Boolean expression in various
combinations.
❖ Tests that exercise the THEN and ELSE parts of conditional statements.
❖ Tests that are within the various CASE conditions of a multi-way decision statement like the
SWITCH statement in C.
❖ Tests that take a loop construct from 0 to the maximum value expected in the loop
variable.
❖ Tests that exercise various error conditions by giving erroneous data.
❖ Tests that check the combinations of correct and incorrect values for input parameters.
75
White Box Testing

76
 White Box Testing
What are the required skill sets?

•White box testing needs a deep understanding of the source

code and program logic.
•Test designers must know the programming language and how
the product works internally.
•They check things like:
•Allowed inputs
•Expected outputs
•Program behavior
•Since source code changes often, testers must update tests to
cover all new or modified code paths.
77
 White Box Testing
Who is responsible for this type of test?

❖ Design of white box tests are best done by the author of the program
himself as he is familiar with the programming language, program logic
and the various paths within a program.

❖ Also, as the intent is to detect any defects as soon as possible, the

developer should himself execute the white box tests before baselining
the code.

78
 White Box Testing
Common Challenges In White Box Testing

• 1️⃣ Developer-related challenges

• Developers often prefer writing new code over writing or running tests.
• They are under pressure to release features quickly, so testing may be
neglected.
• Human limitations: It’s hard to find defects in your own work because of
bias.
• Some may hesitate to admit defects exist in their code.
•

79
 White Box Testing
Common Challenges In White Box Testing

• 2️⃣ Technical challenges

• Combinatorial explosion: Programs can have a huge number
of paths; testing all is practically impossible.
• Time constraints mean only a subset of paths can be tested.

80
3️⃣ Testing perspective challenge
Developers focus on code paths, not real-world usage patterns.
They may miss common or critical paths that users frequently
encounter.
Tests should be designed from an external user perspective, focusing
on:
Common usage paths
Critical paths / potential show stoppers

81
 White Box Testing
Disadvantage of White Box Testing

❖ The primary disadvantage of the White box testing is that it is based on

the knowledge of the program code from the developer's perspective but
not on the product functionality from the user perspective.

82
 Black Box Testing

❖ The most commonly used paths or relative criticality of paths requires

looking at the product from external perspective and making sure that
test cases are designed for such paths.

❖ To bridge this important gap in white box testing we have the black box
testing.

83
 Black Box Testing

❖ Black box testing tests the functionality and external behavior of an

individual module or a feature.

❖ Also called behavioral testing.

84
 Black Box Testing

❖ Black-box testing is a type of software testing in which the tester is not

concerned with the software’s internal knowledge or implementation
details but rather focuses on validating the functionality based on the
provided specifications or requirements.

85
Black Box Testing

86
 Black Box Testing
Why and how is this type of testing
is done?

❖ Black box testing looks at the product as a black box which is

supposed to behave as per the specifications originally laid down and
attempts to identify, from an external perspective, any deviations
from the expected behavior of the product.

87
 Black Box Testing
Why and how is this type of testing is done?

There are multiple approaches possible to design black box tests. Presented below is one
approach that is usually quite effective:

1. From the requirements specifications, the product is broken down into its salient
features or logical modules.

❖ Test designer is allowed to look only at the external behavior of the product and not at the
implementation in the source files.

2. For each feature, tests are designed for functionality, usability, limits, error handling
and interfaces.
88
 Black Box Testing
Why and how is this type of testing is done?

Functionality testing:
❖ This type of tests exercises the basic functionality of the product from an external
perspective.

❖ Some typical functionality tests are:

• Tests to ensure that proper input data is accepted.
• Tests to ensure that the internal calculations and transformations result in appropriate
intermediate results.
• Tests to ensure that the external outputs match what is expected in terms of content and
layout.
89
 Black Box Testing
Why and how is this type of testing is done?

Functionality testing: Black Box Functional Testing Example in C

• Let’s say we have a function that calculates the grade based on a student’s score:
#include <stdio.h>
char getGrade(int score) {
if (score >= 90) return 'A';
else if (score >= 80) return 'B';
else if (score >= 70) return 'C';
else if (score >= 60) return 'D';
else return 'F’;
}

90
 Black Box Testing
Functionality Testing
Test Data for Black Box Testing
❖ We’ll design test cases based on functional requirements, not internal logic.

❖ Techniques like Equivalence Partitioning and Boundary Value Analysis are used.

1. Equivalence Partitioning

Divide input into valid and invalid partitions:

• Valid scores: 0–100

• Invalid scores: 100

91
 Black Box Testing
Functionality Testing
Test Data for Black Box Testing

92
 Black Box Testing
Functionality Testing

Test Data for Black Box Testing

2. Boundary Value Analysis

Divide input into valid and invalid partitions:

• Test values at the edges of partitions:

• Boundaries: 59, 60, 69, 70, 79, 80, 89, 90

93
 Black Box Testing
Functionality Testing
Test Data for Black Box Testing

94
 Black Box Testing
Why and how is this type of testing is done?

Usability testing:

Some examples of usability testing are:

❖ Does the user interface match the GUI standards of the platform?

❖ Is there consistency in the user interface across product(s)?

❖ Does the navigation match what is documented?

❖ Is there consistency in the reports - both in content and format?

95
 Black Box Testing
Why and how is this type of testing is done?

Limit testing: A product feature may have in-built hard limits e.g. maximum size of a
particular data item, maximum logical size of a file, maximum number of concurrent
users on the system, etc.

Limit testing refers to creating tests to verify:

❖ The system is able to handle the values within and including these limits properly.
❖ The system is able to handle the exceeding of such limits in a graceful manner,
• i.e. by displaying an error message rather than a system crash or some similar
catastrophic outcome.

96
 Black Box Testing
Why and how is this type of testing is done?

Error handling tests: It is important to validate product behavior not only for correct
data but also for incorrect data. The system should handle error conditions gracefully
and in a user friendly manner.

Some of the error checking tests that need to be done include:

❖ Testing data inputs of wrong type or format.
❖ Testing data inputs of wrong size.
❖ Testing inconsistent data combinations.
❖ Testing that the right message is displayed for the appropriate error condition.
❖ Testing that error messages are not given for non-error situations.
97
 Black Box Testing
Why and how is this type of testing is done?

Interfaces testing: Each module (feature) should talk to the other modules (features)
interfaces through well defined interfaces.

❖ Interface testing is done to ensure that proper interfaces are passed across modules.

• Interfaces can be external, (like files) or internal, (like internal data structures).

• Since all the modules may not be available at the same time for testing, it is important to
document clearly what the interfaces are so that such interface test data can be generated
manually and the tests carried out.

98
 Black Box Testing
Responsibilities and Skill Sets

❖ Black box testing requires an in-depth understanding of the

requirements and the external functionality of a feature.

❖ The people who design black box tests need not be totally familiar with
the programming language used during development.

99
 Black Box Testing
Responsibilities and Skill Sets

❖ The test designer should be able to visualize

➢ How the feature would be used by the users?

➢ What the commonly used functionality are?

➢ What would be the expected behavior under various scenarios?

❖ Hence the black box tests development team is usually different from the people
who actually develop the code.

100
 Black Box Testing
Common Challenges in Black Box Testing
1. Ambiguous or unclear expected behavior:
❖ The success of black box testing is dictated by how well the external behavior of the
product is understood.
❖ When the development cycles are short, the requirement or design specifications may
leave some loose ends in terms of the expected product behavior.
• E.g. All the possible usage scenarios may not be documented.
• So the tester may have his own view of "what is the correct behavior", which may be distinct
from what the product was originally intended to do.
❖ To overcome this - tighten the Requirement and Design to describe unambiguously the
expected behavior in various scenarios.
101
 Black Box Testing
Common Challenges in Black Box Testing

2. Black box testers viewed as adversaries to development:

❖ The black box testing group is kept separate from the development group.

❖ As the testing group is distinct and their job is to find defects in the products produced
by the developers, there is a possibility of the two groups viewing each other as
adversaries.

102
 Black Box Testing
Common Challenges in Black Box Testing

3. Relating common usage patterns and prioritizing:

❖ When the product usage is viewed from an external perspective,

• A number potential usage patterns are possible.
• Testing every single scenario is impractical.

❖ The challenge (as in white box testing) lies in visualizing the common usage patterns,
prioritizing them and ensuring that they are tested.

103
 Black Box Testing
Common Challenges in Black Box Testing
3. Relating common usage patterns and prioritizing:
❖ Why this Is Difficult?

❖ No access to source code: Unlike white box testing, we can't trace logic paths or
branches.

❖ High variability in user behavior: Different users may use the same feature in vastly
different ways.

❖ Resource constraints: Time, budget, and manpower limit how many test cases can
be executed.
104
 Black Box Testing
Common Challenges in Black Box Testing

3. Relating common usage patterns and prioritizing:

❖ The Solution: Strategic Prioritization

❖ To overcome this, testers must visualize usage patterns and prioritize based on
risk, frequency, and impact.

105
 Black Box Testing
Common Challenges in Black Box Testing

3. Relating common usage patterns and prioritizing:

The Solution: Strategic Prioritization
1. Usage Frequency
• Focus on features or workflows that are used most often.
• Example: In a billing system, generating invoices is more common than exporting
audit logs.
2. Business Criticality
• Test scenarios that are vital to business operations.
• Example: Payment processing must be flawless, even if rarely used.

106
 Black Box Testing
Common Challenges in Black Box Testing
3. Relating common usage patterns and prioritizing:
The Solution: Strategic Prioritization
3. Error-Prone Areas
• Prioritize areas with a history of bugs or complexity.
• Example: Dynamic form validation or multi-step transactions.
4. Boundary and Edge Cases
• Use techniques like Equivalence Partitioning and Boundary Value Analysis to reduce
test cases while maximizing coverage.
5. User Personas and Journeys

107
 Black Box Testing
Common Challenges in Black Box Testing

3. Relating common usage patterns and prioritizing:

❖ The Solution: Strategic Prioritization
❖ Example: Restaurant Management System: Restaurant manager might
log in, check inventory, and place a bulk order—test that flow.
Suppose the tester has to test a dashboard for restaurant staff.

108
 Black Box Testing
Example: Restaurant Management System
Tester might identify:

Prioritization Matrix
❖ Tester will focus his black box tests on the “Add new order” flow first, using real-world inputs
and expected outputs—without needing to know how the backend processes it.
109
 White Box vs. Black Box Testing
Aspect White Box Testing Black Box Testing
Internal logic, code structure, and External behaviour and functional
Focus
control flow correctness
Requires access to and understanding No knowledge of internal code
Tester Knowledge
of source code required
Based on code paths, branches, Based on requirements,
Test Design Basis
loops, and conditions specifications, and expected outputs
Statement coverage, branch Equivalence partitioning, boundary
Techniques Used
coverage, path testing value analysis, error guessing
Typically developers or technically Typically QA engineers or functional
Who Performs It
skilled testers testers
To verify internal operations and logic To validate that the system meets
Purpose
correctness user expectations and requirements
Unit tests, integration tests with UI testing, system testing, acceptance
Examples 110
code-level assertions testing
 White Box vs. Black Box Testing

Summary
❖ White box testing is like inspecting the engine of a car to ensure
every part works as designed.
❖ Black box testing is like driving the car to see if it performs as
expected, without knowing what’s under the hood.

111
 Integration Testing
Why and how is this type of testing is done?

❖ A product works as a whole by interaction between the various

modules/features.

❖ Integration testing is a level of software testing where individual software

modules, which have already undergone unit testing, are combined and tested
as a group.

❖ The primary goal of integration testing is to identify defects in the interfaces and
interactions between these integrated modules.
112
 Integration Testing
Why and how is this type of testing is done?

❖ In black box testing during the step of testing interfaces, all the interfaces may not be
available at the same time, interface data have to be generated manually for testing
purposes.

❖ When we come to Integration testing, the modules that are to be integrated are
available for testing. So, the manual test data is replaced by the data which is
generated automatically from the various modules.

113
 Integration Testing
Why and how is this type of testing is done?
Sequence in which the modules can be integrated and tested depends upon the
following factors:

❖ Availability of modules to be integrated:

• When a module becomes available for testing, it is expedient to test it with other modules
that are ready.
• This will be dictated by the availability of WBS units and milestones.

❖ Modules with maximum impact: Among the modules that are available, integrating
those with the maximum impact will detect the integration errors earlier.

114
 Integration Testing
Why and how is this type of testing is done?
Reveals Interface Issues:
❖ It aims to expose errors related to data flow, parameter passing, and function calls
between modules, as well as potential compatibility issues.
Methods of Integration:
❖ Various strategies exist for integrating modules, including
• "Big Bang" (all at once)
• "Top-Down" (from higher-level modules downwards)
• "Bottom-Up" (from lower-level modules upwards)
• "Sandwich" or "Mixed" (combining top-down and bottom-up)
115
 Integration Testing
Responsibilities and Skill Sets
Similarities with black box test development team:

❖ Both have good product knowledge from an external perspective test development team.
Both know the basic modules / features within the product.

Differences from black box test development team:

❖ More dependencies exist across modules so higher need to communicate across different
groups.
❖ Soft skills required is higher for integration test team.
❖ Different reporting structure as compared to black box team.

116
 Integration Testing
Common Challenges

1. Visualising integration tests:

❖ Successful integration testing calls for a sound knowledge of how the various parts
should fit together.

❖ This requires someone who understands the external product functionality as well
as the internal units of integration. To find this combination of skill sets is not easy.

117
 Integration Testing
Common Challenges

2. Schedule conflicts because of availability:

❖ Once a module reaches a certain stage of completion, it would require some other
modules to proceed.

❖ Because of scheduling conflicts, these other modules might not be available. Thus,
integration testing may come to a standstill.

❖ In order to avoid this situation, one may have to, at times, proceed further with
development and defer integration testing.

118
 Integration Testing
Common Challenges

3. Assigning responsibilities to problem areas:

❖ During white box or black box testing, when a problem/ anomaly is observed, there is
no controversy about assigning responsibility to fix the problem since the unit being
tested is self contained.

❖ In integration testing, we are taking modules developed by different groups and putting
them together for testing. When a problem comes up, it is sometimes very difficult to
pin-point where the problem lies and who should take the corrective action.

119
 Integration Testing
Common Challenges

4. Communication:
❖ In white box and black box testing, the testing is carried out pretty close to the point
of development as there are very few external dependencies.

❖ Integration testing depends on the availability of several modules. Hence, triggering

the start of integration testing and ensuring smooth progress and defect resolution is
dictated by very good and well-defined communication channels across the various
groups.

120
 Integration Testing
Common Challenges

4. Communication:
❖ Multiple Modules: Often developed by different teams (e.g., frontend, backend,
database).

❖ Timing Sensitivity: One module’s delay can block the entire integration test.

❖ Defect Resolution: Bugs may span across interfaces—requiring cross-team debugging.

❖ Environment Setup: Requires coordinated deployment of multiple components.

121
 Integration Testing
Common Challenges

4. Communication:
What This Demands?

❖ Clear ownership of modules and interfaces.

❖ Defined protocols for defect logging.

❖ Regular sync-ups across dev, QA, and ops teams.

❖ Version control discipline to avoid mismatched builds.

122
 Integration Testing
Common Challenges
4. Communication:
Example: Billing System Integration: Suppose a tester is testing a billing engine that pulls data from:
❖ A customer database
❖ A transaction processor
❖ A tax calculation module
If the tax module isn’t ready, the integration test fails—not because of the billing logic, but due to a
dependency.
Resolving this requires:
• Fast communication between module owners
• Shared understanding of interface contracts (input, output, protocols, error handling)
• A test coordinator who tracks readiness and escalates blockers

123
 System Testing

❖ Similarities with Integration testing: System testing is aimed at

exercising the product as a whole.

❖ Dissimilarities with Integration testing: The focus during system testing

is on stressing the system under extreme conditions and ensuring that
if there is any failure, it is well managed.

124
 System Testing
Aspects

1. Load testing: Load testing is a type of performance testing where a

system is subjected to expected or extreme levels of user activity to
evaluate how it behaves under stress. The goal is to ensure the system
can handle the load without crashing, slowing down, or producing
errors.

125
 System Testing
Aspects
1. Load testing:

Some examples of load tests are:

❖ Testing with a high number of concurrent users and ensuring that the system
does not break;

❖ Flooding the system with excessive input requests

❖ Trying to pump in more data than what the network bandwidth or other similar
physical limits can handle.
126
 System Testing
Aspects
2. Configuration testing: It is a type of non-functional testing that
validates how a software product performs across different hardware,
software, network, and system settings. It ensures compatibility,
stability, and performance under all supported configurations.

127
 System Testing
Aspects
2. Configuration testing: Every product should have the specifications of
the hardware configurations that it runs under. Configuration testing
is a means of running the software product under various
combinations of these configurations.

128
 System Testing
Aspects

2. Configuration testing:

❖ For example, if the product is certified to run with a minimum

memory of 16MB, then the tests must ensure that the product does
run in a machine with 16MB memory.

129
 System Testing
Aspects
2. Configuration testing:

Typical Configuration Variables

130
 System Testing
Aspects
3. Integrity testing: Regardless of the things that go wrong in the
environment, the product should not compromise the integrity of the
data it handles.

❖ For example, there should be adequate protection and recovery from

events like power failure, media crashes, etc.

❖ Integrity testing is aimed at testing features like recovery, authentication,

etc.
131
 System Testing
Aspects
3. Integrity testing: It ensures that a software system maintains the
accuracy, consistency, and reliability of its data—even in the face of
unexpected failures or malicious activity.

❖ It validates that the system can recover gracefully, prevent unauthorized

access, and preserve data fidelity across all operations.

132
 System Testing
Aspects
3. Integrity testing:

Core Objectives

❖ Data Preservation: No corruption, loss, or unauthorized modification.

❖ Recovery Assurance: System can restore to a consistent state after failure.

❖ Security Validation: Authentication, authorization, and audit trails are intact.

❖ Transactional Consistency: ACID properties are upheld, especially in databases.

133
 System Testing
Aspects: Integrity testing

Key Features to Test

134
 System Testing
Aspects: Integrity testing
Example Scenarios
1. Power Failure Simulation
• Kill power during a database write.
• Verify rollback or recovery to last consistent state.
2. Media Crash Recovery
• Simulate disk failure.
• Validate backup restoration and data integrity.

135
 System Testing
Aspects
4. Inter-operability testing:
❖ In today's componentized world, a product would be expected to inter-operate with other
products, whether from the same vendor or from different vendors.

❖ While doing Inter-operability testing, the approach should be to specify which versions of
which products should interoperate with each other and test such combinations.

❖ Inter-operability testing usually becomes increasingly complex as the product matures (as
there would be more versions to test inter-operability) and increasingly expensive too
because of the number of combinations possible.
136
 System Testing
Aspects
4. Inter-operability testing:
Why It’s Critical

❖ Modern systems are modular and distributed (think APIs, microservices, cloud
platforms).

❖ Products rarely operate in isolation—they must integrate with databases, browsers,

OSs, payment gateways, third-party services, etc.

❖ Failure in interoperability can lead to data loss, transaction failures, or security

breaches.
137
 System Testing
Aspects: Inter-operability testing

Key Elements of Interoperability Testing

138
 System Testing
Aspects: Inter-operability testing

Complexity Over Time

As the product matures:
❖ More integrations are added (e.g., new payment gateways, analytics tools).
❖ Older versions must still be supported (legacy systems).
❖ Combinations multiply exponentially, making test coverage harder and costlier.

139
 System Testing
Aspects: Inter-operability testing
Example: Billing System Interoperability
Let’s say a billing engine must work with:
• Database: PostgreSQL 12, 13, 14
• Browser: Chrome, Firefox, Edge
• Payment Gateway: Stripe v3, Razorpay v2
• OS: Windows 10, Ubuntu 20.04
We need to test combinations like:
• Stripe v3 + PostgreSQL 14 + Chrome on Windows 10
• Razorpay v2 + PostgreSQL 12 + Firefox on Ubuntu

140
 System Testing
Aspects: Inter-operability testing
Example: Billing System Interoperability
Each combination must be validated for:
❖ Data accuracy
❖ Transaction success
❖ UI rendering (It refers to how the user interface elements like buttons, forms, tables, etc. are visually
displayed and behave across different environments.)

❖ Security compliance

141
 System Testing
Responsibilities and Skill Sets

❖ The skill sets of system testing encompass and transcend the skill sets of integration
testing.

❖ The added requirement is that in tests like load testing, the person developing and
running the tests must be thoroughly familiar with the nuts and bolts of the underlying
system software.

❖ For example, load testing has a significant dependency on the hardware configuration
and system tuning. Such skill sets are fairly niche and difficult to come by.

142
 System Testing
Common Challenges

1. Defining realistic scenarios: Defining what constitutes realistic scenarios is a very

complex task.

❖ Each customer's system could be different so, what constitutes a good system test
for one customer may be inappropriate for another customer.

143
 System Testing
Common Challenges

2. Unique mix of skill sets needed:

❖ System testing requires a delicate combination of visualizing the 20,000-foot level
view of how an end product would be used along with the nuts and bolts of how
to exploit the best features of the underlying system.

❖ This is a fairly specialized skill set.

144
 System Testing
Common Challenges

3. Resources needed to perform tests:

❖ System testing, tests the system under extreme conditions.

❖ Tests like performance tests require high end hardware

configurations that are expensive to acquire.

145
 System Testing
Common Challenges
4. Reproducibility:
❖ In software testing, reproducibility means the ability to consistently recreate a
defect or issue under the same conditions.

❖ This is essential for:

• Diagnosing root causes

• Validating fixes

• Preventing regressions
146
 System Testing
Common Challenges

4. Reproducibility:
❖ Most problems that show up during system testing do so because of a complex
combination of factors from the product, the software environment and the
hardware configuration.

❖ Most "problems" found during the integration testing tend to be difficult to

reproduce and resolve.

147
 System Testing
Common Challenges

4. Reproducibility: Example: Billing System Integration Bug

Imagine a billing engine that intermittently fails to apply tax during peak load:
• Only happens when memory usage spikes

• Depends on a specific version of the tax API

• Triggered by a rare concurrency pattern
Reproducing this requires:
• Matching the exact hardware configuration
• Simulating load conditions
• Aligning API response timing
148
 System Testing
Common Challenges

5. Problem Identification:
❖ A natural consequence of the difficulty to reproduce results in the system tests is
that problem analysis and responsibility allocation becomes a more serious issue
than in the case of integration testing.

❖ Does the problem lie in the product or in the environment? Which should be
corrected? These are difficult questions with no easy answers.

149
 System Testing
Common Challenges
5. Problem Identification: Why Reproducibility Is Hard in System & Integration
Testing?

Unlike unit testing, where inputs/outputs are tightly controlled, system/integration testing involve:
❖ Multiple modules interacting
❖ Variable environments (OS, drivers, memory, network)
❖ Timing-sensitive operations (race conditions, concurrency)
❖ External dependencies (APIs, databases, hardware)

This leads to non-deterministic behavior, where a bug might appear once and vanish on next run.

150
 Integration Testing vs System Testing
Aspect Integration Testing System Testing

Tests interactions between integrated

Scope Tests the entire system as a unified whole
modules or components
Verifies data flow, control flow, and interface Validates overall system behavior against
Focus
contracts between modules functional and non-functional specs
Test Level Mid-level testing, follows unit testing High-level testing, follows integration testing
Top-down, bottom-up, sandwich, and big Functional testing, performance testing,
Techniques Used
bang integration strategies security testing, usability testing
Performed By Developers or integration-focused testers QA teams or system-level testers
Ensure the system meets business and user
Goal Ensure modules work together correctly
requirements
Requires stubs and drivers to simulate Requires complete system setup including
Dependencies 151
missing components hardware, software, and network layers
Strategic Insight
❖ Integration Testing is ideal for catching interface mismatches and data
flow issues early—especially critical in distributed systems with layered
responsibilities.

❖ System Testing ensures that the final product behaves as expected in real-
world scenarios, making it essential for stakeholder validation and release
readiness.
152
 Installation Testing

❖ Before the product can be used on a customer site; it has to be installed

successfully.

❖ Installation tests are done to ensure that the product is packaged

correctly and can be installed successfully using the instructions given in
the installation documentation.

153
 Installation Testing

Installation Testing verifies that:

❖ The software can be installed, configured and launched successfully.

❖ The installation package includes all necessary components.

❖ The installation instructions are accurate and complete.

❖ The product behaves as expected immediately after installation.

It’s often called implementation testing, and it’s typically performed in the final stages
before release or deployment.

154
 Installation Testing

Key Objectives
❖ Ensure correct packaging of files, dependencies, and configurations.

❖ Validate installation paths, registry entries, and environment variables.

❖ Confirm minimum system requirements (disk space, RAM, OS version).

❖ Test upgrade paths from older versions.

❖ Verify uninstallation leaves no residual files or registry clutter.

❖ Check error handling for failed or partial installations.

155
 Installation Testing
Steps to be followed
1. Packaging:
❖ Once the product features are tested, the product needs to be packaged for use by the
customers.

❖ Packaging can be the creation of a master copy in a media like a CD or the creation and
publishing of a location (ftp site / URL) from where the user can download/install the
product.

❖ The packaging has to be easy to use, customizable and conforming to any platform or
product standards.
156
 Installation Testing
Steps to be followed :Packaging

Reproducibility-friendly Matrix

157
 Installation Testing
Steps to be followed
2. Documenting:
❖ Accompanying the product would be some documentation about how to install

the package.

❖ Such installation documentation should contain screen captures of the various

screens that would come up during installation, specify various options possible
and the actions in each of these options and lead the user to successfully install
the product in his environment from the media.

158
 Installation Testing
Steps to be followed: Documenting

Reproducibility-friendly Matrix

159
 Installation Testing
Steps to be followed
3. Installing: The person performing the installation tests should get the media and the
documentation listed above, follow the instructions given in the documentation and
ensure that:
❖ The screen shots and the defaults shown in the documentation are accurate.
❖ There is a match between the documentation and the actual behavior of options
chosen, installation flow and the results observed.
❖ If erroneous inputs are given, the installation system handles the errors gracefully
and gives appropriate error messages.
❖ No extraneous or repetitive information is sought from the user unnecessarily.

160
 Installation Testing
Steps to be followed : Installing

Reproducibility-friendly Matrix
161
 Installation Testing
Steps to be followed
4. Verifying:

❖ Every installation packaging should have some means of verifying that the

installation has been completed successfully.

❖ After installation, a "self test" suite should be designed and run.

❖ Such a self test should have clear success criteria.

162
 Installation Testing
Steps to be followed : Verifying

Reproducibility-friendly Matrix
163
 Regression Testing

❖ Regression Testing is a type of software testing that ensures that the new
changes or updates to the code haven’t unintentionally broken existing
functionality.

164
 Regression Testing

❖ Product development goes in iterations of built-test-fix.

❖ It is obvious that from one cycle to the next, the older defects should not
re-surface.

165
 Regression Testing
Example

Cycle 1 – Initial Defect Discovery and Fixes

166
 Regression Testing
Example
Defects
Step Activity Outcome Regression Risk
Involved
1 Build 2 deployed — Testing begins —

2 New defects discovered D4, D5 Root cause analysis —

3 Fixes implemented D4, D5 Corrections made May impact D1–D3 fixes

4 Re-testing of new fixes D4, D5 Fixes confirmed D1–D3 not re-tested

5 Regression testing D1, D2, D3 Risk of regress Possible

skipped reappearance
6 Recommended: D1, D2, D3 Fixes re-validated Regression avoided
Regression tests

Cycle 2 – New Defects and Regression Risk

167
 Regression Testing

❖ Reappearance of an earlier defect is termed as a regress.

❖ In the example above, if the tests for fixes of D1/D2/D3 would have also
been run at the end of the second cycle, then a regress of D1/D2/D3
would have been captured.

168
 Regression Testing

❖ Regression test can be defined as tests that are run to verify that
problems do not resurface tests are (or then regress).

169
 Regression Testing
What tests should constitute regression tests?

❖ Tests that test out key product functionality.

❖ Tests that exercise those product areas that are historically defect prone.

❖ Tests for defects fixed in the last few cycles

170
 Regression Testing

❖ Regression tests are usually designed to run automatically every time a product is built.

❖ The choice of what to test should be done judiciously, balancing the knowledge of the
internal code and design with the common external usage scenarios.

❖ It is important to keep the size of these tests within manageable limits so that they can be
completed in reasonable time.

❖ Keep the list of regression tests updated by deleting unwanted tests and by adding tests for
the new features periodically.

171
 Regression Testing
Strategy Overview

Purpose:
❖Ensure that new changes (bug fixes, enhancements, refactoring) do
not break existing functionality.

172
 Regression Testing
Strategy Overview
Key Principles
Principle Description

Tests should run automatically after each build to reduce manual

Automation
effort.

Selective Coverage Include high-risk, high-impact, and frequently used features.

Balanced Design Combine internal code awareness with external usage patterns.

Keep test suite lean (minimal & meaningful) to ensure fast

Manageable Scope
execution and actionable (clear & trackable) results.
Continuous Regularly prune outdated tests and add coverage for new
Maintenance features. 173
 Acceptance Testing

Purpose:

❖ To validate that the product meets the customer’s expectations and contractual
requirements in their specific environment, using their data and performance
benchmarks.

174
 Acceptance Testing

❖ When a product is designed for use by a specific customer, it is usually required to

pass certain tests that the customer considers representative of his typical
environment.

❖ At the time of initial requirements gathering, a set of acceptance tests would be

specified by the customer.

❖ The performance of a product would be considered satisfactory (by the customer)

only if it has passed the acceptance tests.

175
 Acceptance Testing
Key Characteristics
Attribute Description

Customer-defined Test cases are specified by the customer during requirement gathering.
Environment-
Tests simulate the customer’s actual usage conditions.
specific
Data-driven Uses customer-provided datasets to validate functionality.

Output-sensitive Expected outputs must match customer-defined results exactly.

Performance-bound May include load(users) tests to validate response time and throughput.

Pass/Fail Criteria Clearly defined success metrics—no ambiguity allowed.

176
 Acceptance Testing
Acceptance Testing Workflow
Step Activity Stakeholder Involvement
1 Requirements finalized Customer, Business Analyst
2 Acceptance test cases documented QA Lead, Customer
3 Environment setup (mirroring customer) DevOps, QA
4 Test data provided by customer Customer
5 Functional and performance tests executed QA Team
6 Results validated against expected output QA + Customer
7 Sign-off or feedback Customer
177
 Ordered Testing Flow in SDLC
Phase Testing Type Purpose

Development White Box Testing Tests internal logic, control flow, and code paths (unit-level).

Post-Unit Testing Integration Testing Validates interactions between modules or components.

Tests functionality without knowing internal code (can span

Functional Testing Black Box Testing
multiple phases).

System Testing System Testing Tests the complete system as a whole against requirements.

Deployment Prep Installation Testing Verifies correct installation, configuration, and environment setup.

Post-Changes Regression Testing Ensures new changes haven’t broken existing functionality.

Final Validation Acceptance Testing Confirms system meets business needs and is ready for release.
178
 Ordered Testing Flow in SDLC
Visual Sequence

White Box Integration Black Box System Installation Regression Acceptance

Note: Black box testing is a technique, not a phase—so it’s applied during System,
Acceptance, and even Integration testing depending on context.

179
 People Issues in Testing

❖ Out of all the life cycle activities in software development, testing activity faces
the highest number of people related issues.
❖ Generally, employees do not want to be in the role of a tester.
❖ There is a perception that testing is a mundane job and does not require any
special skills.
❖ Given a choice, most people prefer to be in development rather than in testing

180
 People Issues in Testing
Issue Category Description Impact
Educational Gaps Testing is rarely taught formally in Leads to undervaluing testing as a skilled
universities or tech programs. discipline.

Lack of Visibility Testing receives less attention than design Fewer role models, less community
or development in forums and media. engagement, lower prestige.

Management Bias Leadership often prioritizes development Testers feel undervalued; defect reports
over testing. may be ignored or deprioritized.

Inaction on Defects Management may not act decisively on Erodes trust, reduces motivation, and
issues raised by testers. weakens quality culture.

Career Testing is seen as a temporary role before High turnover, low retention, and lack of
Misconceptions moving into development. long-term investment in QA. 181
 People Issues in Testing

Root Causes of Misconception

❖ Testing is perceived as repetitive rather than analytical and creative.
❖ Skill depth is underestimated—people don’t realize the complexity of test design,
automation, performance tuning, and defect attribution.
❖ Organizational structures often silo (isolates) testers from strategic decision-
making.

182
 People Issues in Testing
Contending with Negative Perceptions of Testing
Core Principles for Organizational Change
Principle Actionable Implementation Impact
Align compensation, awards, and visibility for
Parity in Recognition Builds morale and signals equal value.
testers with developers.
Mandate defect resolution before release; Reinforces the importance of testing
Defect Accountability
celebrate defect discovery as a win for quality. outcomes.
Balanced Talent Assign top engineers to testing roles, not just
Elevates testing’s technical credibility.
Allocation development.
Create formal growth ladders: QA Analyst →
Encourages long-term commitment to
Career Pathing Automation Lead → Test Architect → QA
testing.
Strategist.
Rotate developers into testing roles Fosters empathy, cross-functional skill,
Role Rotation
periodically. and respect.
Train testers in communication, conflict Improves collaboration and defect
Soft Skill Investment 183
resolution and stakeholder engagement. reporting clarity.
 People Issues in Testing
Soft Skills That Shift the Narrative

❖ Pride in Purpose: Developing better attitude and pride in testing among

testers and developers. Frame testing as a craft that protects users and
upholds product integrity.
❖ Constructive Communication: Report defects with clarity and neutrality—
focus on outcomes, not blame. Report defects with the sole intent of getting the
product defect free.
❖ Collaborative Mindset: Position testers as partners in quality, not adversaries
in fault-finding.

184
 Management Structures for Testing
in Global Teams

❖ Organizing testing and development teams is about collaboration,

accountability and shared ownership of quality.

185
 Management Structures for Testing
in Global Teams
❖ In order to organize the
Integrated Testing Team Model
testing and development
teams to minimize conflicts Dedicated Testing Team Model

and deliver a time bound, Testing Teams in Geographically

Distributed Product Teams
high quality product, some
organizational structures are Hybrid Models

presented.
186
 Management Structures for Testing
in Global Teams
Integrated Testing Team Model

❖ In this model, there is

one Project Manager.
❖ He manages both the
development and the
testing functions.

187
 Management Structures for Testing
in Global Teams
Integrated Testing Team Model: Advantages
Benefit Description

Resource Multiplexing Project Manager can multiplex developers and testers, improving
utilization
Load Balancing Development-heavy early stages and test-heavy later stages are
smoothed out
Reduced Conflict Shared goals and daily interaction foster empathy and alignment

Job Rotation Builds empathy, cross-skills, and deeper system understanding

188
 Management Structures for Testing in Global Teams
Integrated Testing Team Model: Disadvantages
Concern Description

Defect Minimization Bias Project Manager focused on delivery may downplay defects to meet
deadlines.
Loss of Critical Perspective When the same people test what they build, different perspectives may
not come into the picture.
Skill Set Mismatch Testing requires analytical, investigative, and adversarial thinking—skills
not always aligned with development.
Testing Becomes Validation Instead of finding defects, testing may devolve into proving that the
product “works.”
Reduced Checks and Balances Without independent QA, accountability and objectivity may suffer.

Managerial Conflict of Interest A single manager may prioritize delivery over quality, especially under
189
pressure.
 Management Structures for Testing
in Global Teams
Dedicated Testing Team Model

❖ In this model, the testing team

reports directly to the next level of
senior management.
❖ The conflicts between operational
delivery responsibilities and testing
responsibilities are avoided.

190
 Management Structures for Testing
in Global Teams
Dedicated Testing Team Model

❖ Some finer variations are possible in this model:

❑ The testing team can further be broken down into sub-teams and each sub-

team assigned the responsibility of a particular product.

❑ Alternatively, the entire testing team can be considered as one set of

resources who are assigned the testing of different products on a need basis.

191
 Management Structures for Testing in Global Teams
Dedicated Testing Team Model- Disadvantages

Disadvantage Description

Adversarial Perception Direct reporting of the testing team to Senior

Management may lead to the testing and
development teams viewing each other as adversaries.

Lack of Work Diversity Testers are confined to testing and developers to

development, limiting role variety and potentially
reducing motivation over time.
192
 Management Structures for Testing in Global Teams
Testing Teams in Geographically Distributed Product Teams

❖ Geographically distributed product teams present some more opportunities for

organization structures and for maximizing the quality and optimizing the resources.
❖ There are two models possible for geographically distributed product teams.

1. Self-contained locations

2. Cross location testing

193
 Management Structures for Testing in Global Teams
Testing Teams in Geographically Distributed Product Teams
Self-contained locations
Feature Description
Component-Based Product is split into distinct components, each
Division assigned to a specific location.

Mini-Project Each location manages its own development

Ownership and testing for its assigned component.

Integrated Dev-Test Local teams handle both coding and

Teams validation, reducing handoffs and delays.

Location Manager Each Location Manager oversees the full

Accountability lifecycle of their component—design, build,
test. 194
 Management Structures for Testing in Global Teams
Testing Teams in Geographically Distributed Product Teams
Cross location testing

❖ One possible variation to the

above model is to let one
location test the parts of the
product developed in another
location.

195
 Management Structures for Testing in Global Teams
Testing Teams in Geographically Distributed Product Teams
Cross location testing: Advantages

Advantage Explanation

Operational Testing and development tasks are shared across

Responsibilities are Distributed locations, reducing centralization.

Wider Distribution of Both locations engage with all product

Product Knowledge components, leading to broader understanding.

196
 Management Structures for Testing in Global Teams
Testing Teams in Geographically Distributed Product Teams
Cross location testing: Disdvantages

Disadvantage Explanation

Excessive Need for Distributed teams require more structured and frequent
Communication communication to coordinate effectively.
Reduced Face-to-Face Co-located teams benefit from spontaneous discussions and
Interaction quicker issue resolution, which is harder to replicate remotely.
Impact on Throughput Lack of direct interaction may slow down problem-solving and
and Speed reduce overall efficiency.

197
 Management Structures for Testing in Global
Teams
Testing Teams in Geographically Distributed Product
Teams
❖ Another possible variation to the above models is a Globally Dedicated testing
team.
❖ In this testing competency centers are created by designating a particular
location for testing.
❖ This is very similar to the dedicated testing team model.

198
 Management Structures for Testing in Global
Teams
Testing Teams in Geographically Distributed Product
Teams

Feature Explanation
Designated Testing A specific site is assigned as the central hub for testing activities
Location across the product.
Similarity to Dedicated Mirrors the structure of dedicated testing teams, but adapted
Testing Teams for global distribution.
Enables round-the-clock productivity: products developed
Advantage: Exploitation of during day time in one location can be tested during the
Time Zone Differences subsequent hours in another location, maximizing the 24-hour
199
cycle.
 Management Structures for Testing in Global
Teams
Hydrid Models

Testing Type Responsibility Reporting Line

Integrated within
White Box Testing Performed by developers
development teams
Handled by a dedicated QA Reports to the Project
Black Box Testing
team Manager
Managed by a specialized Reports directly to Senior
Integration Testing
team Management

200
 Management Structures for Testing in Global
Teams
Hydrid Models: Advantage

Aspect Explanation
Testing responsibilities are aligned with
Proximity to Defect Injection Points development layers, enabling earlier defect
detection.
Each testing layer(white-box, black-box,
integration) is positioned close to its
Strategic Placement of Testing Teams
relevant development context, improving
efficiency and accountability.
201
 Management Structures for Testing in Global
Teams
Hydrid Models: Practical Interpretation
Why This Improves Efficiency &
Testing Type Aligned With
Accountability
They know the internal logic best, so
White Box Testing Developers writing the code
they can catch logic errors early.

Focuses on user-facing functionality,

Black Box Testing QA team under project manager
ensuring requirements are met.

Ensures system-wide coordination

Integration Testing Senior-level oversight
and catches cross-module defects.
202
 Metrics for Testing Phase

❖ The primary purpose of testing is to uncover the presence of defects and thereafter,
to feed this information back into the system in an effort to minimize recurrence of
the defects.
❖ The purpose of metrics in the testing phase is to transform testing from a subjective
activity into a measurable, improvable, and accountable process.

203
 Metrics for Testing Phase
Core Metrics for the Testing Phase

Metric Definition Significance Implications for Improvement

Percentage of code Higher coverage reduces the Use coverage reports to identify
Code
executed during risk of untested logic and blind spots and refine
Coverage
testing unexpected behaviour unit/integration test suites

Indicates gaps in test design

Post- Number of defects Analyse missed defects to improve
or execution; defects reaching
Testing Defect discovered after test cases, acceptance criteria, and
users increase cost and
Count testing is complete traceability matrices
reputation risk

Reflects regression test Strengthen regression suites,

Rate of Frequency of
effectiveness; recurring bugs prioritize defect root cause
Problem recurring issues
suggest poor test planning or analysis, and refine reproducibility
Reappearance across releases 204
weak coverage checklists
205
 Metrics for Testing Phase
Why These Metrics Matter?

❖ Reproducibility & Attribution: These metrics help build a defensible defect attribution
matrix—who missed what, and why.
❖ Stakeholder Communication: They offer concrete data to support escalation
workflows and justify testing investments.
❖ Organizational Learning: Recurring defects can be used as case studies to redesign
testing protocols and improve team accountability.

206
 Syllabus Module 4:
Project Management in the Testing and
Maintenance Phase
• Project Management in the Testing phase: Introduction, what is testing? what
are the activities that makeup testing? test scheduling and types of tests, people
issue in testing, management structures for testing in global teams, metrics for
testing phase.
• Project Management in the Maintenance Phase: Introduction, Activities during
Maintenance Phase, management issues during Maintenance Phase,
Configuration management during Maintenance Phase, skill sets for people in
the maintenance phase.
207
Project Management in the
Maintenance Phase
 What is the Maintenance Phase?

The maintenance phase begins after a product version is released to the

market. It focuses on responding to real-world usage, where customers
may encounter:

❖ Unexpected behavior due to misaligned expectations.

❖ Actual defects that violate stated requirements.

❖ Requests for changes to better match intended functionality.

209
 What Maintenance Phase deals with?

Activity Purpose

Understand whether customer-reported issues are valid and

Evaluate Change Requests
actionable
Determine if the change aligns with product goals and
Assess Applicability
architecture

Implement Fixes or Enhancements Modify the product to address defects or improve behaviour

Regression Testing Ensure new changes don’t break existing functionality

Deliver Updates Provide the corrected or enhanced version to affected users

210
 Lifecycle Transition:
Maintenance → Obsolescence
Progression takes place in the software product lifecycle, where the maintenance phase
transitions into the obsolescence phase.
Phase Description Customer Impact

Customers receive patches, updates and

Active support phase where defects are
support for reported issues.
Maintenance fixed, enhancements are made and updates
E.g. Latest quality update for the Windows 10
are delivered.
was released on September 9, 2025

End-of-life phase where the product is no No further fixes or enhancements;

longer supported or updated. customers are expected to migrate.
Obsolescence
E.g. Microsoft announced Windows 10 end of E.g. Customers using Windows 10 OS should
support is on October 14, 2025 migrate to Windows 11 211
 Key Characteristics of the
Obsolescence Phase

❖ No fixes or updates are provided—even for critical defects.

❖ Support channels are closed or redirected to newer versions.
❖ Documentation may be archived but not maintained.
❖ Migration paths are typically offered to help customers transition.
❖ Compliance and security risks may increase if customers continue using
unsupported versions.

212
 Activities during Maintenance Phase

Activity Purpose

Customers or internal teams report issues observed in

Problem Reporting
the released product.
Development teams analyze, fix and validate the
Problem Resolution
reported issues.
Verified fixes are packaged and delivered to affected
Solution Distribution
users or environments.
Insights from past issues are used to prevent future
Proactive Defect Prevention
ones.

213
 Central Role of the Problem Repository

❖ The Problem Repository acts as the single source of truth for all
reported and resolved issues.

❖ The problem repository is a database that contains all the information

about all the problems that were encountered or reported.

❖ It enables traceability, accountability, and reproducibility across the

maintenance lifecycle.

214
 Central Role of the Problem Repository
Key Data Stored per Problem Record
Field Description
Unique Identifier Distinct ID for tracking and referencing the issue
Customer Info Who reported the issue

Platform & Environment Where the issue occurred (OS, hardware, configuration)

Product Version Specific release affected

Problem Description Detailed account of the issue

Reproduction Test Case Steps or scripts to reproduce and validate the issue

Root Cause Analysis Technical insight into why the issue occurred
Fix Details What was changed to resolve the issue
Fix Location Where the fix can be accessed or deployed from

215
 Central Role of the Problem Repository

Access Control:
❖ Not all fields are visible to all stakeholders.
❖ For example, fix implementation details are typically restricted to the
development team to protect IP and maintain security.

216
 Activities during Maintenance Phase
Problem Reporting

What Is Problem Reporting?

❖ Problem reporting begins after a product is released and customers
start using it.
❖ If they experience behaviour that differs from what they expect,
whether due to misunderstanding or actual defects—they report it as a
problem to the organization that supplied the product.

217
 Activities during Maintenance Phase
Problem Reporting
Initial Contact: The Support Call
➢ Customers reach out via phone, email, or internet.
➢ The Customer Support Group/Call center (not the development team)
handles these calls.
A Support Analyst is assigned to:
❖ Verify caller authorization (e.g., valid license, role in organization).
❖ Gather detailed information about the issue.

218
 Activities during Maintenance Phase
Problem Reporting
Information Collected by the Support Analyst

Key Questions Asked Purpose

What is the product that is giving the problem? Identify the scope of the issue

What is the environment (hardware/software)? Understand platform-specific behavior

What was the user doing when the issue

Capture the scenario and context
occurred?

Is the issue reproducible or random? Determine consistency and severity

219
 Activities during Maintenance Phase
Problem Reporting
Initial Assessment
❖ If the issue is a user error, the analyst:
• Provides corrective guidance
• Updates the Problem Repository with the report and resolution
• Flags recurring user errors for management review (may indicate poor documentation or
product design)
❖ If the issue is not a user error, the analyst checks the Problem Repository to see if:
• A similar issue with a fix exists → fix is distributed
• A known issue without a fix (Known problem) exists → workaround is offered and
repository updated.
oThis update is necessary to prioritise the fixing of these known problems. 220
 Activities during Maintenance Phase
Problem Reporting
Handling New Problems
If the issue is not previously reported, it is treated as a new problem:
1. Criticality is evaluated based on business impact:
❖ Example: Invoicing failure = high severity (show stopper)
❖ Misaligned report field = low severity
2. A new record is created in the Problem Repository:
❖ Includes all collected details
❖ Assigns a unique reference number
❖ Logs severity/priority to guide resolution efforts
221
Activities during Maintenance Phase
Activities in Problem Reporting

222
 Activities during Maintenance Phase
Problem Resolution

❖ Once a reported issue is confirmed as a new problem without an

existing fix, responsibility shifts from the support team to the product
development organization.
❖ Their goal is to analyze, reproduce, fix, and validate the issue before
distributing the solution.

223
 Activities during Maintenance Phase
Problem Resolution
Step-by-Step Breakdown
1. Criticality Analysis
❖ The development team reviews the problem’s impact on the customer’s
business.
❖ Severity (entered by the support analyst) helps prioritize which issues to
address first—showstoppers take precedence.
2. Developer Assignment
❖ A single developer is assigned to own and track the resolution process.
224
 Activities during Maintenance Phase
Problem Resolution
Step-by-Step Breakdown
3. Problem Reproduction
❖ The developer attempts to recreate the issue using the environment and test
case described in the problem repository.
❖ Accurate reproduction is essential to identify the root cause.

225
 Activities during Maintenance Phase
Problem Resolution
Step-by-Step Breakdown
4. Root Cause Analysis & Fix
❖ If reproducible:
❖ The developer locates the source code files responsible.
❖ Implements the fix and verifies that:
• The issue is resolved
• No other functionality is broken
226
 Activities during Maintenance Phase
Problem Resolution
Step-by-Step Breakdown
4. Root Cause Analysis & Fix
❖ Follows formal change control processes:
• Authorization
• Peer review
• Execution
• Testing
• Re-baselining

227
 Activities during Maintenance Phase
Problem Resolution
Step-by-Step Breakdown
5. Non-Reproducible Issues
❖ If the issue cannot be reproduced:
➢ The developer collaborates with the support analyst and possibly the
customer.
➢ Techniques include:
• Phone walkthroughs
• Remote diagnostics
• On-site visits
228
 Activities during Maintenance Phase
Problem Resolution
Step-by-Step Breakdown
6. Repository Updates
❖ Throughout the process, the problem repository is updated with:
• Status changes
• Diagnostic findings
• Fix progress
• Any unresolved issues
229
Activities during Maintenance Phase
Activities in Problem Resolution

230
 Activities during Maintenance Phase
Problem Resolution

Outcome
❖ Once the fix is implemented and validated, the product is rebuilt and
the solution is ready for distribution to affected customers,
transitioning into the Solution Distribution phase.

231
 Activities during Maintenance Phase
Solution Distribution

When should the fix

Should it be sent as
be sent to the
soon as it is made
customer?
and the product re-
built?

232
 Activities during Maintenance Phase
Solution Distribution
Key Factors Influencing Fix Distribution Timing
❖ Severity of the Issue
• Showstopper: Immediate dissemination is critical.
• Minor Issue: Prefer bundling into scheduled patch releases.
❖ Customer Considerations:
• Installing individual minor fixes is inefficient.
• Bundled patches reduce operational overhead.

233
 Activities during Maintenance Phase
Solution Distribution
Key Factors Influencing Fix Distribution Timing
❖ Supplier Considerations:
• Frequent small releases strain resources.
• Multiple configurations increase support complexity.
❖ Best Practices:
• Accumulate minor fixes into intermediate cumulative releases.
• Maintain a problem repository with fix availability details.
• Ensure support analysts are informed to relay updates to customers.
• Consider proactive distribution if issue is severe and likely to affect others.
234
 Activities during Maintenance Phase
Solution Distribution
Fix Distribution Decision Matrix
Criteria Immediate Fix Bundled Patch Release
Severity: Showstopper Yes Not preferred
Severity: Minor Not preferred Yes
Customer Impact: High Yes Not preferred
Customer Impact: Low Not preferred Yes
Resource Cost to Supplier High Lower
Configuration Complexity Increases Controlled
Risk of Issue Affecting Others Distribute widely Case-by-case

235
 Activities during Maintenance Phase
Proactive Defect Prevention

❖ Reactive maintenance- Carrying out maintenance to fix problems after the

problems surface.
❖ Proactive defect prevention shifts the focus from reactive firefighting to strategic
foresight.
❖ It leverages historical defect data to reduce future maintenance load, improve
product quality, and free up bandwidth for innovation.

236
 Activities during Maintenance Phase
Proactive Defect Prevention
Key Strategies for Prevention
Analysing common user errors and updating documentation as necessary
Trigger: High volume of user errors classified as misunderstandings.
Action: Audit manuals, tooltips, and help content for ambiguity.
Outcome: Reduces support calls and improves user autonomy.
Analysing common user errors and changing the product as necessary
Trigger: Repeated user errors despite clear documentation.
Action: Re-evaluate product design assumptions; align with user expectations.
Outcome: Enhances usability and reduces friction.

237
 Activities during Maintenance Phase
Proactive Defect Prevention
Key Strategies for Prevention
Publishing common user errors and work-arounds on a bulletin board or a website
Trigger: Frequently reported but easily resolvable issues.
Action: Publish FAQs, workarounds, and error resolutions on a public portal.
Outcome: Cuts support costs and improves customer satisfaction.
Performing root cause analysis of problem areas and Code Refactoring
Trigger: High defect density in specific modules.
Action: Conduct root cause analysis; refactor or redesign problematic areas.
Outcome: Improves code maintainability and long-term stability.

238
 Activities during Maintenance Phase
Proactive Defect Prevention
Key Strategies for Prevention

Regression Test Enhancement

Trigger: Recurring bugs in critical paths.
Action: Integrate reproducible test cases into regression suites.
Outcome: Catches defects early and improves release confidence.

239
 Activities during Maintenance Phase
Key Strategies for Prevention
Trigger Condition Recommended Action Impact Area Stakeholder Benefit
High volume of user Update documentation UX / Support Reduces support load,
errors improves clarity
Persistent user errors Modify product behavior Design / Aligns product with user
despite documentation Requirements expectations
Frequently reported Publish self-service Support / Customer Enables faster
minor issues workarounds Portal resolution, lowers cost
High defect density in Refactor or re-architect Engineering / QA Improves
specific modules code maintainability and
stability
Recurring bugs in Expand regression test QA / Release Prevents regressions,
critical workflows coverage Management boosts confidence

240
 Management Issues During the
Maintenance Phase

1. Incomplete information from the customers

❖ Customers often provide vague or emotional problem descriptions.
❖ Support analysts must extract relevant technical details using soft skills.
❖ A checklist helps ensure minimum required information is gathered:
• Product name and version
• Hardware specs (RAM, disk space, devices)
• Software environment (OS, supporting software, drivers)
• Problem reproducibility and concurrent software

241
 Management Issues During the
Maintenance Phase

2. How critical is the problem?

❖ Customer urgency is universal: Every customer perceives their issue as
urgent, even if it's minor (e.g., a formatting error).
❖ Emotional reactions are valid: Discrepancies can cause justified frustration,
regardless of technical severity.
❖ Need for fairness and prioritization: To serve all customers equitably, issues
must be ranked by business impact.

242
 Management Issues During the
Maintenance Phase

2. How critical is the problem?

❖ Objective prioritization framework:
Highest Priority: System is non-operational or severely disrupts business.
Medium Priority: Problem has a messy workaround but affects usability.
Lower Priority: Issue has a reasonable workaround or is cosmetic/peripheral.

243
 Management Issues During the
Maintenance Phase

3. Problems that do not reproduce

❖ Some issues occur only in customer environments and not in development setups.
❖ Causes include large data loads, unique configurations, or third-party interactions.
❖ Solutions:
➢ Remote diagnostics via dial-in
➢ On-site visits by development staff

244
 Management Issues During the
Maintenance Phase

4. Whose problem is it?

❖ Componentized systems involve multiple owners across teams and vendors.
❖ Challenges include:
➢ Identifying the responsible party
➢ Deciding whether to fix the service provider or the client
➢ Coordinating with third-party vendors for resolution

245
 Management Issues During the
Maintenance Phase

5. Who gets the fixes?

❖ Decision needed on whether only the reporting customer receives the fix.
❖ If the issue is likely widespread, proactive distribution to all customers is
recommended.

246
 Management Issues During the
Maintenance Phase

6. How to distribute the fixes?

❖ Two models:
➢ Push Model: Fixes are sent automatically (e.g., CD, email, internet).
➢ Pull Model: Fixes are stored in a repository (typically a web site).
Customers pull the fixes they want based on the symptoms/problems that
they encounter.

247
 Management Issues During the
Maintenance Phase

7. Developers unwilling to do maintenance

❖ Maintenance is often seen as less creative than new development.
❖ Solutions:
➢ Job rotation: Rotating developers between maintenance and development
tasks adds variety and reduces monotony.
➢ Component Ownership: Assigning full responsibility for a component’s
maintenance and enhancement to a team:
• Encourages accountability.
• Motivates developers to produce high-quality, defect-free code upfront.
• Builds deeper expertise and pride in the component’s performance.
248
 Management Issues During the
Maintenance Phase
8. Balancing maintenance with new development
❖ Resource contention:
• Maintenance consumes time and effort that could otherwise be used for new
development.
• Whether handled by separate teams or multiplexed within the same group, this trade-
off affects innovation cycles.
❖ Maintenance is essential but non-revenue generating:
• Timely bug fixes are critical for customer satisfaction and retention.
• Maintenance rarely contributes directly to revenue growth.

249
 Management Issues During the
Maintenance Phase

8. Balancing maintenance with new development

❖ Need for strategic resource allocation:
• Use metrics to track:
✓ Frequency of problem occurrences.
✓ Customer expectations for fix turnaround times.
• These metrics help estimate and allocate appropriate maintenance resources.

250
 Management Issues During the
Maintenance Phase

8. Balancing maintenance with new development

❖ Goal: Reduce maintenance load without sacrificing quality:
• Once resources are allocated, aim to minimize them over time.
• Maintain high standards of quality and customer satisfaction.
❖ Core principle: Prevention is better than cure:
• Invest in processes that reduce defect generation.
• Emphasize robust design, thorough testing, and proactive quality assurance.

251
 Management Issues During the
Maintenance Phase

9. Regressions and chain maintenance

❖ Repeated fixes for the same issue are inefficient and frustrating.
❖ Prevention:
• Effective Software Configuration Management (SCM)

252
 Configuration Management During the
Maintenance Phase

1. Customer's environment is not directly under the control of the development

organisation
❖ Customers may customize screens or delete/change files.
❖ They may use different operating systems or hardware configurations.
❖ Diagnosing issues becomes difficult due to non-standard environments.

253
 Configuration Management During the
Maintenance Phase

2. Fixes have some inherent dependencies

• Fixes may require:
• Co-requisite fixes: Set of fixes must be installed together as one unit
• Pre-requisite fixes: Fixes that must be installed before the current fix can be
applied.
• Installation tools must support dependency handling to ensure proper application
of fixes.

254
 Configuration Management During the
Maintenance Phase

3. A way to identify the configuration in a customer site

❖ Customers may install fixes selectively.
❖ It's essential to:
• Know which fixes are present.
• Detect dependencies - Co-requisite / pre-requisite fixes at the customer site.
• Reproduce the customer’s environment accurately at the development site for
debugging.

255
 Configuration Management During the
Maintenance Phase

4. Mapping the customer environment to source environment

❖ Customers use executable files; developers work with source code.
❖ SCM systems manage source configurations, not customer executables.
❖ SCM should enable traceability from executables back to source components.
❖ This highlights the importance of Design for Diagnosability and
Maintainability.

256
 Configuration Management During the
Maintenance Phase
4. Regression testing for fixes:
❖ SCM system at the development site includes:
➢ Source files
➢ Test cases
➢ Test results
❖ For each fix:
1. Design a test case to reproduce the issue and verify the fix.
2. Re-baseline the source changes.
3. Update the configuration repository and test procedures to include the new test.
257
 Configuration Management During the
Maintenance Phase

4. Regression testing for fixes:

❖ This ensures:
✓ Fix effectiveness
✓ Prevention of regressions - Bug does not regress again (i.e., it does not re-
appear, or if it does re-appear, it should get caught).

258
 Skill Sets for People in the
Maintenance Phase
1. Why developers are not the first point of contact?
❖ Developers are focused on building new products and versions.
❖ They may not be available for immediate customer support.
❖ Developers may show bias when diagnosing issues in their own code.
❖ Their core strength lies in deep technical development, not customer-facing
communication.
❖ Handling frustrated customers requires emotional intelligence and objectivity—
skills not always emphasized in development roles.

259
 Skill Sets for People in the
Maintenance Phase
2. Key skill sets for Support Analysts
Strong understanding of the product’s external functionality.
Excellent communication skills to extract specific details from customers.
Ability to take a balanced, objective view—considering both customer and product
perspectives.
Skill in identifying patterns and abstracting test cases from customer problem
descriptions.
Good follow-through attitude to ensure all stakeholders stay informed and aligned.
Soft skills are critical for Support Analysts to succeed in this role.
260
 Skill Sets for People in the
Maintenance Phase
3. Challenges for development staff in maintenance
❖ Often required to understand and modify someone else’s code.
❖ Maintenance is reactive, unlike planned product development.
❖ Workload is unpredictable due to the nature of incoming issues.
❖ Success depends on:
✓ Flexibility
✓ Quick responsiveness
✓ Adaptability to changing problem contexts
261
262