Quality Throughout the Software Life
Cycle
Jeffrey Feldstein
Cisco Systems
jbf@[Link]
jbf@[Link]
© 2006 Jeffrey Feldstein. All rights reserved. 1
[Link]
Presentation_ID
© 2004 Jeffrey Feldstein. All rights reserved. © 2004 Cisco Systems, Inc. All rights reserved. 2
Agenda
[Link]
• Introduce myself
• Product Phases (Overview)
• Test role in each phase
• Conclusion
• Question and Answer
© 2004 Jeffrey Feldstein. All rights reserved. 3
Introduce Myself
[Link]
• Software Career: 25 years
• Software Test: 7 years
• Test Automation: 6 years
Current position at Cisco:
Manage a group of 65 test engineers located in San
Jose, India (2 sites) and Netanya, Israel, testing a
next generation Network Management Platform
(Active Network Abstraction)
© 2004 Jeffrey Feldstein. All rights reserved. 4
Question 1
[Link]
Software Quality is who’s responsibility?
My answer: Everybody
Quality starts at the beginning and is the
responsibility of Product Management,
Development, Test, Documentation, Program
Management and Support
© 2004 Jeffrey Feldstein. All rights reserved. 5
Question 2
[Link]
Can we test quality into the product
My answer: If you figured out a way to do this,
let’s write the book and retire ☺
© 2004 Jeffrey Feldstein. All rights reserved. 6
Life Cycle Phases
[Link]
• Requirements Definition
• Development
• Test
Internal
External
• Post-Ship
© 2004 Jeffrey Feldstein. All rights reserved. 7
Life Cycle Phases
[Link]
• Requirements Definition
• Development
• Test
Internal
External
• Post-Ship
© 2004 Jeffrey Feldstein. All rights reserved. 8
Life Cycle: Requirements (1 of 4)
[Link]
• Authored by Product Manager (Marketing)
• Developer’s role: Figure out how to build the product
Data structures
Algorithms
Resource Utilization
Security
May be difficult to think about testability, coherence and overall
quality
May already be introducing bugs in their thinking
© 2004 Jeffrey Feldstein. All rights reserved. 9
Life Cycle: Requirements (2 of 4)
[Link]
• Test Role
Find Bugs now (sooner = cheaper)
• Tester’s Concentration during the review
Are the requirements clear and understandable?
Are the requirements testable?
Are their conflicting requirements?
Consider assigning this to one person
Is the overall system cohesive?
© 2004 Jeffrey Feldstein. All rights reserved. 10
Life Cycle: Requirements (3 of 4)
[Link]
• Example 1:
“The GUI must be intuitive and easy to use”
• Example 2:
All data entry screens should have a button labeled
“View Shopping Car”, which resides in a constant
location in the window.
• Which requirement is testable?
• What are some examples from your experience?
© 2004 Jeffrey Feldstein. All rights reserved. 11
Life Cycle: Requirements (4 of 4)
[Link]
• Common omissions
Performance Requirements
Hardware sizing, user response times, database
capacities, scalability requirements
Environmental assumptions
Required Operating systems, browsers, web servers,
database engines, other dependencies
© 2004 Jeffrey Feldstein. All rights reserved. 12
Life Cycle Phases
[Link]
• Requirements Definition
• Development
• Test
Internal
External
• Post-Ship
© 2004 Jeffrey Feldstein. All rights reserved. 13
Life Cycle: Development (1 of 3)
[Link]
• Begins at PRD Approval
• Roughly follows this pattern
Specifications (Functional + others)
Coding
Unit Test
Integration Test
Base-level-performance test
Hand off to test
• Note: All of the testing above is performed by
Development!
© 2004 Jeffrey Feldstein. All rights reserved. 14
Life Cycle: Development (2 of 3)
[Link]
• Functional Specification Review
Is the specification clear and understandable?
Does the specification address all of the requirements
in the PRD?
Is there “extra” functionality (not found in the PRD)?
Is each requirement fully addressed?
Does the specification describe a testable
architecture?
Is all the information required to develop the test
cases present?
© 2004 Jeffrey Feldstein. All rights reserved. 15
Life Cycle: Development (3 of 3)
[Link]
• Encourage informal 1 one 1 between
Development and test engineers
• Have a system for requirements traceability
• Aim for coding the products and the testing
simultanously.
© 2004 Jeffrey Feldstein. All rights reserved. 16
Life Cycle Phases
[Link]
• Requirements Definition
• Development
• Test
Internal
External
• Post-Ship
© 2004 Jeffrey Feldstein. All rights reserved. 17
Life Cycle: Test Planning (1 of 3)
[Link]
• Occurs during development phase
• Typical Documents authored by test
Test Strategy – authored based on approved PRD,
simultaneous with Functional Specification.
Test plans – contains the detailed test cases (scripts)
to be run.
© 2004 Jeffrey Feldstein. All rights reserved. 18
Life Cycle: Test Planning (2 of 3)
[Link]
• Test Strategy
Sometimes called the “Master Test Plan”
Outlines Automation goals and strategy
Describes tools used during test
Lays out the test lab
Security Vulnerability test approach
Lists major testing types (Functional, system, performance,
scale, stress and soak)
Should be able to scope the test effort when done
Allows other teams to give early feedback to the test team
© 2004 Jeffrey Feldstein. All rights reserved. 19
Life Cycle: Test Planning (3 of 3)
[Link]
• Test Plans
Details for each test case (Automated and Manual)
Test to break orientation
Should contain many negative test cases
Testing in hostile environments (low memory, etc.)
• All documents should be reviewed by Product
Management, Development, and Documentation
• Developers can fix bugs simply by reading the
test plan
© 2004 Jeffrey Feldstein. All rights reserved. 20
Life Cycle: Test Execution (1 of 3)
[Link]
• Begins at Handoff to Test (HOT)
• Important scheduling milestone
• Product team defines specific entry criteria
• During test three things happen
Quality improves
Upper management business pressures emerge
Product Managers get anxious
Fatigue sets in for the entire product team
© 2004 Jeffrey Feldstein. All rights reserved. 21
Life Cycle: Test Execution (2 of 3)
[Link]
• Strict Ship Criteria should be established during
the planning phase to avoid bad decisions
• These criteria should be measurable and
objective quality goals
• Agreement should be reached by all teams up
front
© 2004 Jeffrey Feldstein. All rights reserved. 22
Life Cycle: Test Execution (3 of 3)
[Link]
• Some example goals:
All System crash and “must fix” bugs are resolved
All identified and resolved bugs are verified
All customer-found bugs, from previous releases are
verified
Performance goals are met, results are documented
Total number of open bugs not to exceed “XX”
Defect density under a certain threshold
Code and branch coverage test targets are reached
(relative or absolute).
© 2004 Jeffrey Feldstein. All rights reserved. 23
Life Cycle Phases
[Link]
• Requirements Definition
• Development
• Test
Internal
External
• Post-Ship
© 2004 Jeffrey Feldstein. All rights reserved. 24
Life Cycle: Post Ship
[Link]
• Keep testing after ship
• Opportunity to run automation for an extended
period
• Find additional bugs before the customer finds
them
• Good time to invoke Model-Based Testing
• Perform escape analysis on all bugs: Feedback
to improve future test plans
• Post Project Assessment
© 2004 Jeffrey Feldstein. All rights reserved. 25
Conclusion
[Link]
• Product Quality is everybody’s job
• Product development is a team effort
• Plan and set milestones in advance
• All teams agree to phase entry/exit criteria
• Early test involvement leads to higher quality
software shipping sooner
• Test Engineering plays a vital role in every
phase of the product life cycle
© 2004 Jeffrey Feldstein. All rights reserved. 26
Testability – Further Reading
[Link]
Software Test & Performance Magazine (January 2006)
[Link]
© 2004 Jeffrey Feldstein. All rights reserved. 27
[Link]
© 2004 Jeffrey Feldstein. All rights reserved.
[Link]/mbt 28
[Link]
Presentation_ID
© 2004 Jeffrey Feldstein. All rights reserved. © 2004 Cisco Systems, Inc. All rights reserved. 29