Software Testing method

Week 8
Topics covered
- Black-box vs. White-box testing
- Partition and Boundary
- Full Cartesian products
- Test coverage
- Statement coverage
- Branch coverage
- Path coverage
- Tools
 // 1
x = power(2,1);
Test Thinking If ( x == 2 ) print(pass); else print (“failed”);
// 2
- Testing is the process of discovery faults xIf (=xpower (-2,1);
== -2) print(“pass”); else print (“failed”);
- Find if all features work as design & requirement // 3
- Discovery non-functional requirement issues x = power(2,2);
- How do I break software? If (x == 4) print(“pass”); else print (“failed”);
- Test programmatically
- Vary inputs and check if getting output as expected
- When a spec says “the function returns the result of division”, we must
thought of all possible ways to call this function that may give the
correct result or cause errors
- Each scenario test is called a test case
- The role in software development team is tester or QA
- Require coding skills and systematic thinking about possible usage
scenarios
- Automations
 describe ('Test add hour ', function(){
it('toString test', function(){
var t = new Stime("1:1:1");
Testing Terms [Link](2,"hour");
var res = [Link]();
[Link](res,"3:1:1");
- Test case [Link]("hello " + res);
- A test that test a specific purpose of the software });
- Test suite });

- A collection of test cases sharing similar pre-requisites and configuration. Usually can be run
together in sequence
- Different test suites for different purposes
- Test Script
- A script to run a sequence of test cases or a test suite automatically
- Test driver
- A software framework that can load a collection of test cases or a test suite ; handle the
configuration and comparison between expected outputs and actual outputs
- Test automation
- A process of executing test cases automatically when a trigger occurs
Black-Box and White-Box Testing
- Black Box (aka Functional , aka Spec-Based)
- Tests derived from functional requirements
- Input/Output Driven (vary input and compared with expected output)
- Internal source code of software is not seen when designing the tests
- White Box (aka CodeBased, aka Structural)
- Testers can access source code
- Tests derived from source code structure
- Tests are evaluated in terms of coverage of the source code
- Gray Box
- In between black box and white box
- A gray-box tester partially knows the internal structure (from document): data structures,
algorithms etc
Black-Box : Partition and boundary
it('correctly calculates the integer subtraction of 10 and 2', () => {
[Link](subtract(10, 2), 8);
});

- Select input data : random or structure?

- Partition input in the domain similar and test at the boundary
- Partition : Big +/- integer (2 cases), small +/- integer (2 cases)
- Boundary : 0, 1, -1
- Multiply (a,b)
- Example above have 2 inputs with each 7 possible value
- So this will produce 7 × 7 = 49 partitions that completely cover the space of pairs of
integers (Full Cartesian product)
- If full Cartesian is not possible, we need to find other approaches to test them
- This approach is used in both white and black-box
White-box
- When doing white-box testing, tester must take care those implementation for
behavior that isn’t specified by the spec:
- Unnecessary code that has never been used
- Duplicated codes
- Create tests that test every single line of code
- Evaluate the structure and function calls, variable types, etc.
- For example, if the spec says “throws an exception if the input is poorly
formatted,” then test shouldn’t check specifically for a one type of exception
only.
Test scenario documentation
Sl No Summary Pre-requisite Test case Expected results.

1. Fully privileged 1)User account 1) User enters the userid 1) User is logged into the home page
user can make must exist and password 2) Edit screen is presented to the user.
account changes 2) User needs to 2) User sees edit
have the required permissions to modify 3) Account information is saved
privileges account itself 4) User is taken back to login page
3) User modifies account
information and saves.
4) User logs out.

2. Another valid 1)User account 1) User enters the userid 1) User is logged into the home page
user without full must exist and password 2) Edit screen is presented to the user only
privileges 2) User needs to 2) User sees edit on certain fields. The account fields are
have the minimum permissions to modify grayed out.
privileges only certain fields. 3) Fields modified are saved
3) User modifies only 4) User is taken back to login page
those fields and saves.
4) User logs out.
Test case
documentation
 function foo (P,Q) {
White-box: Statement if (P + Q) > 100 {
print("large");
}
- A statement is a unit of an imperative programming if (P > 50) {
language that expresses some action print("P large");
}
- Simple statement : assignment, return, operation, etc. }
- Example of one line statement: x = y+1;
- Compound statement : if-statement, for-loop, while-loop, etc
- Example of one if-statement:
If ( x > 10) { x++; y --; } //1
- Testing wants to touch all the statement of the codes X = foo(10,10);
// 2
at least once. X = foo(20,25);
// 3
X = foo(50,60);
White-box: Branch and Path
- A branch is a decision point to do one or others. The most common
statement used to branch is the "IF" statement. Example of 2 branches:

if (x>0) { y++; }

else { y--; }

- A path through a program is a sequence of statements that starts at an entry,

junction, or decision and ends at another (possible the same), junction,
decision, or exit.
- A link is a single process that lies between 2 nodes.
- The length of a path is the number of links in a path
- Complete paths is a path that starts at a routine’s entry and ends at the exit
Test coverage
- A measurement to evaluate the percentage of code tested
- Statement coverage (SC) : test case is executed in such a way that every statement of the
code is executed at least once.
- Branch coverage (BC) : Test coverage criteria requires enough test cases such that each
condition in a decision takes on all possible outcomes at least once.
- That is, every branch (decision) taken each way, true and false.
- (Number of decisions outcomes tested / Total Number of decision Outcomes) x 100 %
- Path coverage (PC) : All possible control paths taken, including all loop paths
- (Number of path tested / Total Number of all possible paths) x 100 %
- Usually, a test covering/executing more code may indicate better test quality.
Statement, Branch, and Path coverage
- a = true, b = true will give
100% statement coverage, but
not branch coverage
- In the branch coverage we
need to cover all the edges,
which we missed in the
statement coverage shown as
red lines in the above image
 How many test cases needed?
- Statement coverage: find the shortest number of
Example-1 paths following which all the nodes will be covered.
Here by traversing through path 1A-2C-3D-E-4G-5H all
the nodes are covered. so the SC = 1.
- Branch Coverage: find the minimum number of paths
Read(P); covering of all the edges. The aim is to cover all
Read(Q);
possible true/false decisions. By following paths
if (P + Q) > 100 {
print("large"); 1A-2C-3D-E-4G-5H, maximum numbers of edges (A,
} C, D, E, G and H) are covered but edges B and F are
if (P > 50) { left. To covers these edges we can follow 1A-2B-E-4F.
print("P large"); Combining the two paths BC= 2..
}
- Path Coverage: All possible paths are
1A-2B-E-4F
1A-2B-E-4G-5H
1A-2C-3D-E-4G-5H
1A-2C-3D-E-4F
Therefore, PC is 4.
SC=1, BC=2 and PC=4. Src:[Link]
s/for-white-box/statement-branch-and-path-coverage
 How many test cases needed?
Example-2

- SC: returnInput(5, true, true, true) - 100%

statement covered
- BC: Add testcase returnInput(5, false,
false, false) to cover 100% of branches
- PC: Need for example: returnInput(5, true,
false, false) and more
Measure code coverage
- A common way is to abstract program into graphs
- Graph : Usually the control flow graph (CFG), which models all executions of a program by
describing control structures
- Node Sequences of statements (basic block). For example, Basic Block is a sequence of
statements with only one entry point and only one exit point (no branches)
- Edge : Transfers of control
- Tester may draw Graph to explain the flow of code
CFG : if statement

No else
CFG : while and for Loops
CFG: break and continue
CFG: switch
Path coverage
 For a single test case, what is the coverages?

Test coverage measure

Statement Coverage VS. Branch Coverage
- Use only path coverage
- Impossible due to amount of path in codes
- If we decide a test and show that the
expected result is equal to the
program results, can we conclude that
the software has no bug?
- 100% path coverage implies bug free
code?
- 100% coverage of some aspect is
never a guarantee of bug-free
software
Coverage example
Construction a test
- Setup: Initialise some data structures.
- The tests. The actual test scenario implementation
- Teardown. Always! clean up so that other test can run cleanly.
- De-allocate memory, clean up data structure, etc
- Each test should be able to be run independently of the other tests.
- There should NOT be order the tests (test cases) will be run and not
all tests will be run every times.
- An individual test should NOT take too much time to run since it will
discourage the programmer from running individual test often
Test first approach
- Dummy functions, stubs, is a function that do nothing and return null values or
-1 are used when not all the functionality of the code are implemented to help
the test starts.
- Mock or Fake objects implement enough of an object to get the test going or
are used as part of the setup for other tests
- For test driven development, Test is implemented as a mock first and
introduce more codes until the tests pass.
- More tests can be added during the development process
Test tools
- JAVA
- Junit is a framework for java Unit test
([Link])
- Emma is a code coverage tool :
- The source code is instrumented
(source/binary)
- Log code that writes to a trace
file is inserted in every branch,
statement etc.
- When the instrumented code is
executed, the coverage info will
be written to trace file
Emma report example
JavaScript
- A manual test can be done
var string = require(‘./[Link]’);
var s = new string(“hello world”);
assert(‘h’ == [Link](0));
- Testing framework with nodejs (old)
- Chai : provide easy interface for checking statement
- Mocha : Test runner
- Istanbul : Code coverage report
- Grunt : Task runner script
Jest (modern test framework)
Jest is an open-source testing framework built on JavaScript, designed majorly to
work with React and React Native-based web applications.

$ npm install --save-dev jest

"scripts": {
$ npm install -g jest "start": "node [Link]",
"test": "jest"
},
"jest": {
"collectCoverage": true,
"coverageReporters": ["html", "text", "text-summary", "cobertura"],
[Link] "reporters": [
"default",
["jest-junit", { "outputDirectory": "test-results/jest" }]
]
},