Software testing levels
Software testing levels describe the stages of software development when
testing is conducted. Generally, there are four progressive testing levels based
on the area they focus on the software development process: unit
testing, integration testing, system testing, and user acceptance testing
(UAT).
Levels of software testing
Unit Testing
WHAT IS IT
The smallest testable part of the software system is often referred to as a unit: a
new code, a refactoring of legacy code. Unit testing is commonly performed early
in the development process by the engineers themselves, not the testing team.
WHY TO RUN IT
To ensure code is developer correctly. Therefore, this testing level is aimed at
examining every single unit of a software system in order to make sure that it
meets the original requirements and functions as expected.
PERFORMED BY
Developers or Technical architects
HOW TO DO
As soon a new code is written, automated tools with continuous integration as a
part of the build, are very efficient: JUnit, Test NG, PHP Unit.
Integration Testing
WHAT IS IT
The objective of the next testing level is to verify whether the combined units
work well together as a group.
WHY TO RUN IT
Integration testing is aimed at detecting the flaws in the interactions between the
units within a module. The aim is to make sure the interconnections between the
system components run as required.
PERFORMED BY
Developers or Technical architects
HOW TO DO
As soon a new component (new web services, controllers…) is added,
continuous integration and automated tools help: REST client, SOAP UI.
System Testing
WHAT IS IT
At this level, a complete software system is tested as a whole. System testing
includes user flows, performance and security.
WHY TO RUN IT
This stage serves to verify the product’s compliance with the functional and
technical requirements and overall quality standards.
PERFORMED BY
System testing should be performed by a highly professional testing team
(Business analyst, Product owner, Manual QA) in an environment as close to the
real business use scenario as possible.
HOW TO DO
Exploratory testing and automated tools.
User Acceptance Testing (UAT) or End-to-End tests
WHAT IS IT
This is where the product is validated against the end user requirements. It is the
last stage of the testing process that ensure customers’ expectations are met,
verifying acceptance tests on the stories and the features.
WHY TO RUN IT
This final step helps the team decide if the product is ready to be shipped or not.
While small issues should be detected and resolved earlier in the process, this
testing level focuses on overall system quality, from content and UI to
performance issues.
PERFORMED BY
The acceptance stage might be followed by an alpha and beta testing, allowing a
small number of actual users (product end end-users, Product owner,
developers) to try out the software before it is officially delivered.
HOW TO DO
Most of the time with automated tools as Cucumber or Selenium but also with
manual tests, to be centralized with integrated environnement as Tuleap tool.
Manual vs Automated Testing
At a high level, we need to make the distinction between manual and automated tests.
Manual testing is done in human, by clicking through the software or interacting with
the application and APIs. This requires one people to set up an environment and
execute the tests themselves. This can be very expensive : human error can occur as the
tester might make typos or forget one phase during the testing cycle.
On the opposite, automated tests are performed by a machine that executes a scrip
written in advance. These tests can vary a lot in complexity, from checking a single
method in a class to making sure that performing a sequence of complex actions in the
UI leads to the same results. It’s much more robust and reliable than manual tests but
automation can’t replace manual testing, for example to check visually interface or
understand an end-user journey. In addition, keep in mind that the quality of automated
testing depends on the quality of automation, in other words it depends on how well
your test scripts have been written.
In DevOps approach and tools, automated testing is a key component
with continuous integration and continuous development. This’s a great way to
accelerate your quality assurance process as you add new features, so new checks to
your application. But there’s still value in doing some manual testing with what is called
exploratory testing and end-user acceptance validation.
Manual testing pros & cons
PROS OF MANUAL TESTING
Get fast and accurate visual feedback,
Help to identify anomalies related to the look and feel of the
application,
Better to catch usability issues in the application,
While testing a small change, an automation test would require
coding which could be time-consuming. While you could test
manually on the fly.
CONS OF MANUAL TESTING
Some tests are difficult to run manually -as performance tests-,
others are almost not testable at all,
Less reliable testing method because it is conducted by humans, so
mistakes are always possible,
The manual testing process can not more difficult to record and
track,
Not possible to reuse manual tests.
Automated testing pros & cons
PROS OF AUTOMATED TESTING
Minimize the human effort required to efficiently run tests.
Automation helps you to find more bugs compare to a human tester,
Reduce cost of bugs because tests are performed up to 10 times
faster compared to manual testing,
Identify any new bug introduced as a result of the changes, by
running a set of regression checks when the application is upgraded,
Ideally the development team should fix any incidents as soon as
they are appear,
The more you automate tests, the faster is your testing process,
Automation process can be saved and tracked for software
traceability and quality assurance audits,
Automated tests can be reused and executed several times for the
same kind of operations,
Increase teams’ productivity as it provides fast testing results.
CONS OF AUTOMATED TESTING
Without human feedback, it’s harder to get insight into visual aspects
of your user, interface : button location, form sizes, colors, font, sizes,
contrast,
To get started, initial costs can be very high. Writing reliable tests
can be a hard task: too detailed scripts can become unmaintainable
and fuzzy scripts can not serve as expected,
False sense of quality : an automated check only checks what is been
programmed to check.
Basic programming knowledge is required so automated tests should
not be adapted for user acceptance tests
Test maintenance can be costly. This is usually underestimated,
Every automation tool has their limitations which reduces the scope
of automation.
Why do you need both automated and manual
testing?
Automated testing does not have to down play the role of manual testing. They
enhance one another and create more complete quality control for delivering a
better product.
Automated testing increase most of the time testing speed and consistency. But,
as mentioned before, it is only as good as the scripts you wrote. Manual testing is
complementary to the automatic process as it raise mistakes from the user’s
perspective or identify unexpected incidents from unscripted scenarios. Manual
testing still has a critical role in the QA process. There are benefits to manual
testing and automated testing. It’s important to consider both testing approaches
as you design your quality assurance strategy.
Testing in Agile Development
In Agile software development, testing typically represents an iterative process. It
runs continuously alongside the development progress. It is a collaborative effort
between testers, developers, product owners and even customers.
While testing levels seen above, generally refer to the whole software product,
they can also be applied to a new piece of feature. In this case, every small unit
of the new functionality is being verified. The QA team checks the
interconnections between these units, the way the feature integrates with the rest
of the system and if the new update is ready to be delivered.
That’s why, implementing a continuous evolving quality assurance testing policy
is a key of success when launching a new product.
Challenges when managing manual and automatic
testing
The tools to do the job : having a centralized test management environment,
integrated with one or more automated tools is the best way to keep software
quality in check. Most of the time, QA teams use different testing tools,
disconnected to each other. The key here is verify they have strong integration
for a complete testing process overview and to reach an A to Z traceability.
A centralized test management software as the dedicated module in Tuleap, can
unify automated and manual testing to drive reuse and efficiency and gain
visibility into application readiness. QA teams can manage both automated
testing with tools as Jenkins, and manual testing together to gain consistency,
coverage, and a seamless transition between them.
Keep in mind, that automation is an efficient tool if it supports your manual testing
efforts by freeing time, so that complex, exploratory and heuristic manual testing
can be focused and valuable, contributing in whole to a successful QA process.
To combine the best of the two worlds, it is needed to run in parallel automated
and manual test campaigns that reduces overall execution time and drives
efficiency. Taking a look at Tuleap, you will see how real-time reporting and
analytics provides full traceability to issues raised, associated results, and
releases progress.