Comprehensive Guide to Product Testing
Comprehensive Guide to Product Testing
Localization issues where the wrong word is used in page translations How to make sure a bug is valid
Spelling and capitalization errors such as uTEsT
Punctuation is used incorrectly in text ( , . : ; ' " ) Let’s see the factors that determine whether a bug is valid or not.
Working as Designed
4. Performance
You may think you found a bug, but actually, it isn’t a bug and is
Problematic slowness or hanging, sluggish interface. Features take simply how the system works, these are invalid bugs. At uTest, we
longer to load than they should, slow navigation in the application. classify such bugs as Working as Designed or WAD. Understanding
Examples: the product first will help you avoid such bugs. But do not worry too
much about WAD as rejections due to this reason do not have any
Application reacts slowly when navigating throughout features effect on your ratings. WAD bugs can be tricky at times, even for
Application or pages take too long to load experienced testers. Just use your best judgment. Let’s make it more
Application freezes or becomes unresponsive for more than 10 seconds clear by looking at some examples:
5. Crash Example 1: You are testing a website, and you didn’t see any sign-out
button on the site, but as per instructions, a sign-out button should be
Application quits or closes unexpectedly while using the features. present on the homepage, so you reported it. But the sign out button is
Examples: hidden inside a dropdown menu, which you didn’t check. So it turns out to
be a WAD bug. It can be good feedback to make the sign-out button easily
Web - page hangs and doesn’t respond, ultimately resulting with an error, or accessible, but it is not an issue as the developers intentionally put the
the browser closes sign-out button inside the dropdown menu.
Computer - application freezes the device, hangs for a long time or closes Example 2: You are testing a shopping website, and you sorted items
abruptly using price, low to high. There are two prices of a product, actual price, and
Mobile - applications close abruptly with an error discounted price. The site is sorting items using actual price not by
discounted price, so at first, you may think the sorting is incorrect and
report a bug. So the reported bug is invalid as the sorting is working as
What is an issue report? designed. Sorting the items using a discounted price is a good suggestion,
An issue report is a report containing all the required information to but it is not a bug.
understand, reproduce, and fix a bug.
You must always write a well-written issue report with all the required
information.
Out of Scope areas Following Instructions
The bug you are reporting must be in the scope of the cycle. Every You must always follow instructions while testing and reporting a
cycle will have details about in scope and out of scope areas in the bug. If you do not follow any instructions, even though you reported a
overview. Any bug reported outside of the scope is invalid. valid bug, it can be rejected as DNFI (Did not follow instructions).
Example: In the ‘Out Of Scope’ section of the overview, it is explicitly Example: Reporting an unclear issue report, reporting placeholder bugs,
stated that the ‘Help page’ is out of scope. So, if you report any bug within not using uTest proxy when it should be used, etc.
the help page, it will be rejected.
Note: These are the main requirements you need to fulfill to report a
Do not report criteria valid bug, depending on the cycle, there may be additional
requirements, so you must read the overview and follow the
Depending on the test cycle, it may have requirements on what type
instructions.
of bugs testers can submit. Information regarding this will also be
stated in the ‘Out Of Scope’ section of the overview. Useful course regarding when a Bug Is Not a Bug here
Example: In the ‘Out Of Scope’ section, it is stated that visual bugs & Testing a Website or an App
localization bugs are not in scope. So, if you report any visual & localization
Let's discuss how to test a website or an application. We will just talk
bugs, it will be rejected.
about the steps you should follow. More detailed information, like
Duplicate
how to report a bug, how to create attachments will be discussed in
the upcoming tracks.
Any bug can be reported only once by a tester, reporting the same
bug again will result in a rejection. Always review the existing Read the instructions
reported issues before reporting a new one.
This is the most important part. You should always read the overview
and understand the instructions before starting to test. Skipping the
Known Issues overview and not following the instructions properly may result in
Known issues are the bugs that are already known to the customer, your work getting rejected.
and reporting the same bug again will be considered as a duplicate. Here’s why it is important:
Depending on the test cycle, it may or may not have any known
issues list. You can find the known issues in the overview of the Helps you understand the product being tested
cycle or in the issues tab. So if a cycle has known issues, it is Helps you understand which kind of bugs customers are looking for
important to review them before reporting a new one. Often overview includes information about how to access the testing site
or application
Contains information about In Scope and Out Scope sections
Often overview includes known issues. Issues that are already known will
be rejected if reported again
Issue reporting instructions
Any additional important information
Understand the product
You should have an idea of what you’re testing and what it does. It
Exploratory Testing
helps you understand which areas and features of the product are
important and should be tested first. Also, in the overview section,
you will find details on focus areas. Once you have an understanding
of the product, you’re ready to start testing.
The ‘Traditional’ Approach to Testing
Use the product Before we look at Exploratory Testing, it might help if we first talk about a
different, more traditional approach to testing so we can use that as a
Open the site or install the app and start using it. As you learned in reference point to make some comparisons.
the previous courses, testing is evaluating a product to find what is
and isn’t working. Use it as real users will do, test all the features and Scripted Testing (ST) is a two-step approach to testing:
functionalities and bugs will start to appear. As a new tester, you The tests are planned, designed and written.
may find it difficult to find bugs but not lose hope and keep on The tests are executed.
testing.
These two activities are done independently of each other and in many
Report the bug cases, the person who writes the tests is different than the person who
executes them.
Once you find a bug, it’s time to report it. But first, make sure the bug
is valid. In the previous course we already discussed this. Generally, the tester executing the tests has some knowledge of the product,
Now once you’re confident that the bug is valid, it’s time to report it. or the tests include the information needed to execute them. This is
Create all the attachments required (we will teach how in the next important because without that knowledge or information, the tester might
tracks), fill out the issue report form with all details and report the not be able to execute the test or interpret its results.
bug.
Cons: Introduction
Cannot be automated, only some aspects can The software testing phase in the software development life cycle is one of
There is no strict documentation for reference the most crucial components of the quality management process. It requires
Less time to be familiar with the new app, especially for new testers that testing begin as early as possible in the development process and that it
Difficult to reproduce the bug, as sometimes testers are not able to keep track of continue all the way through implementation. The earlier bugs are found in
what they have tested development, the lower the risk and cost a company has to absorb in
The time spent testing will not be paid, but you will be paid only for your deploying their software.
approved work
Proper test case execution is essential in ensuring the success of any testing
project; therefore, focusing on best testing practices helps minimize rework
and reduce time to market for software delivery. As such, a trained,
knowledgeable, informed, and engaged tester provides the best defense
against quality and software mishaps, bringing value and operational
Misconceptions About Exploratory Testing excellence to a customer's product.
Exploratory Testing is Unprepared A common challenge for customers is to ensure appropriate breadth and
Not scripting doesn't mean unprepared. It's about enabling choice, not depth in the testing of their software applications. Testing breadth speaks to
constraining it. There can be detailed documentation in exploratory testing. coverage across the application. For example, are the new features properly
Test Cases are not Used in Exploratory Testing working, is the uninstall executable removing the correct files from the
Testers using exploratory approach can use test cases to assist their testing user's device, and are actions performed via the mobile application syncing
activity. Lack or presence of scripts does not define exploratory testing. with the web application? Testing depth speaks to exhaustive testing of a
Any testing activity where the learning from the previous test influences the particular feature. For example, is the checkout form being tested for
next test is exploratory testing to some degree. potential timing out conditions, pressing the back button on the mobile
device, and are there issues with entering an online promo code after adding
Only an Experienced Tester Can Do Exploratory Testing a product to the user's shopping cart?
The experience of a tester is not the pre-requisite for exploratory testing.
Any tester who is willing to learn and practice the skill of exploration and Since every customer utilizes a different approach or methodology toward
investigation can perform exploratory testing. Explaining and defending the testing, you will also see different types and styles of test case
testing activities is important in exploratory testing. documentation. Testers must not modify a customer's test case, but any
columns or steps must be completed completely and accurately since you do
not know how the customer will compile and analyze the data. Also, it is
essential to be familiar with the product as much as possible before
executing the test case since there are different types of testing that can be Test Case Testing Approach and Tips:
performed.
When a tester executes test cases, they should follow instructions carefully,
Review all related material provided by the customer, complete any as every test case is unique in requirements and formatting. Typically, test
required tutorials, and ensure that you clearly understand the testing case submissions come with guaranteed compensation. As a tester, test
requirements. The more a tester understands what is expected, the better cases may be more appealing than pure exploratory testing.
they will be able to execute the test case according to the customer's The key to a successful test is to follow the test case closely and follow all
specifications and, in the process, identify potential areas for testing that the steps listed. TTLs, TEs, and Customers review all test cases and will be
may not have been included in the test case. This prepares the tester for able to identify test case results that are not completed honestly, accurately,
improved ad hoc or exploratory testing. However, before investing time and and thoroughly.
effort into areas outside of the provided test cases, it's crucial to understand
Preconditions:
what is in scope and out of scope. Typically, this information is provided in
the cycle overview, slot instructions, or test case instructions. In the When you start executing the test case. First, make sure all the
unlikely event that you do not find the information you seek or if you have a preconditions are fulfilled if there are any. If this precondition is not
doubt, please feel free to contact the TTL or TE of the cycle for fulfilled, then you will not be able to execute the test case correctly.
clarification. Completing steps in order: Test cases should be run in the order specified.
Test Cases normally have dependencies on the previous test cases. For
Test Case Definition example, step 2 requires you to execute step 1 first.
Every test case has a series of steps to help the customer determine whether Verification:
a product is working correctly. Generally, most test cases contain several Verify each step of the test case very carefully. When testers execute a test
fields that either inform or require testers to complete, such as case, they should ensure that each verification point mentioned in the test
case is verified correctly as per the expected results.
Step Description
Status: If the actual result matches the expected result of the test case, then
Steps
the Status should be updated to PASS, and if it is not, then the Status should
Expected Result
be updated to FAIL.
Actual Result
Status
PASS: If the Status for a particular step is PASS, then move to the next step.
Observations/Comments columns
FAIL: If the Status for a particular step is FAIL, verify if this bug has already been
Bug ID is another field you may come across, typically at the end of the table or
reported or if it is known issue. If not, report it. After you have reported the bug,
for each test case step
continue completing the subsequent steps in the test case. If subsequent steps
depend on the failed step and there are no workarounds, ask for guidance from
If the test case is created on the uTest platform, it contains steps, expected the TTL or TE of the cycle.
results, Status, and attachments. For more information, review the What is a
Test Case? course. Observations:
Sample Test Case Observations are essential in the test case. Here the tester can highlight the
insight you gathered while completing a particular step. The information
provided serves to help the customer troubleshoot an issue or better Test Case File Name:
understand if there is any potential confusion for end users. Observations
Testers might be required to save test case files with information that helps
can be provided for both PASS and FAIL steps.
identify the test case and who completed it in some cycles. A good practice
Thorough completion of the Test Case: is to retain the customer's name for the test case or the test case name and
add your operating system and browser to the test case name, for example,
Make sure to fill in all steps when executing the test case. uTest_Registration_macOS_Safari.
If, for whatever reason, you cannot complete a certain step, contact the TTL or Additionally, if multiple testers are participating in a test cycle with the
the TE of the cycle for assistance. same OS and Browser, adding each tester's last name may be helpful.
Pay close attention when executing every step of the test case. Finally, be sure to follow each Test Cycle's instructions on how to complete
and execute test cases.
Bugs Reporting:
Exploratory Testing:
As mentioned above, under a failed step, if you encounter a bug, first try to
reproduce it and understand the root cause. Then verify if that particular bug Exploratory tests may be run along with test cases. Therefore, you can
has already been reported in the cycle. Duplicate bugs should not be perform exploratory testing when completing/executing test cases. In this
reported, so make sure to review the previously submitted bugs and the case, ensure you conduct adequate exploratory testing by testing on depth
known issues list, if there are any, before you submit your bug to avoid against the in scope focus areas or in scope areas. You will typically be
getting a rejection. When submitting a bug report, remember to include required to complete the test case before performing extensive exploratory
step-by-step details about what the tester did for easy reproduction, testing. Thus, please complete and submit the test case before further
including the required attachments. exploratory testing. Learn more about Exploratory in the Exploratory
Testing course.
Each failed step should have a bug attached to it. Enter the Bug ID in the
Bug ID column; For example, Bug ID #451178 for step 2. Refrain from
assuming that the person reviewing your observations/comments or
reporting is familiar with the product. Note: This course specifically addresses how to complete test cases and
It is best practice to provide observations and comments as though your includes a few scenarios that are often included in test cases. Also, this
audience is not a user of the application you are testing. This ensures that course only serves as background information, so please follow the
any recipient can reproduce what you have experienced and assist in getting instructions on completing and executing test cases for every Test Cycle.
your bugs approved. Remember, clear communication on observations and
steps taken when experiencing a bug is key to resolving issues found during
testing. Refer to The Bug Reports course for information on bug reporting.
Root cause: This analysis method requires additional instructions on how
to do it effectively and determine if what the user is experiencing is a root
cause of an issue.
In most of the uTest cycles, testers will be asked to download an APK file,
Installing Apps on Mobile which they need to install on their Android device. Follow these steps to
install the APK file:
Devices 1. Open the browser and navigate to the test cycle overview.
2. Download the APK file from the Attachments section.
Android Devices 3. Once the download is complete, open the file.
4. By default Android does not allow users to install unknown apps, so a
Installing the testing app from the Google Play pop-up will display; tap on the Settings button.
Store 5. Enable the toggle for Allow from this source and go back.
6. Tap on the Install button and wait until the app finish installing.
In some of the uTest cycles, testers will need to install the app directly from
the Google Play store to test it. Follow these steps to install the app directly Alternatively, you can click on the QR code and then scan it with your
from the Google Play store: Mobile device, this will download the APK file to your Mobile device
directly and then follow the above steps to install it on your device.
1. Open the test cycle overview on an Android device and scroll down to
the Attachments section. iOS Devices
2. Tap on the Google Play store link.
3. Tap on the Install button when the Google Play store opens.
4. Verify that the app version matches the version mentioned in the Installing the testing app from the App Store
overview after installing the testing app. In some of the uTest cycles, testers will need to install the testing app
o Open Settings, navigate to the Apps page and open the installed app to directly from the App Store. Follow these steps to install the testing app
view the app version. from the App Store:
Note: Testers may face some problems when trying to install the app from 1. Open the test cycle overview on an iOS device and scroll down to
the Google Play store. Some commons problems are: the Attachments section.
2. Tap on the link to download the app.
The app is not available in a specific country. 3. Tap on the Install button when the App Store opens.
The device is not compatible with the app. 4. Verify that the app version matches the version mentioned in the
The version of the installed app is not matching with the version mentioned in the overview after installation.
overview. 5. Open Settings, tap on General and then select iPhone Storage. Select the
app you installed to view its version.
Testers should reach out to the TTL via the test cycle chat if they face such
problems. Installing the testing app from the Test Flight
Installing an APK file Often customers share the testing version of the app through TestFlight.
Follow the below steps to install an app from the TestFlight:
1. From an iOS device, open the App Store and download 3. Copy the downloaded IPA file from the file explorer by pressing Ctrl+C on
the TestFlight app. Windows or Command+C on macOS.
2. Open the test cycle overview and tap on the TestFlight link to open the 4. Connect the iOS device with the computer and trust the computer.
testing application inside TestFlight. 5. Open iTunes, make sure the iOS device is visible, right-click on the iOS
3. Make sure select the correct version and install the app. device name and select paste.
o macOS 10.15 and above users: Open the Apple Configurator 2 app and
Installing an IPA file then drag and drop the IPA file over the device and confirm the
installation.
In most of the uTest cycle testers are required to install an IPA file. Most of 6. Wait until the process finishes.
the cases testers can install the IPA directly from the iOS device by 7. Once the installation is finished, open Settings on the iOS device, navigate
following these steps: to General and open VPN and Device Management (iOS 15 and
newer) OR Profiles and Device Management (Older iOS versions).
1. Open the test cycle overview on an iOS device and scroll down to 8. Select the installed app and trust it; now, the testing app can be opened.
the Attachments section.
2. Find the button named Download on Mobile Device. Note: Make sure to select Library section on iTunes software then try to
o Alternatively, you can click on the Open Scanable QR code and then scan paste the copied IPA file in order to be able to install it on your Mobile
it with your iOS Mobile device. device.
o Note, that some test cycles may not have these buttons, in such cases
testers need to use iTunes to install the app. Installing an IPA file using iMazing
3. Tap on Install and wait until the installation is finished. Testers can also use iMazing to install the .IPA file when the Download on
4. Once the installation is finished, open Settings on the iOS device, navigate Mobile Device button is not present in the cycle overview. Follow these
to General and open VPN and Device Management (iOS 15 and steps:
newer) OR Profiles and Device Management (Older iOS versions).
5. Select the installed app and trust it; now the testing app can be opened. 1. Download and install iMazing on your computer from here.
2. Open the Test Cycle overview on a computer and download the IPA from
Installing an IPA file using iTunes the Attachments section.
Testers need to use iTunes to install the .ipa file when the Download on 3. Open iMazing on your computer.
Mobile Device button is not present in the overview of the cycle. Follow 4. Connect the iOS device to the computer and trust the computer.
these steps: 5. On iMazing click on Manage Apps.
6. Click on the drop-down arrow at the bottom right.
1. Download and install iTunes for Windows from here. 7. Click on Install .IPA File.
o Below macOS 10.15 users: iTunes is installed by default and you can 8. Select the downloaded .IPA file and click on open.
follow the steps. 9. Wait until the process finishes.
o macOS 10.15 and above users: Instead of iTunes, testers need to install 10. Once the installation is finished, open Settings on the iOS device, navigate
and open the Apple Configurator 2 app and drag and drop the IPA file to General and open VPN and Device Management (iOS 15 and
2. Open the Test Cycle overview on a computer and download the IPA from newer) OR Profiles and Device Management (Older iOS versions).
the Attachments section. 11. Select the installed app and trust it; now, the testing app can be opened.
Drop-down menu/list: A menu that offers a list of options. The user can
choose one value from the list.
Common Website Elements Radio Button: An icon representing one of a set of options, only one of
and Pages which can be selected at any time.
Tabs: A navigation element that allows multiple views to be contained with
Elements a single window.
Let's learn about the different elements which are present on a webpage. Footer: An area located at the bottom of every page on a website, below the
Use these terms when writing a bug report to describe it clearly. main body content.
Hero: An oversized banner image at the top of a website. The hero image is Note: We just discussed a few common elements which testers will interact
often the first visual element a visitor encounters on the site; it presents an with. Many other elements can be present on a website or an app; always try
overview of its most important content. to use the correct term when referring to them.
Pages
Banner: A digital advertisement displayed in the header, footer, or sidebars
Now, let's learn about a few common pages most of the websites have. Try
of websites.
to use the correct page name when reporting a bug.
Carousel: A slideshow for cycling through a series of images or any visual
Homepage: The main page of a website; it is the first page that the users
content.
see when they load the website URL. Like [Link] will
Navigation: Allows users to navigate one section of the website to another. open the homepage of uTest.
Hamburger Menu: Used to save space on the screen and denotes a hidden Product List Page (PLP): The page where the products offered by the
menu. website are listed; this page often contains features like sort and filter.
Like [Link] will open a Product List.
Breadcrumb: A type of navigation that reveals the user's location on a
website. Product Detail Page (PDP): The page with information about a specific
product. Like [Link] will open a
Pop-up: A small window that pops up over the top of web pages.
Product Detail Page
Toggle Button: Allows the user to change a setting between two states.
Cart: The page where users can pile up what they want to buy from the
Checkbox: A small interactive box that can be toggled by the user to website. Like [Link] will open a Cart Page
indicate a positive or negative choice.
About us: The about us page is commonly used by all types of businesses
Textbox/Text Field: A section or object on a page that allows a user to to give customers more insight into who is involved with a given business
enter text. and exactly what it does. Like [Link] will open
the About us page for uTest.
Note: We discussed only a few pages, which are very common across Reporting a quality bug is very important. Avoid things like:
websites. Most of the pages are different across websites. Always try to use
the correct name when referring to a page. Spelling & grammar mistakes
Never report placeholder issues, this is against the uTest Guidelines
Not providing enough information or attachments