>>> Data Validation and Program Testing
Data Validation and Program Testing
Section 1: Data Validation and Verification
It is crucial that data is reliable and trustworthy for as long as the data is needed (data integrity).
“Garbage in, garbage out”. This refers to the idea that poor-quality input data leads to flawed output.
Data validation and verification are critical steps to maintain data integrity. Let’s define the terms.
• Data integrity refers to the consistency, accuracy and reliability of data
throughout its lifecycle.
• Data validation: check that data entered
satisfies predefined rules or format before
accepting it.
Its purpose is to reject unreasonable data
input, so that only valid data is processed
or stored in the system.
Note that it does NOT guarantee that the data is accurate.
Validation methods include:
presence check, length check, range check, format check
• Data verification: confirm that data
received by the system is as intended or
matches the original source.
Verification methods include:
1) double data entry (e.g. keying password twice),
2) visual check
- manually comparing entered data against source
- or through a message / pop-up that asks the user to Double entry
confirm the data entered before further processing
DO NOT CONFUSE DATA VALIDATION WITH VERIFICATION!
ASRJC H2 Computing 1
>>> Data Validation and Program Testing
1.1. Data validation
Users often make mistakes when entering data into a system. For example, some might have typed one less
digit for the phone number entry, or they might miss out the ‘@’ symbol for email input. If data validation is
absent, the company may find these people uncontactable. This is undesirable.
Data validation techniques are important for ensuring that data entered into a system satisfies certain
requirements before the data is stored or processed further.
Of course, note that no validation technique can check that the entries are completely correct; for example,
these techniques won’t be able to ensure that the age of the person who signed up for a Google account is
real.
There are many types of validation to ensure data entered is sensible. In the syllabus, you must know
Validation Type Description Examples
Presence check Check that an entry field is not left • Required user input is entered into the
blank textbox
Format check Check that the data is in a specified • Email is provided in a right format, e.g.
format or pattern user@[Link]
• Date is in the form YYYY-MM-DD
Length check Check that a specific number of • Singapore postal code having exactly 6
characters or digits has been entered digits
• Password having at least 8 characters
Range check Check that the data is between a • A probability should be a real number
lower and an upper limit between 0 and 1
• Age of a person should be between 0 and
120 (realistic age range)
• Promo marks should be between 0 and
100
Data Type check Check that the data entered matches • Number of students in a class is entered
the expected data type as a whole number, not text or float
• Price is entered as a numerical value, not
alphabetical characters
Existence check Check whether the data entered • Username is not already registered in the
already exists in the system database when creating a new account
• Student ID exists in records before
allowing exam registration
Check digit An additional digit appended to a • The last digit of a barcode is calculated
numerical code calculated from all from all the numbers before it
the other digits in the code to detect • The last digit of credit card number
errors.
Let’s talk about check digit, and we use barcode as an example.
ASRJC H2 Computing 2
>>> Data Validation and Program Testing
Check digit in barcodes (using modulo-10 system)
A barcode is an optical representation of data that can be easily scanned. It can be
used to store a number such as a product code, or ISBN number of a book. The main
type of barcode used in Singapore is the EAN-13 barcode. These are used on all retail
products, except books and magazines that use ISBN and ISSN barcodes.
It has 13 digits, and the last digit is calculated based on the previous 12 digits.
The last digit is known as the check digit. It is used to ensure that the barcode is scanned correctly.
Let’s calculate the check digit of the barcode using the first 12 digits, as follows:
1. Sum all the digits in even positions and multiply 3.
2. Add all the digits in odd positions (excluding check digit) to the number obtained
in step 1.
3. Divide the number in step 2 by 10 and take the remainder.
4. If the remainder is non-zero, subtract the number from 10.
EAN-13 check digit
For this barcode with first 12 digits: 070564244194,
1. Digits in even positions are 7, 5, 3, 4, 1, 4. Sum = 24. Multiplied by 3 = 72.
2. Digits in odd positions are 0, 0, 6, 2, 4, 9. Sum = 21. Add to previous number = 93.
3. Divide by 10 gives remainder of 3.
4. It is non-zero, so the check digit is 10 – 3 = 7.
Question: Should EAN-13 barcode number be stored as an integer or a string? Explain.
Check digit in credit card numbers (using Luhn Algorithm)
It uses the Luhn Algorithm. Let’s use the number 4532 0151 2345 6789 as an
example. The last digit, 9, is the check digit in this case.
Step 1: Starting from the rightmost digit (check digit), move left and
double every second digit:
• Original: 4 5 3 2 0 1 5 1 2 3 4 5 6 7 8 9
• Double every 2nd digit from right: 8 5 6 2 0 1 10 1 4 3 8 5 12 7 16 9
Step 2: If doubling results in a two-digit number, add those two digits together:
10 → 1 + 0 = 1, 12 → 1 + 2 = 3, 16 → 1 + 6 = 7
• Result: 8 5 6 2 0 1 1 1 4 3 8 5 3 7 7 9
Step 3: Add all the digits together:
• 8 + 5 + 6 + 2 + 0 + 1 + 1 + 1 + 4 + 3 + 8 + 5 + 3 + 7 + 7 + 9 = 70
Step 4: Check if the total is divisible by 10:
• 70 MOD 10 = 0. So the card number is considered valid. ✓
ASRJC H2 Computing 3
>>> Data Validation and Program Testing
Check digit is crucial for credit card validation because:
• There may be typos when user manually enters the card number during online purchases. Studies show
that the Luhn algorithm catches about 98% of single-digit errors.
• It allows websites and payment systems to immediately verify if a card number is potentially valid
before even contacting the bank, saving time and reducing unnecessary processing, avoiding fees
associated with processing invalid payment attempts.
Other applications of check digits
There are of course many other contexts where check digits are used:
• ISBN (to validate book identification numbers). Uses mod 11 algorithm.
• Credit card numbers. Uses Luhn algorithm.
• Vehicle identification numbers.
• IMEI (international mobile equipment identity) number. Uses Luhn algorithm.
For Singapore NRICs, the last letter can be determined by the previous characters. The last letter of Singapore
car plate number is also determined by its previous characters.
About Check Digits
As you have seen from the examples above, a check digit is an additional digit appended to a numerical
code that is calculated from all the other digits in the code using a specific mathematical formula or
algorithm. It serves as a form of redundancy check to detect errors during data entry or transmission.
You must remember that the check digit utilises ALL the digits in the code to generate a single verification
digit. This is important because:
• Every digit in the original code contributes to the calculation
• Changing any single digit will result in a different check digit
Check digits are specifically designed to identify two main types of input errors:
1. Transcription Errors (single digit errors)
A transcription error occurs when a single digit is incorrectly copied or recorded as a different digit.
Example 1: User’s postal code is supposed to be 541123, but it is wrongly transcribed
as 541173.
Note how the digit ‘2’ is incorrect.
Example 2: A customer service agent listens to the customer’s order request, and
writes down an order quantity of “6” when the customer has said “8” over the phone.
2. Transposition Errors (swapping adjacent digits)
A transposition error occurs when two adjacent digits are accidentally swapped
in their positions.
Example: When typing quickly, a user enters “953821” when he intended to enter
“958321”.
ASRJC H2 Computing 4
>>> Data Validation and Program Testing
Importance of weights in check digit algorithms
Weights are specific multipliers assigned to each digit position in the code. Different positions are multiplied by
different weight values before being summed together.
• Without weights, simple addition can’t detect transposition errors, as the sum remains the same.
• Weights make the position of each digit significant in the calculation.
Each weighting system is optimised to detect the most common types of errors for that particular application.
Effectiveness of check digits
• Check digits are extremely effective at catching transcription errors where a single digit is mistyped.
When any single digit changes, the weighted sum changes, making the check digit invalid. Many
modern check digit algorithms (like Luhn) catch virtually all single-digit transcription errors.
• Check digit algorithms are good (~90%) but not perfect at catching transposition errors. This is because
some specific digit swaps can produce the same weighted sum. This means that the error may go
undetected.
For Luhn algorithm, suppose we swap position of 0 and 9:
Original: 09XX → (0×2) + (9×1) = 0 + 9 = 9
Swapped: 90XX → (9×2) + (0×1) = 18 → 1+8 = 9 (same result)
Since both calculations yield 9, this particular transposition is not detected.
1.2. Data verification
Verification of data entry
When data is entered into a system, verification means getting the user to confirm that the data entered was
as intended or matches the original source.
Here are some common methods for data verification:
• Visual check (or proofreading)
Visual check involves comparing the data entered into the system with the original source data (which
could be a digital document or paper records), to ensure that they are the same.
Another form of visual check is the use of a confirmation prompt. The data entered is displayed back to
the user through a preview page or a pop-up, and asking him to confirm before further processing. This
approach is commonly seen in scenarios like confirming bank transfer amount, verifying shipping addresses
or double-checking important form submissions.
• Double entry verification
Requires a user to input the same information twice, and the values are compared.
• Cross-field verification
Validating across different fields, e.g. checking if age entered is consistent with birthdate.
ASRJC H2 Computing 5
>>> Data Validation and Program Testing
Verification during data transfer
It is possible for data to be corrupted during transmission. Some techniques for data verification:
• Using checksum algorithms (CRC, MD5, SHA-1)
Generate a fixed-size value (checksum) based on the transmitted data. The sender computes the checksum
and sends along with the data. The receiver recomputes the checksum upon receiving the data and verifies
against the transmitted checksum. Mismatch indicates potential data corruption. A small change in the data
can result in a vastly different checksum, making it suitable for error detection.
If you use 7-zip, you can use it to generate these checksums of files. For example, these are the values
obtained for one of the lecture notes documents:
o CRC64: 90bb9cfb67461dec
o MD5: e1a51a10cff8da62e0a73018e5e8c680
o SHA1: 2700e9611d09aa5ce83d2329bf1708b338c643b2
Remark: These simple checksums are NOT designed for security purposes. It can be manipulated or
forged by an attacker. So use them to detect accidental errors or file corruption in transmission, not for
cryptography.
• Using cryptographic hash functions
An example would be SHA-256. These can detect both accidental errors and intentional tampering.
With that same lecture notes document above, the SHA256 hash is:
o dac85d7e6099a93821ea19771b3f40abc38f512a5da80473241a309f1bb739f0
(notice how much longer it is compared to simple checksums)
Remark: Hash functions can be used for applications beyond data integrity verification. They can be used
for digital signatures, blockchain and cryptographic operations.
• One-bit parity check
This is a simple error detection method that adds a single extra bit (called a parity bit) to a group of data
bits. There are two types of parity: even and odd. If the parity is even, you must make sure that the total
number of 1s in the data (including parity bit) is even. Likewise, if the parity is odd, the total number of 1s
(including parity bit) has to be odd.
Suppose the parity is decided to be even.
With data bits of 1010101, the number of 1s present is 4. It is already even. So the parity bit is 0.
The data sent to the receiver will be 10101010.
If the next group of data bits is 1111111, the number of 1s is 7. So the parity bit has to be 1.
The data sent to the receiver will be 11111111.
The recipient of the data will check if the parity rule is satisfied. If the rule is violated, an error is detected.
Limitations of one-bit parity check
One-bit parity check can detect single-bit (or odd number of bit) errors.
However, it cannot detect two-bit (or even number of bit) errors, since the parity remains the same.
Also, it cannot be used for error correction.
ASRJC H2 Computing 6
>>> Data Validation and Program Testing
Having seen the limitations of one-bit parity check, we now look at how a parity block might be a more robust
error detection system.
Parity Block (2D Parity)
Here, we arrange data in a 2D grid and calculate parity bits for both rows and columns.
Let’s suppose we are using odd parity. Then the row and column parities are as follows (highlighted).
1 0 1 1 1 0 0 1
1 1 0 0 0 1 1 1
0 1 1 0 0 0 1 0
1 0 0 1 1 1 1 0
1 0 1 0 1 0 0 0
1 1 0 1 0 1 0 1
The data to be sent is:
10111001 11000111 01100010 10011110 10101000 11010101
Exercise: Circle the location of the single-bit error present in this parity block (even parity).
1 1 1 0 1 0 1 1
1 1 0 0 0 1 1 0
0 0 1 0 1 1 0 1
1 1 0 1 1 0 1 0
0 1 1 1 1 1 1 0
1 0 0 0 0 1 0 0
Advantages of Parity Block
• Has better error detection rate than simple parity
• Can detect some multi-bit errors (errors in different rows and columns)
• Can locate the position of single-bit error (row and column intersection), and allows error correction by
flipping that identified bit.
ASRJC H2 Computing 7
>>> Data Validation and Program Testing
1.3. Syntax, Runtime and Logic errors
Syntax errors occur when the code violates the rules of the programming language.
print("Hello, world" # Missing closing parenthesis
Runtime errors (also known as Exceptions) occur during the execution of the program. These errors can be
caused by factors such as division by zero, accessing an index out of range, or using an undefined variable.
def divide(a, b):
return a / b
result = divide(10, 0) # Attempting to divide by zero
print("Result:", result)
Logic errors occur when the code is syntactically correct, but it does not produce the intended results due to a
mistake in the logic or algorithm.
A simple example would be using a wrong formula in the code when computing a result.
These errors can be challenging to spot, because the code may still run without any obvious syntax or runtime
errors, but it produces incorrect results due to incorrect logic.
Example 1
Suppose an amateur programmer wrote this function to calculate the product of odd integers present in a
given list of integers.
def product_of_odd(int_list):
list_length = len(int_list)
product = 1
for index in range(1, list_length): # iterate from first to last num
num = int_list[index]
if isinstance(num, int) and num % 2 == 1:
product = product * num
return product
# Test case
int_list = [2, 5, 1, 6, 3, 4, 7]
print(product_of_odd(int_list))
print(5 * 1 * 3 * 7)
He also tested his function with a list above, and it showed a correct output of 105. He’s happy.
But a fellow programmer had problems with his code. Can you identify two issues?
Ans: It skipped the first element in the number list. Also, if there are no odd numbers, the function
returns 1, but that should not be the case.
ASRJC H2 Computing 8
>>> Data Validation and Program Testing
Example 2
Look at the following code. Identify the logic error and rewrite the line that needs correction.
def binary_to_denary(binary_str):
denary_val = 0
for bit in binary_str:
denary_val = denary_val / 2 + int(bit)
return denary_val
Ans: Should multiply by 2 instead of divide.
Example 3
Identify any logic errors below.
def factorial(n):
result = 1
if n <= 1:
return result
for i in range(1, n):
result *= i
return result
Ans: Factorial is undefined for negative or non-integer numbers. Also, it should be range(1, n+1).
Example 4
The following function is supposed to check if an integer n is prime. We know that we only need to test
whether the given number n is divisible by numbers up to and including √𝑛.
Identify a logic error here and suggest the slight modification to make it correct.
import math
def is_prime(n):
if n < 2:
return False
for i in range(2, int([Link](n))):
if n % i == 0:
return False
return True
Ans: it should be int([Link](n))+1. To optimise, exclude even numbers above 2, as they are not prime.
ASRJC H2 Computing 9
>>> Data Validation and Program Testing
1.4. Test Cases: Normal, Extreme, Abnormal
There are three main types of tests:
Normal Typical data values that are valid. Will be accepted by the system.
(Valid)
Boundary These are the values at the limits of validity (extreme ends of the range of normal
(Extreme) data).
• Include values just inside the boundary that should be accepted
• and include values just outside the boundary that should be rejected.
Abnormal Data that falls outside what is acceptable and should be rejected
(Erroneous)
For instance, we have written a function to calculate factorial of an integer number.
def factorial(n):
if not isinstance(n, int) or n < 0:
raise ValueError("Input should be a non-negative integer")
if n == 0 or n == 1:
return 1
result = 1
for i in range(1, n + 1):
result *= i
return result
Normal test case: print(factorial(5))
Extreme test case: print(factorial(0)) [boundary of what is acceptable]
Abnormal test cases: print(factorial(-1)) [can also be seen as an extreme test case]
print(factorial(1.2345)) [incorrect data type]
print(factorial(“abc”))
ASRJC H2 Computing 10
>>> Data Validation and Program Testing
1.5. Identifying and Handling Programming Errors
Type of error How are they identified? How to avoid?
Syntax error Detected before the program runs, during Use an IDE with syntax highlighting and
(code violates the the compilation process. error detection.
grammatical rules of
the programming The compiler displays error messages Write code incrementally – test small
language) with line numbers. The program will not sections frequently.
run until the errors are fixed.
Runtime error Only occurs during execution. It causes Validate input before processing. Verify
(Code encounters a the program to crash or terminate data types.
problem during abnormally.
execution and the Check for existence before accessing.
program crashes) An exception message with traceback
would be shown when that happens. Use exception handling
(try-except blocks for Python)
Logic error Not detected automatically by compiler. Plan before coding.
(Program runs Program runs without crashing. No error
successfully without messages displayed. Write incremental code.
crashing, but
produces incorrect Detected through Review algorithms, ensuring the correct
results. HARDEST to • manual testing (running program and formula is written and check loop
detect because the compare actual output with expected boundaries for off-by-one errors.
program appears to output),
work normally.) • using test cases, Use multiple test cases and test
• debugging with print statements, boundary values.
• using trace tables to trace the logic,
• have another person review code.
ASRJC H2 Computing 11
>>> Data Validation and Program Testing
1.6. Writing code for Data Validation
Example 1
Below shows an example of age validation in Python. The user will be prompted to enter an age and will be
reprompted until a valid one is supplied.
Insights you can glean from this example:
• The core principle here is to identify and reject invalid input as early as possible.
• Next, notice the strategic sequence of checks. We start from the most basic check to the most specific
one. Presence check is the easiest one. Data type check is next, as we need to ensure that conversion to
integer can work. Finally, range check is the most specific validation here.
• Use a while loop with flag to keep re-prompting the user until a valid input is obtained.
• Write clear comments for each validation check and give targeted, meaningful feedback to user when an
invalid input is received.
ASRJC H2 Computing 12
>>> Data Validation and Program Testing
Example 2
Below shows an example of password validation. The password supplied must:
• Be at least 8 characters long
• Contain at least one uppercase letter and one lowercase letter
• Contain at least one digit
• Not contain any spaces
Again, think about why this specific sequence of checks:
ASRJC H2 Computing 13
>>> Data Validation and Program Testing
Question: What data validation checks do you think would be needed for username input (for the purpose
of account registration)? Try to write a sample Python code to demonstrate these validations.
ASRJC H2 Computing 14
>>> Data Validation and Program Testing
Example 3
Write a function validate_input() that takes in an input string and validates that it is 10 comma-separated
distinct integers between 0 and 100 (both inclusive), and each consecutive pair differs by at least 2 (so numbers
like 34 and 35 cannot be adjacent). It should return True if valid, and False if invalid.
ASRJC H2 Computing 15