STUDENT GUIDE

FACULTY OF INFORMATION TECHNOLOGY

DATA SCIENCE 500

Page | 1
FACULTY OF INFORMATION TECHNOLOGY

STUDY GUIDE
MODULE: DATA SCIENCE 500

Copyright © 2025

Richfield Graduate Institute of

Technology (Pty) Ltd Registration
Number: 2000/000757/07

All rights reserved; no part of this publication may be reproduced in any form or by any means,
including photocopying machines, without the written permission of the Institution

Page | 2
1 Introduction to Data Science ........................................................................ 7
1.1 What is Data Science? ..................................................................................... 7
1.2 Why Data Science? ......................................................................................... 7
1.3 The Data Science Venn Diagram ....................................................................... 8
1.4 Introduction to Python ................................................................................... 10

2 Types of Data.............................................................................................. 16
2.1 What is Data? ............................................................................................... 16
2.2 Structured Data vs Unstructured Data ............................................................ 16
2.3 Quantitative Data vs Qualitative Data ............................................................. 17
2.4 Four Levels of Data ........................................................................................ 17
2.5 Descriptive Statistics using NumPy ................................................................ 18
2.6 Introduction to Pandas .................................................................................. 21

3 Five Steps of Data Science .......................................................................... 25

3.1 Step 1: Asking an Interesting Question ............................................................ 25
3.2 Step 2: Obtaining the Data ............................................................................. 26
3.3 Step 3: Exploring the Data .............................................................................. 26
3.4 Step 4: Modelling the Data ............................................................................. 34
3.5 Step 5: Communicating and Visualizing Results .............................................. 34

4 Data Visualisation using Matplotlib ............................................................ 37

4.1 Introduction to Matplotlib .............................................................................. 37
4.2 Key Concepts in Matplotlib ............................................................................ 37
4.3 Basic Steps to Create a Plot using Matplotlib ................................................... 38

5 Introduction to Probability ......................................................................... 44

5.1 What is Probability?....................................................................................... 44
5.2 Bayesian versus Frequentist .......................................................................... 46
5.3 The law of large numbers ............................................................................... 48
5.4 Rules of Probability ....................................................................................... 50

6 Introduction to Machine Learning .............................................................. 59

6.1 What is Machine Learning? ............................................................................ 59

Page | 3
 6.2 How does Machine Learning work? ................................................................. 60
6.3 Types of Machine Learning ............................................................................. 61
6.4 Supervised Learning ...................................................................................... 62
6.5 Unsupervised Learning .................................................................................. 72
6.6 Reinforcement Learning ................................................................................ 75

7 Introduction to Web Scraping using BeautifulSoup ..................................... 77

7.1 What is Beautiful Soup? ................................................................................. 77
7.2 Installation of Required Libraries .................................................................... 77
7.3 Using Beautiful Soup for Web Scraping .......................................................... 78
7.4 Common Operations ..................................................................................... 79
7.5 Best Practices and Considerations ................................................................. 81
7.6 Case Study – What’s in a job description? ....................................................... 81

Page | 4
The Information Technology (IT) qualification at Richfield College is a dynamic and future-focused
program designed to equip students with advanced technical, analytical, and problem-solving
skills. At the core of the qualification is a commitment to academic excellence, industry alignment,
and innovation, fostering graduates who are proficient in addressing modern technological
challenges. This qualification strategically integrates theoretical knowledge with practical
applications, preparing students for various roles in the IT sector. The IT program is structured to
address the growing complexity of the evolving technological landscape.

The Higher certificate in Information Technology (HCIT) program is a foundational stepping-stone

for students who wish to pursue further studies or enter the workforce. Graduates of this program
are well-prepared to articulate to the Diploma in IT (DIT) or the Bachelor of Science in IT (BSc IT)
qualifications, providing a seamless transition for those seeking to deepen their knowledge and
skills in specialized IT areas. Additionally, the program equips students with the essential
competencies for entry-level IT roles such as IT Support Technicians, Junior Web/ System
Developers, IT Administrators, etc.

The Diploma in Information Technology (DIT) is a comprehensive and practical program designed
to build a strong foundation in IT principles while equipping students with the hands-on skills
required to meet industry demands. Focused on both theoretical knowledge and applied
learning, this qualification prepares students for intermediate-level roles in IT and serves as a
stepping-stone for further academic progression or specialization. Graduates of this program are
well-prepared to articulate to the Bachelor of Science in IT (BSc IT) qualification. The curriculum
covers programming, networking, database management, system analysis etc., ensuring
graduates possess the competencies to solve real-world IT challenges effectively.

The Bachelor of Science in IT (BSc IT) program is structured to address the growing complexity of
the evolving technological landscape. Through carefully curated modules, students gain a deep
understanding of software development, database management, cloud computing,
cybersecurity, IT management, artificial intelligence, machine learning, networking etc.
Graduates of this program are well-prepared to articulate to the Bachelor of Science Honours in
IT qualification. The curriculum is designed to bridge the gap between academic learning and real-

Page | 5
world applications, thus fostering innovation and an entrepreneurial mindset. Students are
encouraged to participate in research and practical learning.

The curriculum is designed to bridge the gap between academic learning and real-world
applications, thus fostering innovation and an entrepreneurial mindset. Students are encouraged
to participate in research and practical learning.

The programming focus within the IT qualification exemplifies academic innovation and
professional alignment. By integrating a diverse range of programming languages with practical
application, the curriculum prepares students to excel in the rapidly evolving tech industry. The
program aligns with industry courses from globally recognized leading tech giants, such as Oracle,
AWS, IBM, etc ensures that graduates possess the credentials to validate their expertise in
software development and cloud-based technologies. This blend of foundational knowledge,
practical experience, and industry-standard courses prepares students for immediate
employment and establishes a strong basis for long-term career advancement in software
development.

Data Science 500 is a foundational module that introduces students to the core principles and
practices of data science, forming the entry point into the Emerging Technologies route. The
module explores the role of data science in solving real-world problems and guides students
through the five essential steps of the data science process—data collection, preparation,
exploration, modeling, and interpretation. Students gain a solid understanding of data types, and
are introduced to key tools and libraries such as Pandas for data manipulation and basic
visualization.

Through practical exercises and real-world examples, students develop the skills needed to work
with structured data and generate meaningful insights. This module lays the groundwork for more
advanced subjects such as machine learning, big data, AI, and autonomous systems, making it an
essential starting point for students pursuing a career in data-driven innovation within the
emerging technologies space.

Page | 6
1 Introduction to Data Science

Learning Objectives:

By the end of this section, students should be able to:

• Define the key concepts of data science, its importance, and how it differs from
traditional analysis
• Explain the need for data science in modern data handling, including current
challenges and limitations of traditional methods
• Identify and explain the three core components of data science

1.1 What is Data Science?

Data Science is an interdisciplinary field that involves extracting valuable insights and knowledge
from data. It combines techniques from various domains such as statistics, computer science,
mathematics, and domain expertise to make data-driven decisions and predictions.

With the increasing availability of data in today's digital world, Data Science has become a critical
tool for businesses, researchers, and organizations to gain a competitive advantage, improve
processes, and solve complex problems.

1.2 Why Data Science?

In this age of information abundance, we're confronted with an unprecedented volume of data
that raises an important question: Why do we need an entirely new vocabulary and field of study
to handle it? The answer lies in the limitations of traditional analysis methods. Today's data
volume has grown so massive that humans can't process it manually within any reasonable
timeframe. The challenge is compounded by the diverse nature of data collection – information
comes from various sources and often arrives highly unorganized. This raw data frequently
presents multiple challenges: it can be incomplete, inaccurate, or presented in vastly different
scales making comparisons difficult.

Page | 7
Consider, for example, a streaming service like Netflix analysing viewer behaviour. They need to
process millions of data points including what shows people watch, when they pause, which
episodes make them binge-watch, what thumbnails they click on, and how ratings correlate with
viewing time – all while accounting for different time zones, devices, and user demographics. The
complexity increases when you consider that this data constantly updates in real time, with new
users joining, viewing patterns changing with seasons, and content libraries expanding daily.

Once we properly clean and organize this data (a process that constitutes a significant portion of
data science work), previously hidden patterns emerge clearly from millions of rows of
information. We might discover that viewers who enjoy cooking shows are also likely to watch
travel documentaries on Sunday evenings, or that sci-fi fans tend to binge-watch series in shorter
time spans than drama viewers. This reveals one of data science's primary objectives: to establish
explicit practices and procedures to uncover and utilize these relationships within data. The field
has emerged not to replace traditional analysis but to address the unique challenges of our data-
rich world.

1.3 The Data Science Venn Diagram

The Data Science Venn diagram (Figure 1: Data Science Venn Diagram Figure 1), often referred to
as Drew Conway's Data Science Venn diagram, is a fundamental visualization used to explain the
interdisciplinary nature of data science. It serves as both an educational tool and a framework for
understanding the skills required to become a proficient data scientist. The diagram helps
organizations structure their data science teams, guides curriculum development in educational
institutions, and assists aspiring data scientists in planning their learning journey. The diagram
consists of three overlapping circles, each representing a core competency.

Page | 8
 Figure 1: Data Science Venn Diagram

Hacking Skills: encompasses programming abilities, coding proficiency, and software engineering
knowledge. This includes expertise in languages like Python or R, database management, and the
ability to manipulate and process large datasets efficiently. These technical skills are essential for
handling the practical aspects of data analysis.

Math & Statistics Knowledge: covering areas like statistical analysis, probability theory, and
mathematical modelling. This component is crucial for understanding data patterns, making valid
inferences, and developing robust predictive models. It includes concepts from basic statistics to
advanced machine learning algorithms, ensuring that analyses are mathematically sound and
reliable.

Substantive Expertise: refers to domain knowledge or subject matter expertise in a specific field.
This could be business understanding, scientific knowledge, or industry-specific insights. This
expertise is vital for asking relevant questions, understanding context, and interpreting results in
meaningful ways that add value to the field being studied.

The intersections of these circles reveal important combinations of skills. Where Hacking Skills
meet Math & Statistics, we find Machine Learning, representing the technical implementation of
statistical methods.

The intersection of Math & Statistics and Substantive Expertise creates Traditional Research,
where classical analytical approaches are applied to domain-specific problems. The "Danger
Page | 9
Zone" occurs where Hacking Skills meet Substantive Expertise without statistical rigour,
potentially leading to technically impressive but statistically unsound analyses.

At the centre of all three circles lies Data Science itself - the ideal combination of all these skills.
This sweet spot represents the ability to write code, apply statistical methods, and understand
domain context simultaneously. This intersection produces the most valuable insights and
solutions, making it the target state for aspiring data scientists and data science teams.

1.4 Introduction to Python

Python is a versatile, high-level programming language favoured in data science for its easy
syntax, extensive libraries, robust community support, and seamless integration with various
platforms. It is used for data manipulation, statistical analysis, machine learning, data
visualisation, and automating data pipelines.

To install Python, you can use the standard installer from [Link] or the Anaconda distribution
from [Link]. While the standard version includes just the core interpreter, Anaconda
comes with Python, key data science libraries, and tools like Jupyter Notebook, making it ideal for
beginners.

1.4.1 Python Keywords

Keywords are reserved words in Python that have special meaning. You cannot use them as
variable names.

Figure 2: Python Keywords

Page | 10
1.4.2 Python Basics
[Link] Naming Conventions
In Python, variable names must be descriptive, start with a letter or underscore, include letters,
numbers, and underscores, and are case-sensitive, distinguishing between different cases like Age
and age. In Python, you do not need to declare the data type of a variable. It is inferred
automatically as can be seen in the below code example.

[Link] Python Comments

A comment is any line that begins with the # symbol. Comments are not executed by the Python
interpreter and are used only for human understanding. They are helpful in explaining what a
specific part of the code does, making the code easier to read and maintain. For example, before
assigning a value to a variable, you can use a comment to describe the purpose of that variable.
Comments are especially useful when working on complex code or when multiple people are
collaborating on the same project, as they make the logic behind the code clearer. They can also
be used to temporarily disable lines of code during testing or debugging.

#Declaring a variable called name

name = "Alice" # string

age = 25 # integer

height = 1.65 # float

is_student = True # Boolean

1.4.3 Control Flow

Control flow allows you to direct the order in which Python executes lines of code. These include
the loops and conditional statements.

[Link] Conditional Statements

These are used to execute code based on certain conditions. They are if, elif, and else statements.
The below are examples of conditional statements:

temperature = 25

#If temperature is greater than 30, show “It's hot outside!”

Page | 11
if temperature > 30:

print("It's hot outside!")

#Or else if temperature is greater than 20, show “It's hot outside!”

elif temperature > 20:

print("Nice and warm.")

#Or else if temperature is not greater than 30 nor 20, show “It's cold”

else:

print("It's cold!")

[Link] Loops
Loops are used to execute a block of code repeatedly. Python has two types of loops, for loops
and while loops. For loops are used to iterate over a sequence (like a list, tuple, or string). The
below is an example of a loop being used to iterate over a list:

fruits = ['apple', 'banana', 'cherry']

for fruit in fruits:

print(fruit)

The while loop is used to execute a block of code for as long as a certain condition is met. The
below example shows a while loop used to print a count variable for as long as the count variable
is less than 3:

count = 0

while count < 3:

print("Counting:", count)

count += 1

1.4.4 Data Structures

Python provides built-in data structures that are essential for storing and manipulating collections
of data.

Page | 12
[Link] Lists
A list is an ordered, mutable collection of items. It can contain different data types. Key concepts
include indexing, slicing, appending, and looping through elements of different data types. The
below code shows an example of list:

#Create a list called fruits

fruits = ['apple', 'banana', 'cherry']

print(fruits[1]) # Prints second element in the list using indexing

print(fruits[0:2]) # Prints elements at index 0 to 2 using slicing

[Link]('date') #Appends date as a new value in the list

for fruit in fruits: #Loops through all elements in the list

print(fruit)

[Link] Tuples
A tuple is like a list, but immutable (cannot be changed after creation). It is useful when you want
to protect the data from modification. An example of a tuple is shown below:

#Create a tuple called dimensions

dimensions = (1920, 1080)

print(dimensions[0]) # 1920

# dimensions[0] = 1280 # This will raise an error (tuple is immutable)

[Link] Dictionaries
A dictionary stores data in key-value pairs. It is unordered (in versions <3.7), and values are
accessed using keys. The below code shows how to declare, and access elements of a dictionary.

#Creating a dictionary named student

student = {'name': 'Moss', 'age': 21, 'grade': 'A'}

print(student['name']) # Access Moss through the key name

print([Link]('age')) # Access 21 through the key age

student['age'] = 22 # Update value through the key age

student['course'] = 'DS' # Add new key-value pair

Page | 13
print(student)

for key, value in [Link]():

print(key, ":", value)

[Link] Sets
A set is an unordered collection of unique items. Sets are useful for removing duplicates and
performing set operations like union and intersection.

#Create two sets a and b

a = {1, 2, 3, 4}

b = {3, 4, 5, 6}

print([Link](b)) # Combine set a with set b. Output: {1, 2, 3, 4, 5, 6}

print([Link](b)) # Show common value of set a and b. Output: {3, 4}

[Link](7) #add a new value 7

print(a) # Output: {1, 2, 3, 4, 7}

1.4.5 Functions
A function is a reusable block of code that performs a specific task. Functions can take inputs
(parameters) and return results. The below code is an example of a function with parameters and
returns results:

#Create a function called add

def add_numbers(a, b):

return a + b

result = add_numbers(3, 5)

print(result) # 8

Chapter Summary

This chapter introduced the fundamental concepts of data science and its critical role in modern
data analysis. We explored how data science differs from traditional analytical methods by
combining advanced computational techniques with statistical analysis and domain expertise.

Page | 14
The chapter emphasised the growing need for data science approaches due to the increasing
complexity and volume of modern data, highlighting how traditional methods often fall short in
handling these challenges. We concluded by examining the three essential pillars of data science:
mathematics/statistics, computer science/programming, and domain knowledge/business
understanding, which work together to transform raw data into meaningful insights. As part of
this foundation, we introduced Python as the primary programming language for implementing
data science techniques. Python’s simplicity, readability, and vast ecosystem of data science
libraries make it an ideal tool for exploring, analysing, and visualizing data.

Review Questions

1. What are the three main components of the Data Science Venn Diagram?
2. True or False: Data Science is primarily focused on programming and coding skills.
3. List two key reasons why organizations implement data science solutions.
4. A pharmaceutical company is using machine learning algorithms to analyse patient data and
predict drug interactions, while its business team uses the insights to make strategic decisions.
Which component of the Data Science Venn Diagram does this scenario best demonstrate?
5. Compare and contrast "data science" and "data analytics." What are the key differences in
terms of their focus and approach?
6. What are two key reasons why Python is widely used in data science?
7. Examine the following code and answer the question below:

student = {'name': 'Moss', 'age': 21, 'grade': 'A'}

student['age'] = 22

Which Python data structure is being used, and how does the code demonstrate one of its
key features?

Page | 15
2 Types of Data

Learning Objectives:

By the end of this section, students should be able to:

• Explain the concept of data and identify different types of data with real-world
examples
• Differentiate between structured vs unstructured data and quantitative vs qualitative
data types
• Apply basic descriptive statistics using NumPy to analyse and interpret data, including
understanding the four levels of measurement (Nominal, Ordinal, Interval, Ratio)

2.1 What is Data?

2.2 Structured Data vs Unstructured Data

Structured data refers to data that is organized and formatted in a specific way, usually in a tabular
format with rows and columns. Each data entry has a well-defined data type and can be easily
stored in databases or spreadsheets. Examples of structured data include data from relational
databases, CSV files, Excel sheets, and tables (Chohan, 2019).

Characteristics of structured data:

• Easy to query and analyse using standard methods and statistical techniques
• Well-defined schema, meaning the data's structure is pre-defined, and each data element
has a clear meaning.
• Ideal for traditional data analysis techniques, like SQL queries and statistical analysis.

Unstructured data, on the other hand, lacks a predefined data model or structure. It does not
conform to a fixed schema and is typically not easily organized into rows and columns.
Unstructured data is often in the form of text-heavy documents, images, videos, audio files, social
media posts, emails, and other free-form content.

Page | 16
Characteristics of unstructured data:

• More challenging to process and analyse compared to structured data due to its lack of
predefined structure.
• Requires specialized tools and techniques, such as Natural Language Processing (NLP) and
Machine Learning, to extract insights and patterns.
• Often contains valuable information, but accessing and utilizing it effectively can be
complex.

As a Data Scientist, practitioners often encounter both structured and unstructured data. The
ability to handle both types of data and integrate them effectively is essential to uncovering
comprehensive insights and solutions to real-world problems (Chohan, 2019)

2.3 Quantitative Data vs Qualitative Data

Quantitative data: This data can be described as numbers, and basic mathematical procedures,
including addition, are possible on the set. It represents quantities or amounts and is typically
collected through standardized methods such as surveys, experiments, or direct measurements
(Chohan, 2019).

Qualitative data: This data cannot be described using numbers and basic mathematics. This data
is generally thought of as being described using "natural" categories and language. In other
words, it is more descriptive information that describes the qualifies or characteristics of an entity
rather than being measured in numbers. It is collected through methods such as interviews,
observations, or open-ended surveys and provides a deeper understanding of the underlying
reasons, opinions, behaviours, or emotions of individuals or groups (Chohan, 2019).

2.4 Four Levels of Data

Data can be broken down into one of four levels which are:

The nominal level - The first level of data, the nominal level, consists of data that is described
purely by name or category. Basic examples include gender, nationality, species, or yeast strain in
a beer. They are not described by numbers and are therefore qualitative (Sinan, 2016).

Page | 17
The ordinal level - The nominal level did not provide us with much flexibility in terms of
mathematical operations due to one seemingly unimportant fact—we could not order the
observations in any natural way. Data in the ordinal level provides us with a rank order, or the
means to place one observation before the other; however, it does not provide us with relative
differences between observations, meaning that while we may order the observations from first
to last, we cannot add or subtract them to get any real meaning (Sinan, 2016).

The interval level - Now we are getting somewhere interesting. At the interval level, we are
beginning to look at data that can be expressed through very quantifiable means, and where
much more complicated mathematical formulas are allowed. The basic difference between the
ordinal level and the interval level is, well, just that—difference. Data at the interval level allows
meaningful subtraction between data points (Sinan, 2016).

The ratio level - Finally, we will look at the ratio level. After moving through three different levels
with differing levels of allowed mathematical operations, the ratio level proves to be the strongest
of the four. Not only can we define order and difference, but the ratio level also allows us to
multiply and divide as well. This might seem like not much to make a fuss over, but it changes
almost everything about how we view data at this level (Sinan, 2016).

2.5 Descriptive Statistics using NumPy

2.5.1 What are Descriptive Statistics?

Descriptive statistics are fundamental tools used to summarize and describe the main features of
a dataset clearly and understandably. They help us make sense of large amounts of data by
reducing complex information into simple numerical or visual representations. These statistics fall
into two main categories: measures of central tendency (like mean, median, and mode) which
tell us about the typical or average values in our data, and measures of spread (like range,
variance, and standard deviation) which tell us how dispersed the data is.

Page | 18
2.5.2 What is NumPy?
NumPy (Numerical Python) is a fundamental package for scientific computing in Python. It
provides support for large, multi-dimensional arrays and matrices, along with an extensive
collection of high-level mathematical functions to operate on these arrays efficiently. NumPy
forms the foundation for many other data science and machine learning libraries in the Python
ecosystem.

Key Features of NumPy

1. N-dimensional Array (ndarray): NumPy's primary data structure is the ndarray, which is a
multi-dimensional array that can hold elements of the same data type. This allows efficient
storage and manipulation of large datasets.
2. Broadcasting: NumPy supports broadcasting, which enables element-wise operations
between arrays of different shapes, making array operations more convenient and less
memory-consuming.
3. Mathematical Functions: NumPy provides a wide range of mathematical functions, including
basic arithmetic operations, trigonometric functions, exponential functions, statistical
functions, linear algebra operations, and more.
4. Indexing and Slicing: NumPy offers powerful indexing and slicing capabilities, like Python lists,
but with additional features for multi-dimensional arrays.
5. Random Number Generation: NumPy has a subpackage called [Link] that allows
generating various types of random numbers and random samples from distributions.
6. Interoperability: NumPy arrays can interact with other libraries like SciPy, pandas, and scikit-
learn, enhancing the overall capabilities of Python for scientific computing.

Page | 19
Example 1:

Imagine you have conducted a survey among your employees asking, "How happy are you to be
working here on a scale from 1-5", and your results are: 5, 4, 3, 4, 5, 3, 2, 5, 3, 2, 1, 4, 5, 3, 4, 4, 5,
4, 2, 1, 4, 5, 4, 3, 2, 4, 4, 5, 4, 3, 2, 1
import numpy
results = [5, 4, 3, 4, 5, 3, 2, 5, 3, 2, 1, 4, 5, 3, 4, 4, 5, 4, 2, 1, 4, 5, 4, 3,

2, 4, 4, 5, 4, 3, 2, 1]

sorted_results = sorted(results)

print(sorted_results)

'''
[1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5,

5, 5, 5, 5]

'''

print [Link](results) # 3.4375

print [Link](results) # 4.0

Example 2:

Suppose we look at the temperature of a fridge containing a pharmaceutical company's new

vaccine. We measure the temperature every hour with the following data points (in Fahrenheit):
31, 32, 32, 31, 28, 29, 31, 38, 32, 31, 30, 29, 30, 31, 26.

import numpy
temps = [31, 32, 32, 31, 28, 29, 31, 38, 32, 31, 30, 29, 30, 31, 26]

print([Link](temps)) # 30.73
print([Link](temps)) # 31.0

Page | 20
Example 3:

Standard deviation is the most common measure of variation of data at the interval level and
beyond. The standard deviation can be thought of as the "average distance a data point is at from
the mean". In this example we determine the Standard deviation of the temperature dataset used
in the previous example:

2.6 Introduction to Pandas

Pandas is a powerful open-source library for data manipulation, analysis, and cleaning in Python.
It provides data structures and functions that make working with structured data, such as CSV
files or SQL tables, efficient and intuitive. Pandas is built on top of NumPy and is an essential tool
in the data science and data analysis workflows in Python.

Page | 21
2.6.1 Key Features of Pandas
1. DataFrame: The DataFrame is the central data structure in Pandas. It is a two-dimensional,
labelled data structure with columns of potentially different data types. It is like a spreadsheet
or SQL table, allowing easy data manipulation.
2. Series: A Series is a one-dimensional labelled array that can hold data of any data type. A
DataFrame consists of multiple Series, one for each column.
3. Data Alignment: Pandas aligns data based on row and column labels, making it easy to
perform operations on datasets with different shapes.
4. Missing Data Handling: Pandas provides robust tools to handle missing data, either by
removing or replacing them.
5. Data I/O: Pandas supports reading and writing data from various file formats, including CSV,
Excel, SQL databases, and more.
6. Data Aggregation: Pandas offers flexible and powerful group-by functionality, enabling data
aggregation and summarisation.
7. Time Series: Pandas has excellent support for working with time series data, including date
range generation, resampling, and time zone handling.

2.6.2 Basic Usage of Pandas

Importing Pandas:

To use Pandas, you need to import it first. It is conventionally imported as pd for simplicity.

import numpy as np

Creating a DataFrame:

You can create a DataFrame from various data sources, such as lists, dictionaries, NumPy arrays,
or by reading data from files

Page | 22
# Creating a DataFrame from a dictionary
data = {'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35],
'City': ['New York', 'San Francisco', 'Los Angeles']}

df = [Link](data)

# Creating a DataFrame from a CSV file

df = pd.read_csv('[Link]')

Chapter Summary

This chapter focused on the fundamental importance of understanding data types and their
impact on analytical approaches. We explored how proper data classification must precede
analysis, as it determines which analytical methods are appropriate and possible. The chapter
outlined three essential questions for initial data assessment: whether the data is organized or
unorganized, if variables are quantitative or qualitative, and the measurement level of each
variable (nominal, ordinal, interval, or ratio). We discussed how these classifications influence
subsequent analytical choices, from visualization methods to modelling approaches, and
introduced the concept of data type conversion for enhanced analytical insights.

Review Questions

1. Fill in the blanks: Data can be divided into _______ and _______ based on its format, and into
_______ and _______ based on its characteristics.
2. Name the four levels of measurement data in order from lowest to highest.
3. True or False: Unstructured data typically follows a predefined model or format.
4. Which of these is an example of ordinal data? Explain why.
5. Identify the type of data (structured/unstructured) and level of measurement for each:
a. Email subject lines
b. Temperature in Celsius
c. Employee ID numbers
d. Customer rankings of products
Page | 23
6. What is the main difference between measures of central tendency and measures of
dispersion in descriptive statistics?
7. Name three key features of NumPy that make it useful for data analysis.
8. For a dataset containing students' grades (0-100), heights (cm), and favourite colours, classify
each variable by its level of measurement and type (qualitative/quantitative).
9. Given the following NumPy array of test scores: 85, 92, 78, 65, 88, 95, 74, 89, 70, 85 Calculate:
a. The mean, median, and standard deviation
b. How many students scored above 80
c. The percentile rank of 89
10. Using NumPy, create an array of monthly sales data and:
a. Find the months with sales above the yearly average
b. Calculate the quarter-over-quarter growth rate
c. Identify the peak sales month and its value

Page | 24
3 Five Steps of Data Science

Learning Objectives:

By the end of this section, students should be able to:

• Explain the five fundamental steps of the data science process

• Describe the purpose and importance of each step in the data science workflow
• Identify the key considerations when exploring a new dataset
• Recognize the common challenges in working with real-world data, such as missing
values and data transformation needs
• Import, read, and handle various data formats using pandas, demonstrating
proficiency in basic data loading and saving operations
• Clean and preprocess data by handling missing values, removing duplicates, and
managing outliers
• Transform and analyse data through filtering, sorting, grouping, and applying
aggregate functions
• Combine and reshape datasets using merge, join, and pivot operations
• Apply best practices in pandas for efficient data manipulation, including
optimisation techniques and proper documentation

The practice of data science follows a systematic approach that begins with curiosity and ends
with actionable insights. This process can be broken down into five essential steps, each building
upon the previous one.

3.1 Step 1: Asking an Interesting Question

At its core, data science begins with curiosity. The quality of our question often determines the
value of our final insights, making this initial step crucial for the entire process.

Page | 25
3.2 Step 2: Obtaining the Data
When gathering data to answer our chosen question, creativity becomes essential. Data can come
from numerous sources - public databases, surveys, sensors, or even social media - and each
source brings its unique perspective to our analysis.

3.3 Step 3: Exploring the Data

The exploration phase is where we uncover the hidden insights within our data. Using Python
libraries like pandas and matplotlib, we examine our data not just as numbers, but as patterns
and relationships that tell a meaningful story.

As we delve into data exploration, several fundamental questions guide our investigation. First,
we examine the data's structure - is it organized in a clear row-and-column format? This basic yet
crucial assessment determines our next steps. We then identify what each row represents, which
might be individual transactions, customer profiles, or periods. Understanding each column's
nature - whether it contains quantitative measurements, qualitative descriptions, or categorical
information - helps us choose appropriate analytical techniques.

Real-world data often comes with imperfections. Missing data points, inconsistencies, and
unusual values all require careful consideration. As data scientists, we must make informed
decisions about how to handle these challenges. Sometimes this means filling in gaps with
estimated values; other times, it means excluding problematic data points altogether.

Many datasets require transformation before they're suitable for advanced analysis. For instance,
categorical data like "red," "blue," and "green" might need conversion into numerical values for
statistical modelling. Text data might need standardization or encoding. Python provides robust
tools for these transformations, allowing us to prepare our data for deeper analysis.

Page | 26
 3.3.1 Exploring the Data using Pandas:
# Displaying the first few rows of the DataFrame

print([Link]())

# Getting basic statistics about the DataFrame

print([Link]())

# Accessing columns and rows

age_column = df['Age']

row_0 = [Link][0]

[Link] Data Manipulation:

# Adding a new column

df['Salary'] = [50000, 60000, 70000]

# Filtering rows

filtered_df = df[df['Age'] > 30]

# Grouping and Aggregation

grouped_df = [Link]('City').mean()

[Link] Handling Missing Data:

# Checking for missing values

print([Link]())

# Dropping rows with missing values

df_cleaned = [Link]()

# Filling missing values with a specific value

df_filled = [Link](0)

Page | 27
[Link] Data I/O:
# Writing DataFrame to a CSV file

df.to_csv('[Link]', index=False)

# Reading data from an Excel file

df_excel = pd.read_excel('[Link]')

Example:

The first dataset we will look at is a public dataset made available by the restaurant review site,
Yelp. All personally identifiable information has been removed. Let's read the data first, as shown
here:

import pandas as pd

yelp_raw_data = pd.read_csv("[Link]")

yelp_raw_data.head()

The preceding code does:

• Import the pandas package and name it as pd.

• Read in the .csv from the Web - call is yelp_raw_data.
• You can download a similar CSV file from [Link]
• Look at the head of the data (just the first few rows).

Page | 28
Let’s ask some questions about the data:

a. Is the data organised or not?

We can conclude that this data seems organised because we have a nice row/column
structure.
b. What does each row represent?
Each row represents a user giving a review of a business. The next thing we should do is to
examine each row and label it by the type of data it contains. At this point, we can also use
Python to figure out just how big our dataset is. We can use the shape quality of a DataFrame
to find this out, as shown:
yelp_raw_data.shape

Output # (10000,10)

# (10000, 10) tells us that the dataset has 10000 records and 10 fields/columns. [Link] file will
yield different results or in this case shape. Observe the table shown to confirm that we have 10
columns/Fields. Using the information learnt in the previous chapter try and identify what level
of data each column represents. Additionally, determine which column(s) can be regarded as
quantitative and qualitative data.

c. Are there any missing data points?

Perform an isnull operation. For example, if your DataFrame is called awesome_dataframe, try
the Python command awesome_dataframe.isnull().sum(), which will show the number of missing
values in each column.

d. Do we need to perform any transformations on the columns?

Before starting, let's go over some quick terminology for pandas, the Python data exploration
module.

Page | 29
Dataframes - When we read in a dataset, Pandas creates a custom object called Dataframe. Think
of this as the Python version of a spreadsheet (but way better). In this case, the variable,
yelp_raw_data, is a Dataframe. To check whether this is true in Python, type in the following code:

type(yelp_raw_data)

Output #[Link]

Dataframes are two-dimensional, meaning that they are organized in a row/column structure just
as a spreadsheet is. The main benefit of using Dataframes over, say, a spreadsheet software would
be that a Dataframe can handle much larger data than most common spreadsheet software. If
you are familiar with the R language, you might recognize the word Dataframe (Sinan, 2016).

As most of the data that we will deal with is organized, Dataframes are likely the most used object
in pandas, second only to the Series object.

The Series object is simply a DataFrame, but only with one dimension. Essentially, it is a list of
data points. Each column of a DataFrame is a Series object. The first thing we need to do is grab
a single column from our DataFrame; we generally use what is known as bracket notation. The
following is an example:

yelp_raw_data['business_id'] # grabs a single column of the DataFrame

Now we use the type of function to determine if it is a Series:

type(yelp_raw_data['business_id'])

Output # [Link]

3.3.2 Exploration tips for qualitative data

Using these two Pandas objects, let's start performing some preliminary data exploration. For
qualitative data, we will specifically look at the nominal and ordinal levels.

Page | 30
Nominal level columns

As we are at the nominal level, let's recall that at this level, data is qualitative and is described
purely by name. In this dataset, this refers to the business_id, review_id, text, type, and user_id.
Let's use Pandas to investigate. Execute the following:

yelp_raw_data['business_id'].describe()

Output: # count 10000

# unique 4174

# top JokKtdXU7zXHcr20Lrk29A

# freq 37

The describe function will give us some quick stats about the column whose name we enter the
quotation marks. Note how Pandas automatically recognized that business_id was a qualitative
column and gave us stats that make sense. When describe is called on a qualitative column, we
will always get the following four items (Sinan, 2016):

• count: How many values are filled in

• unique: How many unique values are filled in
• top: The name of the most common item in the dataset
• freq: How often does the most common item appear in the dataset

Explanation: For the business_id column, we have a count of 10000. Don't be fooled though! This
does not mean that we have 10,000 businesses being reviewed here. It just means that of the
10,000 rows of reviews, the business_id column is filled in all 10,000 times. The next qualifier,
unique, tells us that we have 4174 unique businesses being reviewed in this dataset. The most
reviewed business is business JokKtdXU7zXHcr20Lrk29A, which was reviewed 37 times.

Try this now for another column such as review_id:

yelp_raw_data['review_id'].describe()

Page | 31
3.3.3 Filtering in Pandas
Filtering rows based on criteria is quite easy in Pandas. In a DataFrame, if we wish to filter out
rows based on some search criteria, we will need to go row by row and check whether a row
satisfies that condition. Pandas handle this by passing in a Series of Trues and Falses (Booleans).
Try the following code to achieve this:
duplicate_text = yelp_raw_data['text'].describe()['top']

Explanation: This code snippet finds the most frequently used piece of text in the 'text' column
of a DataFrame called yelp_raw_data. The value stored in duplicate_text is the most frequently
used piece of text in the 'text' column.

text_is_the_duplicate = yelp_raw_data['text'] == duplicate_text

Explanation: This code snippet compares the value of the 'text' column from the yelp_raw_data
data frame to the value of the variable duplicate_text. The result of the comparison is then stored
in the variable text_is_the_duplicate, which will be a Boolean (True or False) indicating if the text
in yelp_raw_data['text'] is the same as the duplicate_text.

type(text_is_the_duplicate) # it is a Series of Trues and Falses

text_is_the_duplicate.head() # shows a few Falses out of the Series

Explanation: The code snippet is checking the type of the variable 'text_is_the_duplicate', which
is likely to be a data type in Python known as a Series. The code then uses the head() function to
display the first few values in the Series, which are False. This suggests that the Series contains a
list of Booleans (True/False) values and that the first few values are False.

sum(text_is_the_duplicate) # == 2

Explanation: The code snippet is referring to a Boolean variable called "text_is_the_duplicate".

This variable is likely a list of values, either 0s or 1s, that indicate whether a given text is a duplicate
or not. The code asks the program to count the number of times the value 1 appears in the list,

Page | 32
which would indicate the number of text duplicates present. In this case, the output of the code
snippet would be 2, meaning there are two duplicates.

# the filtered Dataframe

filtered_dataframe = yelp_raw_data[text_is_the_duplicate]

Output: filtered_dataframe

Here we can see that the user_id, gave the same comment and star rating for two different
business_ids on the same day.

Ordinal level columns

As far as ordinal columns go, we are looking at dates and stars. For each of these columns, let's
look at what the describe method returns (Sinan, 2016):

#yelp_raw_data['stars'].describe()
count 10000.000000

# mean 3.777500

# std 1.214636

# min 1.000000

# 25% 3.000000

# 50% 4.000000

# 75% 5.000000

# max 5.000000

Page | 33
yelp_raw_data['stars'].value_counts()

# 4 3526

# 5 3337

# 3 1461

# 2 927

# 1 749

Explanation: The value_counts method will return the distribution of values for any column. In this
case, we see that the star rating 4 is the most common, with 3526 values, followed closely by the
rating 5.

3.4 Step 4: Modelling the Data

After thorough exploration, we move into the modelling phase, where we apply statistical and
machine-learning techniques to test hypotheses and uncover patterns. These models need
careful validation to ensure their reliability and effectiveness. This step involves choosing
appropriate algorithms, and training models, and evaluating their performance using
mathematical validation metrics.

3.5 Step 5: Communicating and Visualizing Results

The final step - communication - transforms our technical findings into clear, actionable insights.
Despite seeming straightforward, this step often proves challenging. The key lies in translating
complex analytical results into compelling visualizations and narratives that resonate with our
audience, whether they're technical experts or business stakeholders.

Throughout this process, Python serves as our primary tool, offering powerful libraries and
functions that enable us to handle each step with precision and efficiency. By following this
systematic approach and asking the right questions at each stage, we can transform raw data into
valuable insights that drive decision-making.

Page | 34
Chapter Summary

This chapter detailed the systematic approach to conducting data science projects through five
essential steps: data collection, data cleaning, data exploration, modelling, and communication.
We examined how each step serves a crucial function in the data science workflow, from
gathering relevant data sources to presenting actionable insights. The chapter emphasized the
importance of thorough data exploration techniques, including statistical summaries, distribution
analysis, and relationship identification between variables. Special attention was given to real-
world data challenges, highlighting common issues like missing values, inconsistent formats, and
the need for data transformation. Understanding these fundamentals provides the foundation for
handling complex data science projects effectively. The chapter demonstrated key data cleaning
procedures, including strategies for handling missing values, duplicate records, and outlier
detection. We examined powerful data transformation capabilities through filtering, sorting, and
aggregation functions, along with methods for combining datasets through merge and join
operations. Special attention was given to data reshaping techniques like pivot operations and
best practices for optimising Pandas operations for larger datasets.

Review Questions

1. List the five steps of data science in the correct order.

2. True or False: Data cleaning can be skipped if the data comes from a reliable source.
3. What key activities should be performed in the data cleaning step? List at least three.
4. Match each problem with its appropriate data science step:
a. Missing values in customer records
b. Presenting results to executives
c. Understanding business requirements
d. Running statistical tests
e. Collecting survey responses
5. Your data analysis reveals unexpected patterns. At which step should you revisit to ensure
your findings are valid?

Page | 35
6. What are the two primary data structures in Pandas? Explain when you would use each one.
7. Name three key features that make Pandas suitable for data manipulation.
8. Fill in the blanks: A Pandas DataFrame index must be _______, while column names must be
_______.
9. What is the difference between .loc and .iloc in Pandas? When would you use each?
10. True or False: In Pandas, you can have duplicate column names in a DataFrame.
11. How do you handle missing values in Pandas? List three different methods.
12. Write code to:
a. Read a CSV file named '[Link]'
b. Display the first 5 rows
c. Show basic information about the DataFrame
13. What is the difference between [Link]() and df.drop_duplicates()? What parameters would
you use with each?
14. In a DataFrame with columns 'Product', 'Category', and 'Price', write code to:
a. Calculate the average price by category
b. Find the most expensive product in each category
c. Count the number of products in each category

Page | 36
4 Data Visualisation using Matplotlib

Learning Objectives:

By the end of this section, students should be able to:

• Construct fundamental data visualizations using matplotlib by selecting and

implementing appropriate plotting functions for line graphs, bar charts, and scatter
plots
• Apply and demonstrate graph customization techniques by modifying visual elements
such as colours, labels, legends, and axes settings

4.1 Introduction to Matplotlib

Matplotlib is an open-source data visualisation library that was originally inspired by MATLAB's
plotting capabilities. It allows users to create various types of plots, such as line plots, scatter
plots, bar plots, histograms, pie charts, 3D plots, and more. Matplotlib provides both a simple
procedural interface (like MATLAB) and an object-oriented interface for creating plots.

4.2 Key Concepts in Matplotlib

a. Figure: A Figure in Matplotlib is the top-level container that holds all plot elements. It
represents the entire window or page where one or more plots can be drawn. A figure can
contain multiple subplots.
b. Axes: An Axes is the actual plot area within a Figure. It includes the data space, ticks, labels,
and other plot elements. Within a figure, you can have one or more Axes, arranged in a grid-
like structure.
c. Subplot: Subplots are used to create multiple plots within a single figure. They allow you to
organize multiple plots in rows and columns. The subplot() function or add_subplot() method
is used to create subplots.

Page | 37
d. Plotting Functions: Matplotlib provides various functions for creating different types of plots.
For example, plot() for line plots, scatter() for scatter plots, bar() for bar plots, hist() for
histograms, etc.
e. Customization: Matplotlib allows you to customize almost every aspect of your plots,
including colours, markers, line styles, labels, titles, legends, and more

4.3 Basic Steps to Create a Plot using Matplotlib

Import Matplotlib: To start using Matplotlib, import it first. Commonly, it is imported as plt for
convenience.

1. Create Data: Prepare the data you want to visualize.

2. Create a Figure and Axes: Use [Link]() to create a new figure and fig.add_subplot() or
[Link]() to add one or more axes (subplots) to the figure.
3. Plot Data: Use appropriate plotting functions (e.g., plot(),scatter(), etc.) to plot your data on
the axes.
4. Customize the Plot: Add labels, titles, legends, change colours, adjust axis limits, etc.
5. Show or Save the Plot: Use [Link]() to display the plot or [Link]() to save the plot as an
image file.

To learn more about matplotlib visit: Matplotlib — Visualization with Python

Example - Simple Line Plot using Matplotlib

In this example, we create a simple line plot with five data points (x, y). We use the plot() function
to plot the data, and set_xlabel(), set_ylabel(), and set_title() to add labels and title to the plot.
The legend() function is used to display the legend. Finally, we call [Link]() to display the plot.

Page | 38
import [Link] as plt

# Sample data

x = [1, 2, 3, 4, 5]

y = [2, 4, 6, 8, 10]

# Create a new figure and axis

fig, ax = [Link]()

# Plot the data

[Link](x, y, marker='o', linestyle='-', color='b', label='Line Plot')

# Customize the plot

ax.set_xlabel('X-axis')
ax.set_ylabel('Y-axis')
ax.set_title('Simple Line Plot')
[Link]()

# Show the plot

[Link]()

Page | 39
Example - Simple Vertical Bar Chart using Matplotlib

In this example, we create a simple vertical bar chart showing sales data for different products.
We use the bar() function to plot the data, and set_xlabel(), set_ylabel(), and set_title() to add
labels and title to the plot. We also customize the colors and add value labels on top of each bar.
Finally, we call [Link]() to display the plot.

import [Link] as plt

# Step 1: Create data

categories = ['Product A', 'Product B', 'Product C', 'Product D', 'Product E']

values = [23, 45, 56, 78, 32]

# Step 2: Create figure and axes

fig, ax = [Link](figsize=(10, 6))

# Step 3: Create bar chart

bars = [Link](categories, values, color=['#FF6B6B', '#4ECDC4', '#45B7D1',

'#96CEB4', '#FECA57'])

# Step 4: Customize the plot

ax.set_xlabel('Products', fontsize=12)

ax.set_ylabel('Sales (in thousands)', fontsize=12)

ax.set_title('Product Sales Comparison', fontsize=16, fontweight='bold')

# Add value labels on top of bars

for bar in bars:

height = bar.get_height()

[Link](bar.get_x() + bar.get_width()/2., height + 1,

f'{height}k', ha='center', va='bottom', fontsize=10)

# Rotate x-axis labels if needed

[Link](rotation=45, ha='right')

# Add grid for better readability

[Link](axis='y', alpha=0.3, linestyle='--')

# Adjust layout to prevent label cutoff

plt.tight_layout()

# Step 5: Show the plot

[Link]()

Page | 40
Matplotlib offers much more flexibility and complexity for creating sophisticated visualizations. It
allows you to fine-tune the appearance of your plots to best convey the information you want to
present. Additionally, it integrates well with other libraries like NumPy and Pandas, making it a
powerful tool for data exploration and analysis.

Chapter Summary

This chapter introduced fundamental data visualization techniques using matplotlib, focusing on
creating clear and effective visual representations of data. We explored the implementation of
core plot types including line graphs for trend analysis, bar charts for categorical comparisons,
and scatter plots for relationship examination. The chapter then covered essential customization
techniques to enhance visualization clarity and impact, demonstrating how to modify visual
elements such as colour schemes, axis labels, legends, and scale settings. Special attention was
given to best practices in visualization design to ensure clear communication of data insights.

Review Questions

Page | 41
1. You have a dataset containing monthly sales data over a 5-year period. Create a line plot that:
• Shows the sales trend over time
• Includes appropriate x and y-axis labels
• Features a title "Monthly Sales Performance (2019-2023)"
• Uses a different colour than the default blue
• Adds gridlines for better readability
2. Given a DataFrame containing customer satisfaction scores (1-5) across different product
categories:

data = {

'Category': ['Electronics', 'Clothing', 'Books', 'Home', 'Food'],

'Score': [4.2, 3.8, 4.5, 3.9, 4.1]

Create a horizontal bar chart that displays this information with custom colours and sorted
bars from highest to lowest score.

3. You have two lists: hours_studied and exam_scores. Create a scatter plot that:
• Shows the relationship between study hours and exam performance
• Includes axis labels and a title
• Features a trend line
• Uses custom markers (e.g., triangles instead of circles)
• Has a custom figure size of 10x6 inches
4. You need to create a comparison of quarterly sales data for 2022 and 2023 using a grouped
bar chart. How would you:
• Set up the data structure
• Create grouped bars
• Add a legend to distinguish between years
• Include value labels on top of each bar
• Adjust the spacing between bar groups
Page | 42
5. Your dataset shows the distribution of customer ages across different store locations. Create
a series of subplots that:
• Shows histograms for each location
• Uses a 2x2 grid layout
• Shares the same x and y axes across all subplots
• Includes appropriate titles for each subplot
• Features different bin colours for each location

Page | 43
5 Introduction to Probability

Learning Objectives:

By the end of this section, students should be able to:

• Apply probability concepts to calculate and analyse the likelihood of events in

both theoretical and practical contexts.
• Analyse the differences between Bayesian and Frequentist probability
approaches to determine which method is most appropriate for specific
probability scenarios.
• Evaluate experimental data to demonstrate how the Law of Large Numbers
validates theoretical probability predictions in practical applications.
• Integrate various probability rules (addition, multiplication, conditional) to solve
complex problems involving compound and dependent events.

5.1 What is Probability?

Probability serves as a cornerstone in data science, providing a mathematical framework to
quantify the likelihood of events occurring. With values ranging from 0 (impossible) to 1 (certain),
probability helps data scientists and analysts navigate uncertainty and make data-driven
decisions. This fundamental concept is crucial for building predictive models, conducting
statistical inference, and understanding patterns within datasets. Whether analysing simple
events like coin tosses or complex scenarios in machine learning, probability theory provides the
tools to reason about uncertain outcomes systematically.

The building blocks of probability start with procedures - specific actions that lead to outcomes,
such as rolling dice or measuring user behaviour on a website. These procedures generate events,
which are collections of possible outcomes. Events can be further broken down into simple events
- basic outcomes that cannot be subdivided further. Understanding this hierarchy is essential for
properly analysing and modelling real-world scenarios.

Page | 44
For instance, while throwing two dice might seem like a single event, it consists of two simple
events (the outcome of each individual die), demonstrating how complex situations can be
decomposed into more manageable components.

Sample spaces represent the complete set of possible outcomes for a given procedure, forming
the foundation for probability calculations. A classic example is flipping a coin three times, which
yields a sample space of eight possible combinations (HHH, HHT, HTT, HTH, TTT, TTH, THH, THT).
This illustrates how even seemingly simple procedures can generate multiple possible outcomes
that must be considered in probability calculations. In data science applications, understanding
sample spaces becomes crucial when working with larger datasets and more complex scenarios,
as it helps in properly accounting for all possible outcomes and their respective probabilities,
leading to more accurate analyses and predictions.

The probability of an event represents the frequency, or chance, that the event will happen. For
notation, if A is an event, P(A) is the probability of the occurrence of the event. We can define the
actual probability of an event, A, as follows:

In this equation, A is the event in question. Think of an entire universe of events where anything
is possible, and let's represent it as a circle. We can think of a single event, A, as being a smaller
circle within that larger universe, as shown in the following diagram:

Page | 45
Let's now pretend that our universe involves a research study on humans, and the A event is
people in that study who have cancer. If our study has 100 people and A has 25 people, the
probability of A or P(A) is 25/100. The maximum probability of any event is 1. This can be
understood as the red circle grows so large that it is the size of the universe (the larger circle).

The most basic examples (I promise they will get more interesting) are coin flips. Let's say we have
two coins and we want the probability that we will roll two heads. We can very easily count the
number of ways two coins could end up being two heads. There's only one! Both coins have to
be heads. But how many options are there? It could either be two heads, two tails, or a heads/tails
combination (Sinan, 2016).

First, let's define A. It is the event in which two heads occur. The number of ways that A can occur
is 1. The sample space of the experiment is {HH, HT, TH, TT}, where each two-letter word indicates
the outcome of the first and second coin simultaneously. The size of the sample space is four. So, P
(getting two heads) = 1/4. Let's refer to a quick visual table to prove it. The following table denotes
the options for coin 1 as the columns and the options for coin 2 as the rows. In each cell, there is
either a True or False. A True value indicates that it satisfies the condition (both heads) and a False
indicates otherwise.

5.2 Bayesian versus Frequentist

The previous example was almost too easy. In practice, we can barely ever truly count the number
of ways something can happen. For example, let's say that we want to know the probability of a
random person smoking cigarettes at least once a day. If we wanted to approach this problem
using the classical way (the previous formula), we would need to figure out how many ways a
person is a smoker—someone who smokes at least once a day—which is not possible!

Page | 46
When faced with such a problem, two main schools of thought are considered when it comes to
calculating probabilities in practice: The Frequentist approach and the Bayesian approach. This
chapter will focus heavily on the Frequentist approach while the subsequent chapter will dive into
the Bayesian analysis.

5.2.1 Frequentist Approach

In a Frequentist approach, the probability of an event is calculated through experimentation. It
uses the past to predict the future chance of an event. The basic formula is as follows:

Fundamentally, we observe numerous instances of the event and count the number of times A
was satisfied. The division of these numbers approximates the probability (Sinan, 2016).

5.2.2 Bayesian Approach

The Bayesian approach differs by dictating that probabilities must be distinguished using
theoretical means. Using the Bayes approach, we would have to think a bit more critically about
events and why they occur. Neither methodology is 100% the correct answer all the time.
Typically, it comes down to the problem and the difficulty of using either method. The core of the
Frequentist approach is the relative frequency (Sinan, 2016). The relative frequency of an event
is how often an event occurs divided by the total number of observations.

Example

Assume that you are interested in discovering how often a person who visits your website is likely
to return at a later date. This is occasionally called the rate of repeat visitors. In the previous
definition, we would define our A event as being a visitor coming back to the site. We would then
have to calculate the number of ways a person can come back, which doesn't make sense at all!
In this case, many people would turn to a Bayesian approach; however, we can calculate what is
known as relative frequency.

Page | 47
So, in this case, we can take the visitor logs and calculate the relative frequency of event A (repeat
visitors). Let's say, of the 1,458 unique visitors in the past week, 452 were repeat visitors. We can
calculate this as follows:

So, about 31% of your visitors are repeat visitors.

5.3 The law of large numbers

The law of large numbers, states that if we repeat a procedure over and over, the relative
frequency probability will approach the actual probability. Let's try to demonstrate this using
Python.

If I were to ask you the average of the numbers 1 and 10, you would very quickly answer around
5. This question is identical to asking you to pick the average number between 1 and 10. Let's
design the experiment to be as follows:

Python will choose n random numbers between 1 and 10 and find their average. We will repeat
this experiment several times using a larger n each time, and then we will graph the outcome.
The steps are as follows:

Pick a random number between 1 and 10 and find the average.

Pick two random numbers between 1 and 10 and find their average.

Pick three random numbers between 1 and 10 and find their average.

Pick 10,000 random numbers between 1 and 10 and find their average.

Graph the results.

Page | 48
import numpy as np import
pandas as pd

from matplotlib import pyplot as plt

%matplotlib inline results

= []

for n in range(1,10000):

nums = [Link](low=1,high=10, size=n) # choose n numbers between 1

and 10

mean = [Link]() # find the average of these

numbers

[Link](mean) # add the average to a running

list

# POP QUIZ: How large is the list results? len(results) #

9999

# This was tricky because I took the range from 1 to 10000 and usually we do from 0 to 10000

df = [Link]({ 'means' : results})

print [Link]() # the averages in the beginning are all over the place!

# means
# 9.0

# 5.0

# 6.0

# 4.5

# 4.0

print [Link]() # as n, our size of the sample size, increases, the averages get closer
to 5!

# means

# 4.998799

# 5.060924

# 4.990597

# 5.008802

# 4.979198

[Link](title='Law of Large Numbers')

[Link]("Number of throws in sample")

[Link]("Average Of Sample")

Page | 49
What this is essentially showing us is that as we increase the sample size of our relative frequency,
the frequency approaches the actual average (probability) of 5.

5.4 Rules of Probability

In probability, we have some rules that become very useful when visualization gets too
cumbersome. These rules help us calculate compound probabilities with ease.

5.4.1 Compound Events

Sometimes, we need to deal with two or more events. These are called compound events. A
compound event is any event that combines two or more simple events. When this happens, we
need some special notation (Sinan, 2016).

Given events A and B:

• The probability that A and B occur is P(A ∩ B) = P(A and B)

• The probability that either A or B occurs is P(A B) = P(A or B)

Understanding why we use set notation for these compound events is very important. Remember
how we represented events in a universe using circles earlier? Let's say that our Universe is 100
people who showed up for an experiment, in which a new test for cancer is being developed:

Page | 50
In the above diagram, the red circle, A, represents 25 people who have cancer. Using the relative
frequency approach, we can say that P(A) = number of people with cancer/number of people in
the study, that is, 25/100 = ¼ = .25. This means that there is a 25% chance that someone has
cancer.

Let's present a second event, called B, as shown, which contains people for whom the test was
positive (it claimed that they had cancer). Let's say that this is for 30 people. So, P(B) = 30/100 =
3/10 = .3. This means that there is a 30% chance that the test is positive for any given person.

These are two separate events, but they interact with each other. Namely, they might intersect
or have people in common, as shown here:

Page | 51
Anyone in the space that both A and B occupy, otherwise known as A intersect B or A ∩ B, are
people for whom the test claimed they were positive for cancer (A) and they do have cancer. Let's
say that's 20 people. The test said positive for 20 people, that is, they have cancer, as shown here:

This means that P(A and B) = 20/100 = 1/5 = .2 = 20%.If we want to say that someone has cancer
or the test came back positive. This would be the total sum (or union) of the two events, namely,
the sum of 5, 20, and 10, which is 35. So, 35/100 people either have cancer or had a positive test
outcome. That means, P(A or B) = 35/100 = .35 = 35%. All in all, we have people in the following
four different classes:

• Pink: This refers to the people who have cancer and had a negative test outcome
• Purple (A intersect B): These people have cancer and had a positive test outcome

Page | 52
 • Blue: This refers to the people with no cancer and a positive test outcome
• White: This refers to the people with no cancer and a negative test outcome

So, effectively, the only times the test was accurate were in the white and purple regions.

5.4.2 Conditional Probability

Let's select a random person from this study of 100 people. Let's also assume that you are told
that their test result was positive. What is the probability of them having cancer? So, we are told
that event B has already taken place and that their test came back positive. The question now is:
what is the probability that they have cancer, that is P(A)? This is called a conditional probability
of A given B or P(A|B). Effectively, it is asking you to calculate the probability of an event given
that another event has already happened (Sinan, 2016).

You can think of conditional probability as changing the relevant universe. P(A|B) (called the
probability of A given B) is a way of saying, given that my entire universe is now B, what is the
probability of A? This is also known as transforming the sample space.

Zooming in on our previous diagram, our universe is now B, and we are concerned with AB (A and
B) inside of B.

The formula can be given as follows:

P(A|B) = P(A and B) / P(B) = (20/100) / (30/100) = 20/30 = .66 = 66%

Page | 53
There is a 66% chance that if a test result came back positive, that person had cancer. This is the
main probability that the experimenters want. They want to know how good the test is at
predicting cancer.

5.4.3 The Addition Rule

The addition rule is used to calculate the probability of either-or events. To calculate P(A B) = P(A
or B), we use the following formula: P(A B) = P(A) + P(B) − P(A ∩ B)

The first part of the formula (P(A) + P(B)) makes complete sense. To get the union of the two
events, we must add together the area of the circles in the universe. But why the subtraction of
P(A and B)? This is because when we add the two circles, we are adding the area of intersection
twice, as shown in the following diagram:

See how both the red circles include the intersection of A and B? So, when we add them, we need
to subtract just one of them to account for this, leaving us with our formula.

Recall that we wanted the number of people who either had cancer or had a positive test result?
If A is the event that someone has cancer, and B is that the test result was positive, we have:

P(A or B) = P(A) + P(B) – P(A and B) = .25 + .30 - .2 = .35

This was calculated before visually in the diagram.

Page | 54
5.4.4 Mutual Exclusivity
We say that two events are mutually exclusive if they cannot occur at the same time. This means
that A∩B= 0 just that the intersection of the events is the empty set. When this happens, P(A∩B)
= P(A and B) = 0.

If two events are mutually exclusive, then:

P(A B) = P(A or B)= P(A) + P(B) − P(A ∩ B) = P(A) + P(B)

This makes the addition rule much easier. Some examples of mutually exclusive events include
the following:

• A customer seeing your site for the first time on both Twitter and Facebook
• Today is Saturday and today is Wednesday
• I failed Econ 101 and I passed Econ 101

5.4.5 The Multiplication Rule

The multiplication rule is used to calculate the probability of events. To calculate P(A ∩ B) =

P(A and B), we use the following formula:

P(A ∩ B) = P(A and B) = P(A) · P(B|A)

Why do we use B|A instead of B? This is because B may depend on A. If this is the case, then just
multiplying P(A) and P(B) does not give us the whole picture. In our cancer trial example, let's find
P(A and B). To do this, let's redefine A to be the event that the trial is positive and B to be the
person having cancer (because it doesn't matter what we call the events). The equation will be as
follows:

P(A ∩ B) = P(A and B) = P(A) · P(B|A) = .3 * .6666 = .2 = 20%

Page | 55
This was calculated visually. It's difficult to see the true necessity of using conditional probability,
so, let's try another, more difficult problem.

For example, of a randomly selected set of 10 people, 6 have iPhones and 4 have Androids. What
is the probability that if I randomly select two people, they both will have iPhones? This example
can be retold using event spaces, as follows:

I have the following two events:

a. This event shows the probability that I choose a person with an iPhone first
b. This event shows the probability that I choose a person with an iPhone second

So, I want the following:

P(A and B): P(I choose a person with an iPhone and a person with an iPhone)

So, I can use my P(A and B) = P(A) · P(B|A) formula.

P(A) is simple, right? People with iPhones are 6 out of 10, so, I have a 6/10 = 3/5 = 0.6 chance of
A. This means P(A) = 0.6.

So, if I have a 0.6 chance of choosing someone with an iPhone, the probability of choosing two
should just be 0.6 * 0.6, right?

But wait! We only have 9 people left to choose our second person from, because one was taken
away. So, in our new transformed sample space, we have 9 people in total, 5 with iPhones and 4
with droids, making P(B) = 5/9 = .555.

So, the probability of choosing two people with iPhones is 0.6 * 0.555 = 0.333 = 33%.

I have a 1/3 chance of choosing two people with iPhones out of 10. The conditional probability is
very important in the multiplication rule as it can drastically alter your answer.

Page | 56
5.4.6 Independence
Two events are independent if one event does not affect the outcome of the other, that is P(B|A)
= P(B) and P(A|B) = P(A).

If two events are independent, then:

P(A ∩ B) = P(A) · P(B|A) = P(A) · P(B)

Some examples of independent events are as follows:

• It was raining in San Francisco, and a puppy was born in India

• Flip a coin and get heads and flip another coin and get tails

Chapter Summary

This chapter explored fundamental concepts of probability theory and its practical applications in
data analysis. We examined different approaches to probability calculations, contrasting Bayesian
and Frequentist methodologies and their appropriate use cases. The chapter demonstrated how
the Law of Large Numbers connects theoretical probability predictions with experimental
outcomes in real-world scenarios. We concluded by covering the application of essential
probability rules, including addition, multiplication, and conditional probability, showing how
these principles combine to solve complex probability problems involving multiple and
dependent events.

Review Questions

1. A data science company conducts technical interviews in three stages: coding test, system
design, and behavioural interview. Historical data shows that:
• 75% of candidates pass the coding test
• 60% of those who pass coding succeed in system design
• 80% of candidates who reach the behavioural stage receive an offer

What is the probability that a candidate will successfully receive a job offer?

Page | 57
2. You're analysing customer behaviour data from an e-commerce platform:
• 40% of customers are premium members
• 70% of premium members make purchases monthly
• 30% of non-premium members make purchases monthly

If you randomly select a customer who makes monthly purchases, what is the probability they
are a premium member? (Use Bayes' Theorem)

3. In a dataset of user interactions:

• Event A: User clicks on an ad (probability 0.3)
• Event B: User makes a purchase (probability 0.2)
• Probability of purchase given ad click: 0.4

Find:

a. The probability of both events occurring

b. The probability of either event occurring
c. Are these events independent? Explain why or why not.
4. A data centre has three servers with the following failure probabilities per day:
• Server 1: 0.05
• Server 2: 0.03
• Server 3: 0.04

Assuming server failures are independent:

a. What is the probability that all servers work correctly for a day?
b. What is the probability that at least one server fails?
c. What is the probability that exactly two servers fail?

Page | 58
6 Introduction to Machine Learning

Learning Objectives:

By the end of this section, students should be able to:

• Evaluate and critique the fundamental concepts of machine learning and

their role in modern data analysis
• Create and defend examples of appropriate use cases for different types of
machine-learning approaches
• Compare and contrast supervised, unsupervised, and reinforcement
learning methods to determine optimal application scenarios
• Design appropriate machine learning solutions by synthesising knowledge of
the advantages and disadvantages of each learning type
• Construct and validate linear regression models while critically assessing their
assumptions and limitations

6.1 What is Machine Learning?

We define machine learning as giving computers the ability to learn from data without being given
explicit rules by the developer. Machine learning is concerned with the ability to ascertain certain
patterns (signals) out of data, even if the data has inherent errors in it (Sinan, 2016).

Machine learning models can learn from data without the explicit help of a human. That is the
main difference between machine learning models and classical algorithms. Classical algorithms
are told how to find the best answer in a complex system and the algorithm then searches for
these best solutions and often works faster and more efficiently than a human. However, the
bottleneck here is that the human must first come up with the best solution. In machine learning,
the model is not told the best solution and instead, is given several examples of the problem and
is told, figure out the best solution (Sinan, 2016).

Machine learning is just another tool in the tool belt of a data scientist. It is on the same level as
statistical tests (chi-square or t-tests) or uses base probability/statistics to estimate population
parameters. Machine learning is often regarded as the only thing data scientists know how to do,
Page | 59
and this is simply untrue. A true data scientist can recognize when machine learning is applicable
and more importantly, when it is not (Sinan, 2016).

Machine learning is widely used across various industries and applications, including image and
speech recognition, natural language processing, recommendation systems, autonomous
vehicles, and more. Its ability to analyse and interpret large amounts of data has made it a
powerful tool for solving complex problems and making data-driven decisions.

6.2 How does Machine Learning work?

Each type of machine learning and discrete model functions in very dissimilar ways, exploiting
different parts of mathematics and data science. Nevertheless, in general, machine learning
works by taking in data, finding relationships within the data, and giving as output what the model
learned, as illustrated in the following diagram:

Key Steps involved in Machine learning (Mayo, 2022):

• Data Collection: The first step is to gather relevant data from various sources. The quality and
quantity of the data play a crucial role in the success of machine learning algorithms.
• Data Preprocessing: Once the data is collected, it needs to be cleaned and prepared for
training. This involves handling missing values, normalizing the data, and converting it into a
suitable format for the machine learning algorithm.
• Feature Extraction: Feature extraction involves identifying and selecting the most relevant
features (characteristics) from the data that will be used to make predictions.

Page | 60
• Model Selection: Depending on the type of problem and the nature of the data, a suitable
machine learning model is chosen. There are various types of models, including decision
trees, support vector machines, neural networks, and many more.
• Model Training: The selected model is fed with the preprocessed data, and the algorithm
adjusts its internal parameters to find patterns and relationships within the data. During this
training process, the model learns from the data to make predictions.
• Model Evaluation: Once the model is trained, it needs to be evaluated on a separate set of
data that was not used during the training process. This testing data allows us to measure the
model's accuracy and effectiveness in making predictions.
• Model Optimization: Based on the evaluation results, the model may be fine-tuned and
optimized.
• Model Deployment: Once the model is trained and optimized, it can be integrated into real-
world applications to make predictions or automate decision-making.

6.3 Types of Machine Learning

There are many ways to segment machine learning and dive deeper. In Chapter 1, we discussed
statistical and probabilistic models. These models utilize statistics and probability, which we've
seen in the previous chapters, to find relationships between data and make predictions. In this
chapter, we will implement both types of models. In the following chapter, we will see machine
learning outside the rigid mathematical world of statistics/probability. One can segment machine
learning models by different characteristics, including (Sinan, 2016):

• The types of data/organic structures they utilize (tree/graph/neural network)

• The field of mathematics they are most related to (statistical/probabilistic)
• The level of computation required to train (deep learning)

For educational purposes, we offer our breakdown of machine learning models. Branching off
from the top level of machine learning, there are the following three subsets:

• Supervised learning
• Unsupervised learning

Page | 61
 • Reinforcement learning

6.4 Supervised Learning

Supervised learning is a type of machine learning where an algorithm is trained on a labelled
dataset that includes both input variables (features) and output variables (labels). The algorithm
learns to map the input variables to the correct output variable by identifying patterns and
relationships within the data. The goal of supervised learning is to accurately predict the labels of
new, unseen data. Common supervised learning algorithms include linear regression, logistic
regression, decision trees, and artificial neural networks. Supervised learning is commonly used
in applications such as image classification, speech recognition, and prediction modelling.
Supervised machine learning models are often called predictive analytics models, named for their
ability to predict the future based on the past.

Specifically, supervised learning works using parts of the data to predict another part. First, we
must separate data into two parts, as follows (Sinan, 2016):

• The predictors, which are the columns that will be used to make our prediction. These are
sometimes called features, inputs, variables, and independent variables.
• The response, which is the column that we wish to predict. This is sometimes called
outcome, label, target, and dependent variable.

Example

Suppose we wish to predict if someone will have a heart attack within a year. To predict this, we
are given that person's cholesterol, blood pressure, height, smoking habits, and perhaps more.
From this data, we must ascertain the likelihood of a heart attack. Suppose to make this
prediction, we look at the previous patients and their medical history. As these are previous
patients, we know not only their predictors (cholesterol, blood pressure, and so on), but also
know if they had a heart attack (because it already happened!).

Page | 62
The hope is that a patient will walk in tomorrow and our model will identify whether the patient
is at risk for a heart attack based on her/his conditions (just like a doctor would!). As the model
sees more and more labelled data, it adjusts itself to match the correct labels given to us. We can
use different metrics (explained later in this chapter) to pinpoint exactly how well our supervised
machine-learning model is doing and how it can better adjust itself.

One of the biggest drawbacks of supervised machine learning is that we need this labelled data,
which can be very difficult to get a hold of. Suppose we wish to predict heart attacks, we might
need thousands of patients along with all their filled-in medical information and years' worth of
follow-up records for each person, which could be a nightmare to obtain. In short, supervised
models use historically labelled data to make predictions about the future (Sinan, 2016).

6.4.1 Types of Supervised Learning

There are, in general, two types of supervised learning models: regression and classification. The
difference between the two is quite simple and lies in the response variable:

• Regression - Regression models attempt to predict a continuous response. This means that
the response can take on a range of infinite values.

Page | 63
 • Classification - Classification attempts to predict a categorical response, which means that
the response only has a finite number of choices.

6.4.2 Advantages of Supervised Learning

• Predictive Power: Supervised learning algorithms can make accurate predictions on new,
unseen data once they are trained on labelled data. This is especially useful in tasks like
classification and regression.
• Feedback Mechanism: With labelled data, the algorithm receives feedback during training,
enabling it to learn from its mistakes and improve its performance iteratively.
• Interpretable Results: In many cases, the trained models are interpretable, allowing users to
understand the factors influencing predictions.
• Well-studied Techniques: Supervised learning has been extensively studied and applied in
various domains, resulting in a wide range of robust algorithms and tools.
• Wide Applicability: Supervised learning can be applied to both structured and unstructured
data, making it versatile for various problem domains (Valcheva, 2022).

6.4.3 Disadvantages of Supervised Learning

• Data Dependency: Supervised learning requires labelled data for training, which can be
expensive and time-consuming to obtain, especially for large-scale datasets.
• Overfitting: If the model is too complex or the dataset is too small, it may memorize the
training data and perform poorly on new, unseen data.
• Bias and Noise: The quality of predictions heavily depends on the quality and
representativeness of the training data. Biased or noisy data can lead to biased models.
• Limited Generalisation: Models trained on specific data may not generalise well to different
distributions or real-world scenarios not seen during training (Valcheva, 2022).

Page | 64
6.4.4 Introduction to Linear Regression
Linear regression is an algorithm that offers a linear relationship between an independent
variable and a dependent variable to predict the outcome of future events. It is a statistical
method used in data science and machine learning for predictive analysis (Kanade , 2023).

The independent variable is also the predictor variable that remains unchanged due to the change
in other variables. However, the dependent variable changes with fluctuations in the independent
variable. The regression model predicts the value of the dependent variable. Therefore, linear
regression is a supervised learning algorithm that simulates a mathematical relationship between
variables and makes predictions for continuous or numeric variables such as sales, salary, age,
product price, etc. (Kanade , 2023).

[Link] Example 1
Let's consider a simple example with one independent variable. Suppose we want to predict a
person's exam score based on the number of hours they studied. We have a small dataset with
the number of hours studied and the corresponding exam scores. The Python code shown
attempts to realize this prediction using the scikit-learn library:
import numpy as np

import [Link] as plt

from sklearn.linear_model import LinearRegression

# Sample data

hours_studied = [Link]([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]).reshape(-1, 1)

exam_scores = [Link]([50, 60, 70, 75, 85, 90, 95, 100, 110, 120])

# Create a linear regression model

model = LinearRegression()

# Fit the model to the data

[Link](hours_studied, exam_scores)

# Make predictions for new data points

new_hours = [Link]([11, 12, 13]).reshape(-1, 1)

predicted_scores = [Link](new_hours)

Page | 65
# Print the coefficients (slope and intercept)

print("Coefficient (slope):", model.coef_[0])

print("Intercept:", model.intercept_)

# Plot the data and the regression line

[Link](hours_studied, exam_scores, label="Data Points")

[Link](hours_studied, [Link](hours_studied), color='red', label="Regression

Line")

[Link](new_hours, predicted_scores, color='green', label="Predictions")

[Link]("Hours Studied")

[Link]("Exam Score")

[Link]()

[Link]()

Output:
Coefficient (slope): 7.242424242424245

Intercept: 45.66666666666665

[Link] Example 2
We will attempt to find a linear relationship between our predictors and our response variable
(Sinan, 2016). Formally, we wish to solve for a formula of the following format:

Page | 66
 • y is our response variable
• xi is our ith variable (ith column or ith predictor)
• B0 is the intercept
• Bi is the coefficient for the xi term

Let's look at some data before we go in-depth. This dataset is publicly available and attempts to
predict the number of bikes needed on a particular day for a bike sharing program.

# read the data and set the datetime as the index

# taken from Kaggle: [Link]

import pandas as pd

import [Link] as plt

%matplotlib inline

url = '[Link]

bikes = pd.read_csv(url)

[Link]()

Here we can see that every row represents a single hour of bike usage. In this case, we are
interested in predicting count, which represents the total number of bikes rented in the period of
that hour.

Let's, for example, look at a scatter plot between temperature (the temp column) and count.
[Link](kind='scatter', x='temp', y='count', alpha=0.2)

Page | 67
Output:

And now, let's use a module, called seaborn, to draw ourselves a line of best fit, as follows:

import seaborn as sns

#using seaborn to get a line of best fit

[Link](x='temp', y='count', data=bikes, aspect=1.5, scatter_kws={'alpha':0.2})

Output:

The line in the graph tries to visualize and quantify the relationship between temp and count. To
make a prediction, we simply find a given temperature, and then see where the line would predict
the count. For example, if the temperature is 20 Degrees Celsius, then our line would predict that
bikes[['count', 'temp']].corr()

Output # 0.3944

Page | 68
about 200 bikes will be rented. If the temperature is above 40 degrees, then more than 400 bikes
will be required. It appears that as temp goes up, our count also goes up. Let's see if our
correlation value, which quantifies a linear relationship between variables, also matches this
notion, demonstrated in the code:

There is a (weak) positive correlation between the two variables! Now, let's go back to the form
of the linear regression:

Our model will attempt to draw a perfect line between all the dots in the preceding graph, but of
course, we can see that there is no perfect line between these dots! The model will then find the
best-fit line possible. How? We can draw infinite lines between the data points, but what makes
a line the best?

In this model, we are given the x and the y and the model learns the Beta coefficients, also known
as model coefficients:

• The black dots are the observed values of x and y.

• The blue line is our line of best fit.
• The red lines between the dots and the line are called the residuals; they are the distances
between the observed values and the line. They are how wrong the line is.

Each data point has a residual or a distance to the line of best fit. The sum of squared residuals is
the summation of each residual squared. The best-fit line has the smallest sum of squared residual
values. Let's build this line in Python!

# create X and y

feature_cols = ['temp'] # a lsit of the predictors

X = bikes[feature_cols] # subsetting our data to only the predictors

y = bikes['count'] # our response variable

Page | 69
Note how we made an X and a Y variable. These represent our predictors and our response
variable. Now, we will import our machine learning module, scikit learn, as shown:

# import scikit-learn, our machine learning module

from sklearn.linear_model import LinearRegression

Finally, we will fit our model to the predictors and the response variable, as follows:

linreg = LinearRegression() #instantiate a new model

[Link](X, y) #fit the model to our data

# print the coefficients

print linreg.intercept_

print linreg.coef_

6.04621295962 # our Beta_0

[ 9.17054048] # our beta parameters

Interpretation

B0 (6.04) is the value of y when X = 0. It is the estimation of bikes that will be rented when the
temperature is 0 Celsius. So, at 0 degrees, six bikes are predicted to be in use (it is cold!).

• B1 (9.17) is our temp coefficient.

• A change of 1 degree Celsius is associated with an increase of about 9 bikes rented.

Using Scikit we can predict a temp of 20 degrees.

[Link](20)

# 189.4570

This means that 190 bikes will likely be rented if the temperature is 20 degrees. Try the following
code that takes includes another feature (weather conditions) to the mix:

This means that:

• 1 unit increase in temperature is associated with a rental increase of 7.86 bikes

• 1 unit increase in season is associated with a rental increase of 22.5 bikes
Page | 70
 • 1 unit increase in weather is associated with a rental increase of 6.67 bikes
• 1 unit increase in humidity is associated with a rental decrease of 3.12 bikes

# create a list of features

feature_cols = ['temp', 'season', 'weather', 'humidity']

# create X and y

X = bikes[feature_cols]

y = bikes['count']

# instantiate and fit

linreg = LinearRegression()

[Link](X, y)

# pair the feature names with the coefficients

zip(feature_cols, linreg.coef_)

Output:
[('temp', 7.8648249924774403),

('season', 22.538757532466754),

('weather', 6.6703020359238048),

('humidity', -3.1188733823964974)]

Plot the Results

# multiple scatter plots

feature_cols = ['temp', 'season', 'weather', 'humidity']

[Link](bikes, x_vars=feature_cols, y_vars='count', kind='reg')

Page | 71
Output:

Chapter Summary

This chapter provided a comprehensive exploration of machine learning fundamentals and their
practical applications. We examined the core principles of machine learning, analyzing how it
differs from traditional programming approaches through its ability to learn from data. The
chapter delved into three primary types of machine learning: supervised learning, which uses
labeled data for prediction and classification; unsupervised learning, which discovers hidden
patterns in unlabeled data; and reinforcement learning, which learns through environment
interaction and feedback. Special attention was given to linear regression as a foundational
supervised learning technique, exploring its implementation, assumptions, and real-world
applications.

6.5 Unsupervised Learning

The second type of machine learning does not deal with predictions but has a much more open
objective. Unsupervised learning takes in a set of predictors and utilizes relationships between
the predictors to accomplish tasks, such as the following:

• Reducing the dimension of the data by condensing variables together. An example of this
would be file compression. Compression works by utilizing patterns in the data and
representing the data in a smaller format.
• Finding groups of observations that behave similarly and grouping them.

Page | 72
The first element on this list is called dimension reduction and the second is called clustering.
Both are examples of unsupervised learning because they do not attempt to find a relationship
between predictors and a specific response and so are not used to make predictions of any kind.
Unsupervised models, instead, are used to unearth patterns and representations of the data that
were previously unknown (Sinan, 2016).

The above screenshot is a representation of a cluster analysis. The model will recognize that each
uniquely coloured cluster of observations is like another but different from the other clusters in
some unique way.

Unsupervised learning has the key benefit of not requiring labelled data, making it significantly
simpler to get data that supports unsupervised learning models. Obviously, this has the
disadvantage that we lose any predictive ability, because without the response variable, our
model would be entirely impractical at generating any sort of predictions.

A major disadvantage is that it is difficult to ascertain how well we are doing. In a regression or
classification problem, we can easily tell how well our models are predicting by comparing our
models' answers to the actual answers. For example, if our supervised model predicts rain and it
is sunny outside, the model was incorrect. If our supervised model predicts the price will go up
by 1 Rand and it goes up by 99 cents, our model was very close! In supervised modelling, this
concept is foreign because we have no answer to compare our models to. Unsupervised models

Page | 73
are simply suggesting differences and similarities, which then requires a human's interpretation
(Sinan, 2016).

6.5.1 Advantages of Unsupervised Learning

• No Labels Required: Unsupervised learning does not rely on labelled data, making it more
scalable and applicable to a wide range of datasets.
• Data Exploration: It can help discover patterns, structures, and relationships within the data,
even when the underlying structure is not well-defined.
• Anomaly Detection: Unsupervised learning can identify unusual patterns or outliers in the
data, which can be valuable for anomaly detection and fraud detection (Valcheva, 2022).

6.5.2 Disadvantages of Unsupervised Learning

• Lack of Clear Objectives: Without predefined labels, it can be challenging to evaluate the
performance of unsupervised learning algorithms objectively.
• Interpretability: Unsupervised learning models are often more complex and harder to
interpret compared to supervised models.
• Subjectivity: The interpretation of the results from unsupervised learning can be subjective
and may require domain expertise for meaningful analysis.
• Difficulty in Validation: Due to the absence of ground truth labels, validating the performance
of unsupervised learning models can be difficult (Valcheva, 2022).

Page | 74
6.6 Reinforcement Learning
In reinforcement learning, algorithms get to choose an action in an environment and then are
rewarded (positively or negatively) for choosing this action. The algorithm then adjusts itself and
modifies its strategy to accomplish some goal, which is usually to get more rewards (Sinan, 2016).

This type of machine learning is very popular in AI-assisted gameplay as agents (the AI) are
allowed to explore a virtual world and collect rewards and learn the best navigation techniques.
This model is also popular in robotics, especially in the field of self-automated machinery,
including cars:

It can be thought that reinforcement is like supervised learning in that the agent is learning from
its past actions to make better moves in the future; nevertheless, the main difference lies in the
reward. The reward does not have to be tied in any way to a "correct" or "incorrect" decision. The
reward simply encourages (or discourages) different actions (Sinan, 2016).

Review Questions

1. A retail company wants to implement a machine-learning solution to predict customer churn.

They have historical data including customer purchase history, demographics, and whether
they churned or not. Which type of machine learning would be most appropriate and why?
Explain your reasoning for considering the advantages and disadvantages of your chosen
approach.
2. You are given a dataset of customer transactions with no labels:

Page | 75
 data = {

'customer_id': [1, 2, 3, 4, 5],

'annual_spending': [1200, 4500, 800, 5000, 1100],

'frequency_visits': [12, 52, 8, 48, 10],

'avg_basket_size': [100, 86, 100, 104, 110]

What type of machine learning would you use to identify customer segments? Outline your

approach.

3. Using linear regression, analyse the following house price data to predict the price of a 2,000
sq ft house:

square_feet = [1500, 1800, 2200, 2500, 3000]

prices = [200000, 250000, 300000, 350000, 400000]

Page | 76
7 Introduction to Web Scraping using BeautifulSoup

Learning Objectives:

By the end of this section, students should be able to:

• Analyse web page HTML structures to identify the appropriate elements and
attributes for data extraction.
• Apply BeautifulSoup library methods to parse and navigate HTML documents
effectively.
• Construct Python scripts that extract specific data from websites using
BeautifulSoup's searching and filtering capabilities.
• Evaluate web scraping results and implement error handling to ensure robust
data collection.

Web scraping is the process of automatically extracting data from websites. It's a powerful
technique used for various purposes, such as data analysis, price monitoring, and content
aggregation. In this chapter, we'll introduce you to web scraping using Python and the Beautiful
Soup library.

7.1 What is Beautiful Soup?

Beautiful Soup is a Python library that makes it easy to scrape information from web pages. It sits
atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying
the parse tree.

7.2 Installation of Required Libraries

1. To install Beautiful Soup, you can use pip, Python's package installer. Open your terminal and
run pip install beautifulsoup4
2. You'll also need an HTML parser.
We recommend lxml: pip install lxml
Alternatively, you can use Python's built-in [Link]: pip install html5lib
3. It's also a good practice to install the requests library for making HTTP requests: pip install
requests
Page | 77
7.3 Using Beautiful Soup for Web Scraping

The script above does the following:

• Import the necessary libraries: requests for making HTTP requests and BeautifulSoup from
the bs4 module.
• Send a GET request to the specified URL using [Link]().
• Create a BeautifulSoup object by passing the response text and specifying the parser
('lxml' in this case).
• Print the title of the page using [Link]. This navigates to the <title> tag and
extracts its text content.

Page | 78
 • Use soup.find_all('p') to locate all <p> tags in the document, then iterate over them and
print their text content.

The "requests" library is used for grabbing data from the web. The "BeautifulSoup" library is used
for parsing HTML documents to find specific information. The output of the code displays all the
paragraphs found on the page.

7.4 Common Operations

7.4.1 Finding Elements

Beautiful Soup provides several methods to locate elements in the document:

• find() returns the first matching element, while find_all() returns a list of all matching
elements.
• The class_ parameter is used because the class is a reserved keyword in Python.
• select() method allows you to use CSS selectors, which can be very powerful for complex
selections.
• You can combine multiple attributes in find() and find_all() to narrow down your search.
• Custom functions can be passed to find_all() for more complex selection logic.

Page | 79
7.4.2 Navigating the Tree

• You can use dot notation to navigate nested elements directly.

• next_sibling and previous_sibling allow you to move between elements at the same level.
• parent lets you move up the tree to the containing element.
• children give you direct child elements, while descendants include all nested elements at any
level.

7.4.3 Extracting Data

Once you've located the elements you're interested in, you can extract various types of
data:

Page | 80
7.5 Best Practices and Considerations
1. Respect [Link]: Always check a website's [Link] file for scraping guidelines.
2. Be polite: Don't overwhelm servers with too many requests. Use delays between requests.
3. User-Agent: Set a proper User-Agent in your requests to identify your bot.
4. Handle errors: Implement error handling to deal with network issues or changes in website
structure.
5. Stay up to date: Websites change frequently. Regularly check and update your scraping code.

To learn more about web scraping visit these sites:

1. Beautiful Soup Documentation

2. Requests Library Documentation
3. Web Scraping Ethics
4. Python Web Scraping Tutorial

7.6 Case Study – What’s in a job description?

Looking for a job in data science? Great, let me help. In this case study, I have "scraped"
(taken from the Web) 1,000 job descriptions for companies actively hiring data scientists (as
of January 2016). The goal here is to look at some of the most common keywords people
use in their job descriptions.

Page | 81
Code Extract 1:
import requests

# used to grab data from the web

from BeautifulSoup import BeautifulSoup #

used to parse HTML

from sklearn.feature_extraction.text import CountVectorizer

# used to count number of words and phrases (we will be using this module a lot)

The code extract imports three libraries (or packages) commonly used for web scraping and data
analysis. The "requests" library is used to grab data from the web. The "BeautifulSoup" library is
used to parse HTML documents to find specific information. Finally, the "CountVectorizer" library
from "sklearn" is used to count the number of words and phrases in a given dataset. This library is
used for various types of data analysis such as text classification.

Code Extract 2:
texts = []

# hold our job descriptions in this list

for index in range(0,1000,10): # go through 100 pages of indeed

page = '[Link]/jobs?q=data+scientist&start='+str(index) #
identify the url of the job listings

web_result = [Link](page).text

# use requests to actually visit the url

soup BeautifulSoup(web_result)

# parse the html of the resulting page

for listing in [Link]('span', {'class':'summary'}: #

for each listing on the page

[Link]([Link]) # append the text of the listing to our list

Page | 82
This code extract uses a combination of requests, BeautifulSoup, and a range to scrape job
descriptions from a job board. The first line is creating an empty array in which job descriptions
will be stored. The for loop using range is looping through the first 100 pages of the job board
Indeed and creating a URL and making a GET request for each page of job listings. The code is
then using BeautifulSoup to parse the HTML of the pages and find all listings within the page with
the class "summary". Finally, for each of those listings, the text of the listing is being appended to
the texts array created in line 1.

Code Extract 3:
type(texts) # == list

vect = CountVectorizer(ngram_range=(1,2), stop_words='english') #

Get basic counts of one and two word phrases

matrix = vect.fit_transform(texts)

# fit and learn to the vocabulary in the corpus

print len(vect.get_feature_names()) # how many features are there

Output # There are 11,293 total one and two-word phrases in my case!!

The code snippet creates a variable called vect which is an instance of the CountVectorizer class.
The CountVectorizer class can be used to create a vector matrix of all of the one and two-word
phrases in the text. The code specifies the ngram_range to be (1,2) and for the stop words to be
English, which will prevent common words like "the" and "and" from being included in the vector
matrix. The code then creates a matrix variable which is fitted and learned to the vocabulary in
the corpus. The last line is printing out the number of feature names which in this case is 11,293
one and two-word phrases.

Chapter Summary

This chapter focused on web scraping fundamentals using Python's BeautifulSoup library. We
explored techniques for analysing HTML structure and identifying key elements for data
extraction. The chapter covered essential BeautifulSoup methods for parsing and navigating

Page | 83
HTML documents, including various selector techniques and traversal methods. We examined
practical approaches for constructing robust web scraping scripts, with emphasis on error
handling and data validation. Special attention was given to best practices for reliable data
collection and handling common web scraping challenges.

Review Questions

1. Explain the difference between these two BeautifulSoup selection methods:

• find('div', {'class': 'content'})
• select('[Link]')

When would you choose one over the other

2. Consider a scenario where you're scraping data from an e-commerce website:

a. What ethical and legal considerations should you take into account?
b. How might you implement these considerations in your code?
c. What potential consequences could occur if these aren't considered?
3. A website's HTML structure changes frequently, breaking your scraping script.

Describe:

a. Why might this happen?

b. How could you make your scraper more resilient to changes?
c. What monitoring systems could you implement?

Page | 84
8 References
Anaconda, 2025. Anaconda Distribution. [Online]
Available at: [Link]
[Accessed 25 July 2025].

Danguti, S., 2023. Exploring Logical Operators in Python: And, Or, Not. [Online]
Available at: [Link]
and-or-not-d7c6a10edc83
[Accessed 26 July 2025].

EDUCBA, 2025. Python Keywords. [Online]

Available at: [Link]
[Accessed 25 July 2025].

OpenStax, 2024. Introduction to Python Programming. [Online]

Available at: [Link]
[Accessed 25 July 2025].

Python Software Foundation, 2025. Welcome to [Link]. [Online]

Available at: [Link]
[Accessed 25 July 2025].

Altexsoft, 2020. Altexsoft. [Online]

Available at: [Link]

[Accessed 28 July 2023].

Burns, E., 2023. TechTarget Enterprise AI. [Online]

Available at: [Link]

ML

[Accessed 25 July 2023].

Chohan, F., 2019. Implementing Natural Language processing techniques for the Detection
of Stegosamp Files Generated using a Probabilistic Context-Free Grammar. UNISA
Repository, 2(1), pp. 1-199.

Kanade, V., 2022. SpiceWorks. [Online]

Available at: [Link]

[Accessed 30 July 2023].

Page | 85
Kanade, V., 2023. Spiceworks - What is Linear Regression?. [Online]

Available at: [Link]

linear-regression/

[Accessed 30 July 2023].

Mayo, M., 2022. Frameworks for Approaching the Machine Learning Process. [Online]

Available at: [Link]

[Link]

[Accessed 30 July 2023].

Quinn, R., 2021. Microsoft - What is a machine learning model?. [Online]

Available at: [Link]

machine-learning-model

[Accessed 30 July 2023].

Sinan, O., 2016. Principles of Data Science. Birmingham, UK: Packt.

TIBC, 2023. TIBC: What is Unsupervised Learning?. [Online]

Available at: [Link]

[Accessed 23 July 2023].

Valcheva, S., 2022. IntellSpot. [Online]

Available at: [Link]

[Accessed 30 July 2023].

Page | 86