DATA ANALYTICS (ISM321)

Comprehensive Study Notes

 LECTURE 1: Introduction to Data Analytics

1.1 What is Data and Why Does it Matter?

Data is simply raw facts and figures — numbers, text, images, videos, or sounds — that have been
collected but not yet processed or interpreted. Think of it like the raw ingredients in a kitchen: by
themselves they don't mean much, but once you cook them (process them), they become something
useful and valuable.

📌 Data: Raw, unprocessed facts and figures collected from the world around us. On its
own, data has no meaning until it is processed and analysed.

In today's world, companies generate and collect enormous amounts of data every single second —
from customers buying things online, people posting on social media, sensors in machines, GPS
signals from phones, and much more. The challenge is not just collecting this data, but making sense of
it.

💡 Key Point: In 1999, Bill Gates wrote that businesses that succeed in the future will be
those that do a good job with data. This has proven absolutely true — companies like
Google, Facebook, and Amazon are worth trillions largely because of how well they use
data.

Why Do We Need to Analyse Data?

Simply having data is not enough. Its real value lies in what you can learn from it. Data must be:
• Processed — organised and structured so it can be read
• Analysed — examined using techniques and tools
• Interpreted — understood in context to extract meaning

✅ Real-World Example: Banks use data analytics to monitor every transaction and
automatically detect fraudulent activity. Mobile phone companies analyse call patterns to
understand customer behaviour. Facebook and LinkedIn's enormous company valuations
are largely based on the data they collect and the insights they derive from it.

1.2 What is Data Analytics?

📌 Data Analytics: The process of collecting data from various sources, standardising it,
analysing it, and interpreting the results in order to gain insights and draw actionable
conclusions that help make better business decisions.
In simple terms: Data Analytics = taking raw data and turning it into useful knowledge that helps people
make smarter decisions.

1.3 Big Data — The 3 Vs

The huge increase in data generation led to the term 'Big Data'. Big Data is so large, complex, and fast-
moving that traditional tools cannot handle it. It is defined by three main characteristics, often called the
3 Vs:

The 3 Vs Explanation
Volume The sheer amount of data. Instead of
thousands of rows, we're talking billions of
rows and millions of columns. Example:
Facebook stores over 100 petabytes of
photos alone.
Variety The different types and structures of data —
from traditional databases (structured) to
social media posts, images, and videos
(unstructured).
Velocity The speed at which new data is created and
must be processed. Think of live stock
market prices, real-time GPS tracking, or
social media feeds updating every second.

💡 Key Point: Big Data requires new technical architectures and tools to process and
analyse. Traditional databases (like Microsoft Access or even SQL Server) simply cannot
keep up with the scale, variety, and speed of Big Data.

1.4 Types of Data Structures

Not all data looks the same. Data comes in different 'structures' — the way it is organised and
formatted. Understanding this helps you choose the right tools to work with it.

📌 Structured Data: Data that has a clearly defined format, data type, and organisation —
like a spreadsheet or a traditional database table. Every row follows the same pattern.
Examples: Transaction records in a bank, CSV files, Excel spreadsheets.

📌 Semi-Structured Data: Data that doesn't fit neatly into a table but still has some
recognisable pattern or tags that give it structure. Examples: XML files, JSON data, HTML
web pages. These can be 'parsed' (read by a computer) because of their tags.
 📌 Quasi-Structured Data: Data that is mostly text with some inconsistencies in format
that require extra work to clean up. Example: Web clickstream data (records of what links
users click), which may have inconsistent formats.

📌 Unstructured Data: Data with no inherent structure at all. It cannot be stored in

traditional database tables without significant processing. Examples: Text documents, PDF
files, images, videos, audio recordings. This is the most common form of Big Data.

💡 Key Point: Most Big Data is unstructured. This is why traditional databases struggle
with Big Data — they were designed for structured data. About 80–90% of all data
generated today is unstructured.

1.5 Data Repositories

A data repository is simply a place where data is stored. Different repositories suit different purposes:

Repository Type Description

Spreadsheets & Data Marts (Spreadmarts) Simple spreadsheet tools and small
databases used for basic recordkeeping.
Good for small-scale analysis, but limited.
The analyst depends on data extracts.
Data Warehouses Large, centralised databases purpose-built
for storing and querying big datasets.
Support business intelligence (BI) and
reporting. However, they require IT support
for access and changes, and are slow for
advanced analytics.
Analytic Sandboxes Flexible working spaces where data
scientists can experiment with data from
multiple sources. 'Analyst owned' rather than
'IT owned', allowing faster, more creative
analysis without affecting production
systems.

✅ Real-World Example: Think of a Data Warehouse like a library — well-organised and

controlled, but you need to follow rules to access it. An Analytic Sandbox is like your
personal desk — you can spread out, experiment, and work freely without worrying about
disturbing anyone else.
1.6 Analysis vs. Analytics
These two terms sound similar but mean different things. This is a very common exam question!

Analysis Analytics
Looks at PAST data Uses past data to predict the FUTURE
Answers: 'What happened? Why?' Answers: 'What will happen next?'
Includes: data cleaning, visualisation, Includes: data acquisition, extraction,
data stories aggregation
Example: Why did sales drop last rainy Example: What will sales be in Makeni next
season? rainy season?

💡 Key Point: Think of Analysis as a detective examining past evidence, and Analytics as
a forecaster using that evidence to predict what comes next. Analysis is a SUBSET of
Analytics — Analytics is the bigger process.

The Two Types of Analytics

📌 Qualitative Analytics: Uses human judgement, intuition, and experience alongside

data to make business decisions. It is less mathematical and more about interpretation and
meaning. Example: A manager using their industry knowledge to decide which market to
enter next.

📌 Quantitative Analytics: Uses formulas, algorithms, and mathematical models applied

to numerical data. It is objective and measurable. Example: Using regression models to
calculate predicted sales figures for next quarter.

1.7 Business Drivers for Data Analytics

Organisations use data analytics for four main business reasons:
• Optimise business operations: improving sales, pricing, profitability, and efficiency.
• Identify business risks: detecting customer churn (when customers leave), fraud, or loan
defaults.
• Predict new business opportunities: identifying which customers to upsell or cross-sell to,
and finding new customer prospects.
• Comply with laws and regulations: Anti-Money Laundering (AML) compliance, fair lending
regulations, and financial reporting standards like Sarbanes-Oxley (SOX).
1.8 Data Science Disciplines
Data Science is an umbrella term covering several related disciplines. Understanding where each sits
helps you know which tools and skills to use:

Discipline Description & Tools

Business Intelligence (BI) Uses past data to create reports,
dashboards, and KPIs to help managers
understand what has happened. Tools:
Excel, Power BI, Tableau.
Traditional Statistical Methods Uses statistical techniques to assess
potential future scenarios. Tools: SPSS,
Stata, Excel, R.
Machine Learning (ML) Uses algorithms that learn from data to make
predictions and find patterns automatically.
Tools: Python (scikit-learn), TensorFlow,
PyTorch.

1.9 Current Analytical Architecture and Its Problems

Most organisations have a data architecture (the way their data systems are set up) that looks like this:
• Step 1 — Data Sources: Data comes from various internal systems.
• Step 2 — Enterprise Data Warehouse (EDW): Data is cleaned and loaded into a central
warehouse.
• Step 3 — Dashboards, Reports & Alerts: BI tools read from the warehouse to create reports
for managers.
• Step 4 — Data Science Users: Analysts extract copies of data for deeper analysis using tools
like R or Excel.

The problems with this setup are significant:

• High-value data is hard to reach — analysts are last in line after operational processes.
• Data moves in batches — analysts work with old copies of data, not live data.
• In-memory limitations — tools like R and Excel can only handle data that fits in a computer's
memory (RAM), limiting the size of datasets.
• Analysis stays isolated — insights from data scientists rarely feed back into the main systems.
• Slow time-to-insight — it can take days or weeks to get access to data and produce results.

💡 Key Point: The solution to these architecture problems is the Analytic Sandbox — a
dedicated, flexible environment where data scientists can work with large datasets without
disturbing production systems. This sits alongside the traditional data warehouse, not
replacing it.
1.10 The Big Data Ecosystem
As more data is created, a new economy has emerged around it. There are four main groups of
players:

Player Description & Examples

Data Devices (The Sensornet) Devices that generate raw data continuously.
Examples: mobile phones, GPS devices,
ATMs, credit card readers, video surveillance
cameras, medical imaging machines. For
every 1GB of new data, an additional 1
petabyte of metadata (data about that data)
is created!
Data Collectors Organisations that collect data directly from
devices and users. Examples: cable TV
companies tracking viewing habits, mobile
network operators (MNOs) tracking call data,
retailers tracking purchases.
Data Aggregators Organisations that compile and package data
collected by others, then sell it as products.
Examples: credit bureaus, information
brokers, advertising platforms like Google.
Data Users/Buyers Organisations that directly benefit from the
data collected and aggregated by others.
Examples: banks, retailers, media
companies, law enforcement, employers
screening job candidates.

1.11 The Data Scientist

The Data Scientist is the key person who bridges the gap between raw data and business value.

Key Activities of a Data Scientist

• Reframe business problems: Take a business challenge and translate it into an analytics
problem that can be solved with data.
• Design and implement models: Apply statistical and machine learning methods to real
business problems.
• Derive and communicate insights: Turn results into clear, actionable recommendations for
decision-makers.

Key Skills of a Data Scientist

• Quantitative skills: Strong in mathematics and statistics.
• Technical aptitude: Programming (Python, R), machine learning, software engineering.
• Critical thinking: Ability to examine their own work sceptically and spot weaknesses.
 • Creativity and curiosity: Passion for finding patterns and telling stories with data.
• Communication: Ability to explain complex findings in simple language to non-technical
stakeholders.

✅ Real-World Example: Target (US Retailer): Target's data scientists analysed shopping
behaviour and discovered they could predict whether a customer was pregnant — and even
which month of pregnancy they were in — based on what products they were buying. This
allowed Target to send personalised coupons at just the right time. In one famous case,
Target knew a teenage girl was pregnant before her own parents did, based on her buying
patterns.
 LECTURE 2: The Data Analytics Lifecycle

2.1 Key Roles for a Successful Analytics Project

A data analytics project is a team effort. Seven key roles need to be filled for a project to succeed.
Think of it like a football team — everyone has a specific position that contributes to the overall goal.

Role Responsibilities
Business User Understands the business domain and
benefits from the results. Advises the team
on the context and value of the project.
Usually a business analyst, line manager, or
subject matter expert.
Project Sponsor Responsible for why the project exists.
Provides funding, defines the core business
problem, sets priorities, and judges the value
of the final output.
Project Manager Ensures the project meets its milestones,
deadlines, and quality standards. Keeps
everyone on track.
Business Intelligence (BI) Analyst Brings deep knowledge of business data,
KPIs, and metrics. Creates dashboards and
reports, and knows where data comes from
and how it flows.
Database Administrator (DBA) Sets up and configures the databases.
Manages data access permissions and
ensures data security.
Data Engineer Extracts, moves, and transforms data into the
analytic sandbox. Works closely with the data
scientist to shape data for analysis.
Data Scientist The analytics expert. Designs and executes
statistical models and machine learning
techniques. Ensures the overall analytics
objectives are met.

💡 Key Point: The DBA sets up the database environment, while the Data Engineer
actually moves and transforms the data. The Data Scientist then analyses it. These three
roles work very closely together.

2.2 The Data Analytics Lifecycle — Overview

The Data Analytics Lifecycle is a structured, repeating process that guides a data analytics project from
start to finish. It has 6 phases and is cyclical — meaning results from later phases can send the team
back to earlier phases if needed.

Phase Simple Summary

Phase 1 — Discovery Understand the problem and the data
available.
Phase 2 — Data Preparation Gather, clean, and organise the data.
Phase 3 — Model Planning Decide what type of model to build.
Phase 4 — Model Building Actually build and test the model.
Phase 5 — Communicate Results Present findings to stakeholders.
Phase 6 — Operationalise Deploy the model into real-world use.

2.3 The Six Phases in Detail

Phase 1 — Discovery
This is the starting point of every analytics project. The team must first understand the landscape
before diving into data.
• Learn the business domain: Understand the industry, the organisation, and the history of similar
projects.
• Assess available resources: What data, people, technology, and time are available?
• Frame the problem: Translate the business problem into an analytics challenge. For example,
'We're losing customers' becomes 'Build a model to predict which customers are at risk of
leaving (churning)'.
• Formulate Initial Hypotheses (IHs): Make educated guesses about what might be true, which
the data will later confirm or disprove.

💡 Key Point: The quality of the Discovery phase determines the success of everything
that follows. A poorly defined problem leads to wasted effort and wrong answers, even if the
analysis is technically perfect.

Phase 2 — Data Preparation

This phase is where data is gathered, cleaned, and made ready for analysis. It is often the most time-
consuming phase — data scientists spend up to 80% of their time here!
• Set up the Analytic Sandbox: The team needs a dedicated workspace where they can work with
data freely.
• Run ETL/ELT processes: Extract, Transform, Load (or Extract, Load, Transform) — moving
data from its source to the sandbox.
• Clean and condition the data: Fix errors, handle missing values, remove duplicates, and correct
data types.
 • Explore the data: Begin to understand patterns, distributions, and relationships in the data.

📌 ETL (Extract, Transform, Load): A process for moving data from source systems to a
data warehouse or sandbox. First, data is Extracted from sources, then Transformed
(cleaned, formatted, standardised), then Loaded into the target system.

Phase 3 — Model Planning

Here the team decides HOW they will analyse the data. No actual models are built yet — this is the
planning stage.
• Explore relationships between variables: Which factors are related to each other? Which ones
might predict outcomes?
• Select key variables (features): Which columns in the data will be used in the model?
• Choose the model type: Will they use regression, classification, clustering, or another
technique? (More on this in Lecture 4.)

Phase 4 — Model Building

Now the team actually builds and tests the model.
• Create training, validation, and test datasets: Split the data so some is used to teach the model,
some to tune it, and some to test it.
• Build the model: Use the chosen algorithms and techniques on the training data.
• Evaluate: Check if the model works well enough, and refine if needed.
• Consider infrastructure: Is more powerful hardware needed? Can the model run fast enough on
the available systems?

Phase 5 — Communicate Results

A great analysis is worthless if nobody understands it. This phase is about turning results into a story.
• Determine success or failure: Compare results against the criteria set in Phase 1.
• Identify key findings: What are the most important things the data revealed?
• Quantify business value: What does this mean in financial or operational terms?
• Create a narrative: Build a clear, compelling story that non-technical stakeholders can
understand.

Phase 6 — Operationalise
The final phase is about putting the model into real, ongoing use.
• Deliver final reports, code, and technical documentation.
• Run a pilot project: Test the model in a real environment with limited scope before full
deployment.
• Deploy to production: Integrate the model into the organisation's systems so it can run
automatically.
 ✅ Real-World Example: Imagine a bank building a fraud detection model. Phase 1:
Define 'fraud detection' as the goal. Phase 2: Gather transaction data and clean it. Phase 3:
Plan to use a classification model. Phase 4: Build and test the model. Phase 5: Present
findings to the risk management team. Phase 6: Deploy the model so it automatically flags
suspicious transactions in real time.

2.4 Practical Analytics Workflow

In practice, analytics projects follow a six-step workflow:

Step What Happens

1. Data Collection Gather data from databases, APIs, CSV files,
web scraping, sensors, etc.
2. Data Cleaning / Preprocessing Handle missing values, fix outliers, correct
data types, remove duplicates.
3. Exploratory Data Analysis (EDA) Summarise the data, create visualisations,
check correlations, understand patterns.
4. Model Building Apply statistical or machine learning models
to the prepared data.
5. Model Evaluation & Tuning Measure the model's performance (accuracy,
error rates) and improve it.
6. Reporting & Visualisation Present results clearly to stakeholders using
charts, dashboards, and narratives.

2.5 Data Gathering in Python

Reading Files
The most common way to load data in Python is using the pandas library:

import pandas as pd

# Load a CSV file

df = pd.read_csv('[Link]')

# Load an Excel file

df = pd.read_excel('[Link]', sheet_name='Sheet1')

# Load a text file with a custom separator

df = pd.read_csv('[Link]', delimiter='|')
 📌 DataFrame (df): A table-like data structure in Python's pandas library. Think of it as an
Excel spreadsheet but inside Python — rows and columns of data that you can filter, sort,
calculate, and visualise.

Connecting to Databases
import pandas as pd
import sqlalchemy

# Connect to a SQLite database

engine = sqlalchemy.create_engine('sqlite:///my_database.db')

# Write a SQL query

query = 'SELECT * FROM sales_table'

# Load query results into a DataFrame

df = pd.read_sql(query, engine)

APIs and Web Scraping

• APIs (Application Programming Interfaces): Many services (like Twitter, weather sites)
provide APIs that return data in JSON format. Python's requests library can fetch this data.
• Web Scraping: For websites that don't have APIs, Python tools like BeautifulSoup or Selenium
can extract data directly from HTML pages. Always check the website's Terms of Service before
scraping!

2.6 Data Cleaning in Python

Data cleaning (also called 'data wrangling' or 'data preprocessing') is the process of fixing problems in
raw data to make it suitable for analysis.

Initial Data Inspection

print([Link]()) # Show first 5 rows
print([Link]()) # Show last 5 rows
print([Link]) # (number of rows, number of columns)
print([Link]) # List of column names
print([Link]()) # Data types and non-null counts
print([Link]()) # Basic statistics for numeric columns

Handling Missing Values

Missing data is represented as NaN (Not a Number) in pandas. You have several choices for how to
deal with it:

# Count missing values in each column

 [Link]().sum()

# Option 1: Drop rows with any missing values (simple, but loses data)
df_clean = [Link]()

# Option 2: Fill missing values with a replacement

df_filled = [Link]({
'Age': df['Age'].mean(), # Fill with column average
'City': 'Unknown' # Fill text columns with a default
})

Removing Duplicates
# Find duplicate rows
duplicates = [Link]()
print(df[duplicates]) # Show the duplicates

# Remove all duplicate rows

df_clean = df.drop_duplicates()

Fixing Data Types

# Convert a column to numeric (bad values become NaN)
df['Salary'] = pd.to_numeric(df['Salary'], errors='coerce')

# Convert a column to datetime

df['HireDate'] = pd.to_datetime(df['HireDate'], format='%Y-%m-%d')
 LECTURE 3: Exploratory Data Analysis (EDA)

3.1 Key Concepts and Terminology

Before diving into EDA, it is important to understand some key terms that come up repeatedly in data
analytics and machine learning:

📌 Data: Raw facts and figures collected from various sources. Data alone has no meaning
without context and analysis.

📌 Attribute / Feature: A single characteristic or property of an observation. In a dataset

about students, attributes might include Name, Age, Grade, and Attendance. In machine
learning, these are usually called 'features' — the inputs used to make predictions.

📌 Variable: A quantity that can take different values. In a dataset, each column is typically
a variable. Variables can be: Continuous (e.g., height: 1.65m, 1.72m, 1.80m) or Categorical
(e.g., gender: Male, Female).

📌 Target Variable (Dependent Variable): The variable you are trying to predict or
explain. This is the 'output' or 'label' in machine learning. Example: In a house price
prediction model, the target variable is the house price.

📌 Predictive Model: A mathematical function or algorithm that takes input features and
produces a prediction of the target variable. Example: A model that takes a house's size,
location, and age and predicts its selling price.

3.2 What is Exploratory Data Analysis (EDA)?

📌 Exploratory Data Analysis (EDA): The process of visually and statistically examining a
dataset to understand its main characteristics, discover patterns, spot anomalies, and check
assumptions before building models. EDA is like getting to know your data before you start
working with it seriously.

EDA was pioneered by statistician John Tukey, who argued that analysing data without first exploring it
visually was like a detective trying to solve a crime without examining the scene.
The key goals of EDA are:
• Understand the distribution and spread of each variable.
• Identify relationships and correlations between variables.
• Detect outliers and unusual observations.
• Check if the data meets the assumptions required for modelling.
• Generate new hypotheses about the data.

EDA can be divided into three levels based on how many variables are being analysed at once:
• Univariate Analysis: Examining one variable at a time.
• Bivariate Analysis: Examining relationships between two variables.
• Multivariate Analysis: Examining relationships among three or more variables.

3.3 Handling Outliers

📌 Outlier: A data point that is significantly different from the rest of the data. Outliers can
be caused by data entry errors, measurement errors, or genuine extreme values in the real
world.

Outliers matter because they can badly skew statistical measures like the mean and can reduce the
accuracy of machine learning models.

How to Detect Outliers

• Box Plot / IQR Method: Values below Q1 - 1.5*IQR or above Q3 + 1.5*IQR are flagged as
outliers. (IQR = Interquartile Range = Q3 - Q1)
• Z-Score Method: Values more than 3 standard deviations from the mean are flagged.
• Visual inspection: Scatter plots and histograms often make outliers obvious.

Strategies for Dealing with Outliers

Strategy When to Use It
Remove the outlier When the outlier is clearly a data entry error
or measurement mistake.
Replace with mean/median When you want to keep the row but the
extreme value is unreliable.
Transform the variable Apply a mathematical transformation (e.g.,
log transform) to reduce the impact of
extreme values.
 Keep the outlier When the outlier is a genuine extreme value
that is meaningful and important to the
analysis (e.g., a fraudulent transaction really
is an extreme outlier — that's the point!).

💡 Key Point: Always investigate outliers before removing them! Sometimes what looks
like an error is the most important piece of information in your dataset.

3.4 Univariate Analysis

Univariate analysis examines ONE variable at a time. The goal is to understand that variable's
distribution, central tendency, and spread.

Descriptive Statistics
Measure Definition & Meaning
Mean (Average) The sum of all values divided by the number
of values. Simple to calculate but sensitive to
outliers.
Median The middle value when all values are sorted
in order. Less sensitive to outliers than the
mean. Use median for skewed data.
Mode The most frequently occurring value. Useful
for categorical data.
Standard Deviation (Std) Measures how spread out the values are
around the mean. A high std means values
vary a lot; a low std means they are clustered
close to the mean.
Variance The square of the standard deviation.
Another measure of spread.
Min / Max The smallest and largest values in the
dataset.
Quartiles (Q1, Q2, Q3) Q1 = 25th percentile, Q2 = median (50th
percentile), Q3 = 75th percentile. These
divide the data into four equal parts.
IQR (Interquartile Range) Q3 - Q1. The range of the middle 50% of the
data. Used in outlier detection.

In Python using pandas:

[Link]() # Gives count, mean, std, min, Q1, median, Q3, max
 df['Salary'].mean() # Mean salary
df['Salary'].median() # Median salary
df['Salary'].std() # Standard deviation
df['Salary'].mode() # Most common salary

Visualising Univariate Data

Chart Type What It Shows & When to Use
Histogram Shows the frequency distribution of a
continuous variable. How many values fall
into each 'bin' (range)? Use to see the shape
of the distribution (normal, skewed, bimodal).
Box Plot (Box-and-Whisker) Shows the five-number summary (min, Q1,
median, Q3, max) and highlights outliers as
dots outside the 'whiskers'. Great for
comparing distributions.
KDE Plot (Kernel Density Estimate) A smooth curve version of a histogram.
Shows the probability density of a variable.
Useful when you want a smooth picture of
the distribution.
Bar Chart Shows frequency of categories for
categorical variables. How many students
passed vs failed?

import [Link] as plt

import seaborn as sns

# Histogram
df['Salary'].hist(bins=20)
[Link]('Salary Distribution')
[Link]()

# Box Plot
[Link](x=df['Salary'])
[Link]()

# KDE Plot
[Link](df['Salary'])
[Link]()

3.5 Bivariate Analysis

Bivariate analysis examines the relationship between TWO variables at a time. The goal is to
understand whether and how two variables are related.

Correlation — The Key Measure of Relationship

 📌 Correlation: A statistical measure that describes the strength and direction of the linear
relationship between two numeric variables. Correlation ranges from -1 to +1.

Correlation Value Meaning

Close to +1 Strong positive relationship. As one variable
increases, the other also increases. Example:
Height and weight tend to correlate positively.
Close to -1 Strong negative relationship. As one variable
increases, the other decreases. Example:
Hours of TV watched and exam scores might
correlate negatively.
Close to 0 Little or no linear relationship between the
variables.

💡 Key Point: IMPORTANT: Correlation does NOT mean causation! Just because two
variables are correlated does not mean one causes the other. Example: Ice cream sales
and drowning rates are positively correlated — but ice cream doesn't cause drowning. Both
are caused by a third factor: hot weather.

# Calculate correlation between all numeric columns

[Link]()

# Correlation between two specific columns

df['Price'].corr(df['Quantity'])

Visualising Bivariate Data

Chart Type What It Shows
Scatter Plot Plots two numeric variables against each
other as dots. Each dot is one observation.
The pattern of dots reveals the relationship
(positive, negative, no relationship, non-
linear).
Line Chart Shows how one variable changes over time
(or along another ordered variable).
Grouped Bar Chart Compares one categorical variable broken
down by another category.
Pair Plot A grid of scatter plots for every pair of
numeric variables in the dataset. Useful for
 getting a quick overview of all relationships at
once.

# Scatter plot
[Link](df['Price'], df['Quantity'])
[Link]('Price')
[Link]('Quantity')
[Link]('Price vs Quantity')
[Link]()

# Pair plot (seaborn) — shows all variable relationships

[Link](df)
[Link]()

3.6 Multivariate Analysis

Multivariate analysis examines relationships among THREE or MORE variables simultaneously. This
gives a more complete picture of how different factors interact.

Key Techniques
Technique Description
Grouped Summaries Calculate summary statistics for subgroups of
data. Example: Average salary broken down
by both gender and department.
Crosstabs (Cross-tabulation) A table showing the frequency distribution of
two or more categorical variables
simultaneously. Like a pivot table of counts.
Pivot Tables Summarise data by grouping rows and
columns and computing aggregate values
(sum, mean, count, etc.).
Heatmap A colour-coded grid where each cell's colour
represents a numeric value. Excellent for
visualising correlation matrices — you can
instantly see which variables are strongly or
weakly correlated.
Facet Grid / Subplots Create multiple plots side-by-side, each for a
different subgroup, allowing visual
comparison.

# Correlation heatmap
import seaborn as sns

corr_matrix = [Link]() # Calculate correlations

[Link](corr_matrix, annot=True, cmap='coolwarm')
[Link]('Correlation Heatmap')
[Link]()

# Grouped summary (pivot table)

pivot = [Link](['Product', 'Region'])['Sales'].sum().unstack()
print(pivot)

# Cross-tabulation
[Link](df['Gender'], df['Department'])

✅ Real-World Example: A supermarket might use multivariate analysis to understand

which combination of product, store location, and time of day drives the highest sales. A
simple univariate analysis (just looking at total sales) would miss these important
interactions.
 LECTURE 4: Machine Learning

4.1 What is Machine Learning?

📌 Machine Learning (ML): A branch of artificial intelligence where computer systems

learn from data to make predictions or decisions without being explicitly programmed for
each specific task. Instead of telling the computer exactly what to do, you show it examples,
and it figures out the rules itself.

Think of it this way: teaching a child to recognise a cat. You don't give them a mathematical formula —
you show them many pictures of cats and non-cats, and they learn to recognise the pattern. Machine
learning works the same way.

Machine learning algorithms are divided into two main categories based on whether the training data is
labelled:

4.2 Supervised vs Unsupervised Learning

Supervised Learning Unsupervised Learning

Training data has LABELS (correct Training data has NO labels.
answers).
The algorithm learns to map inputs to The algorithm finds hidden patterns on its
outputs. own.
Think: Learning with a teacher. Think: Learning by exploring independently.
Tasks: Regression and Classification. Tasks: Clustering and Association.
Example: Predicting house prices given Example: Grouping customers by purchasing
size, location, age. behaviour without knowing the groups in
advance.

💡 Key Point: Supervised = you know the answer in advance and teach the model.
Unsupervised = you don't know the answer; the model discovers patterns itself.

4.3 Supervised Learning — Regression

 📌 Regression: A supervised learning task where the goal is to predict a CONTINUOUS
numeric output variable. The output can be any number within a range.

Examples of regression problems:

• Predicting a house's selling price.
• Predicting tomorrow's temperature.
• Predicting a student's exam score based on hours studied.

Simple Linear Regression

📌 Simple Linear Regression: The simplest form of regression. It models the relationship
between ONE input variable (X) and ONE output variable (Y) as a straight line.

The equation of the line is:

Y = mX + c
Y = predicted output | X = input feature | m = slope (gradient) | c = y-intercept (where the line
crosses the Y axis)

✅ Real-World Example: Predict exam score (Y) from hours studied (X). If m = 8 and c =
20, then: Y = 8X + 20. A student who studies 5 hours is predicted to score: Y = 8(5) + 20 =
60 marks.

The 'best fit line' is found using a method called Ordinary Least Squares (OLS) — it finds the line that
minimises the total squared difference between the predicted values and the actual values.

4.4 Cost Functions — Measuring Model Error

A cost function (also called a loss function) measures how wrong our model's predictions are compared
to the actual values. The lower the cost, the better the model.

Cost Function Formula & Explanation

MAE (Mean Absolute Error) Average of the absolute differences between
predicted and actual values. Easy to
understand — 'on average, my predictions
are X units off.' Less sensitive to outliers.
MSE (Mean Squared Error) Average of the SQUARED differences.
Squaring penalises large errors more heavily.
More sensitive to outliers than MAE.
 RMSE (Root Mean Squared Error) The square root of MSE. Brings the error
back to the same units as the target variable.
Most commonly used.
R² (R-Squared / Coefficient of Measures how much of the variance in Y is
Determination) explained by X. Ranges from 0 to 1. R² = 1
means perfect fit; R² = 0 means the model
explains nothing. A good model typically has
R² > 0.7.

💡 Key Point: R² is one of the most important metrics for regression models. If R² = 0.85, it
means your model explains 85% of the variation in the target variable — which is generally
considered very good!

4.5 Gradient Descent — How Models Learn

📌 Gradient Descent: An optimisation algorithm used to find the values of model

parameters (like m and c in linear regression) that minimise the cost function. It works by
iteratively adjusting parameters in the direction that reduces the error.

Imagine you are blindfolded on a hilly landscape and you want to reach the lowest point (the valley).
You can feel the slope under your feet, so you take small steps in the direction that goes downhill.
Eventually, you reach the bottom — the minimum error. That's gradient descent!

Key concepts in Gradient Descent:

• Learning Rate: Controls the size of each step. Too large and you might overshoot the minimum
and never converge. Too small and it takes forever to reach the minimum.
• Iterations/Epochs: The number of times the algorithm updates the parameters. More iterations
usually means better convergence (up to a point).
• Convergence: When the cost function stops decreasing significantly, the algorithm has
converged — it has found a good minimum.

✅ Real-World Example: Gradient Descent is used in virtually all modern machine learning
— from linear regression to deep neural networks. The neural networks that power
ChatGPT and image recognition were all trained using variants of gradient descent.

4.6 Supervised Learning — Classification

 📌 Classification: A supervised learning task where the goal is to predict a
CATEGORICAL (discrete) output — assigning observations to one of a fixed set of classes
or categories.

Examples of classification problems:

• Predicting whether an email is spam or not spam (binary classification).
• Predicting whether a bank transaction is fraudulent or legitimate.
• Diagnosing whether a patient has a disease (yes/no).
• Classifying a handwritten digit as 0, 1, 2, ..., 9 (multi-class classification).

Logistic Regression

📌 Logistic Regression: Despite having 'regression' in the name, this is actually a

CLASSIFICATION algorithm. It predicts the probability that an observation belongs to a
particular class, using the sigmoid function to map any value to a probability between 0 and
1.

The Sigmoid Function:

σ(z) = 1 / (1 + e^(-z))
Output is always between 0 and 1 — interpreted as a probability

If the sigmoid output is > 0.5, the observation is classified as Class 1 (e.g., 'spam'). If < 0.5, it is
classified as Class 0 (e.g., 'not spam').

Types of Logistic Regression

Type Use Case
Binomial / Binary Logistic Regression Only two possible output classes. Example:
Yes/No, Spam/Not Spam, Fraud/Legitimate.
Multinomial Logistic Regression Three or more output classes with no natural
order. Example: Classifying a film as Action,
Comedy, Drama, or Horror.
Ordinal Logistic Regression Three or more ordered output classes.
Example: Rating a restaurant as Poor,
Average, Good, or Excellent (there is a
natural order).

4.7 Unsupervised Learning — Clustering

 📌 Clustering: An unsupervised learning task that groups observations into clusters
(groups) based on their similarity. Observations within the same cluster are more similar to
each other than to observations in other clusters.

Examples of clustering:
• Grouping customers into segments based on purchasing behaviour (e.g., 'bargain hunters',
'premium buyers', 'occasional shoppers').
• Grouping news articles by topic automatically.
• Identifying groups of similar genes in biological data.

📌 K-Means Clustering: The most popular clustering algorithm. You specify the number of
clusters (K), and the algorithm assigns each data point to the nearest cluster centre, then
iteratively moves the centres until they stabilise.

✅ Real-World Example: A mobile network like Orange SL might use clustering to group
their subscribers into segments: heavy data users, mostly voice callers, occasional users,
and roaming business users. Each segment might then receive different marketing offers
tailored to their behaviour.

4.8 Unsupervised Learning — Association Rules

📌 Association Rule Learning: An unsupervised technique that discovers interesting

relationships (associations) between variables in large datasets. Most famously used in
'market basket analysis' — finding which products are frequently bought together.

✅ Real-World Example: Classic example: Analysis of supermarket transactions revealed

that customers who buy nappies (diapers) on Friday evenings also frequently buy beer. This
is a famous association rule: {Nappies} => {Beer}. This finding was used to place beer near
the nappy section to increase sales!

Key measures in Association Rules:

• Support: How often the items appear together in all transactions.
• Confidence: Given that someone bought item A, how likely are they to also buy item B?
• Lift: How much more likely are the items to be bought together compared to if they were bought
independently?
4.9 Data Partitioning — Training, Validation & Test Sets
When building a machine learning model, you should never evaluate your model on the same data you
used to train it — that would be like giving students the exam questions in advance. Instead, you split
your data into separate sets:

Dataset Purpose & Typical Size

Training Set Used to teach (train) the model. The model
sees this data and learns the patterns.
Typically 60-80% of the total data.
Validation Set Used to tune the model's settings
(hyperparameters) during development.
Helps you compare different model
configurations. Typically 10-20% of the data.
Test Set Used only at the very end, to get an honest
final evaluation of model performance on
completely unseen data. Never used during
training or tuning. Typically 10-20% of the
data.

💡 Key Point: A common split is 70% Training / 15% Validation / 15% Test, or simply 80%
Training / 20% Test (when a separate validation set is not used). The test set must NEVER
be used during model development — only once, at the very end, to report the final
performance.

Why Is This Important?

• Overfitting: When a model performs very well on training data but poorly on new data. It has
memorised the training data rather than learning general patterns. Think of a student who
memorises past exam papers but can't handle new questions.
• Underfitting: When a model is too simple to capture the underlying patterns. It performs poorly
even on training data. Think of a student who barely studied and can't answer any questions.
• Generalisation: The goal — a model that performs well on new, unseen data. Data splitting
and the test set help us measure this.

from sklearn.model_selection import train_test_split

# Split data into 80% training and 20% testing

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)

print(f'Training samples: {len(X_train)}')

print(f'Test samples: {len(X_test)}')
✅ Real-World Example: Building a spam email detector: Train the model on 8,000 emails
(labelled spam/not spam). Tune the model using 1,000 emails. Finally, test the finished
model on the remaining 1,000 emails to get an honest measure of how it will perform on
future emails it has never seen before.
 QUICK REFERENCE: Key Formulas & Definitions

Essential Formulas
Formula What It Calculates
Y = mX + c Simple Linear Regression: Predicts Y from X.
m = slope, c = intercept.
Mean = Σx / n Average: Sum of all values divided by count.
IQR = Q3 - Q1 Interquartile Range: Spread of the middle
50% of data.
Outlier if: x < Q1 - 1.5*IQR or x > Q3 + IQR-based outlier detection rule.
1.5*IQR
Correlation (r): -1 ≤ r ≤ +1 Strength and direction of linear relationship
between two variables.
R² = 1 - (SS_res / SS_tot) Proportion of variance explained by the
regression model.
σ(z) = 1 / (1 + e^(-z)) Sigmoid function: Converts any value to a
probability between 0 and 1.

Essential Definitions at a Glance

Term Simple Definition
Data Analytics Turning raw data into actionable insights to
support better decisions.
Big Data (3 Vs) Volume (huge amount), Variety (many types),
Velocity (fast speed).
Structured Data Organised in rows and columns (like a
spreadsheet).
Unstructured Data No fixed format (text, images, video).
Analysis Examines past data to explain what
happened and why.
Analytics Uses past data to predict future outcomes.
EDA Exploring data visually and statistically before
building models.
Correlation Statistical measure of the strength of
relationship between two variables.
Supervised Learning ML with labelled data. Learns to map inputs
to known outputs.
Unsupervised Learning ML without labels. Finds hidden patterns in
data.
Regression Predicts a continuous numeric value (e.g.,
price, temperature).
Classification Predicts a category (e.g., spam/not spam,
fraud/legitimate).
Clustering Groups similar observations together without
predefined labels.
Overfitting Model performs well on training data but
poorly on new data.
Training/Test Split Dividing data so the model is evaluated on
data it has never seen.
Gradient Descent Algorithm that iteratively reduces model error
by adjusting parameters.
Cost Function A measure of how wrong the model's
predictions are.
Logistic Regression Classification algorithm that predicts
probabilities using sigmoid function.