CSC 4792 (2024/25)

Data Mining and Warehousing

Lecture 2: Tools, Technologies & Services
for Data Mining and Warehousing
Lighton Phiri <[Link]@[Link]>
Department of Computing and Informatics
University of Zambia
[Link]
 Lecture Series Outline

● Part I: Jupyter Notebooks and Google Colaboratory

● Part II: Datasets and Python Core Libraries

July 19, 2025 CSC 4792 (2024/25) L02 - 2

 Lecture Series Outline

● Part I: Jupyter Notebooks and Google Colaboratory

○ Jupyter Notebooks Interface
○ Textual Content
○ Live Code
○ Visualisations
○ Google Colab
● Part II: Python Core Libraries

July 19, 2025 CSC 4792 (2024/25) L02 - 3

 About Jupyter Notebooks (1/2)

[Link]

July 19, 2025 CSC 4792 (2024/25) L02 - 4

 About Jupyter Notebooks (2/2)

● A notebook is a web application that contains descriptive

textual content, live code, equations and visualizations.
○ While we shall predominantly use the Python kernel, additional
kernels for other languages like R can be installed.
July 19, 2025 CSC 4792 (2024/25) L02 - 5
 About Jupyter Notebooks: Installation

● Installation instructions are available online (

[Link]

[Link]

July 19, 2025 CSC 4792 (2024/25) L02 - 6

 About Jupyter Notebooks: UI

● Text and live code is specified in cells and menubar and/or

toolbar are used to execute cell contents
● Output appears immediately below the input cell

July 19, 2025 CSC 4792 (2024/25) L02 - 7

 About Jupyter Notebooks: Markdown—Text
(1/4)

● Textual content is primarily specified using the markup

language “Markdown”
○ You essentially specify the structure of your text, similar to HTML

July 19, 2025 CSC 4792 (2024/25) L02 - 8

 About Jupyter Notebooks: Markdown—Text
(2/4)

● Common markup: Headings

○ # h1
○ ## h2
○ ### h3
○ #### h4

July 19, 2025 CSC 4792 (2024/25) L02 - 9

 About Jupyter Notebooks: Markdown—Text
(3/4)

● Common markup: Lists—Unordered

○ * Jupyter Notebooks
○ * Python 3
○ * Core Python Libraries

July 19, 2025 CSC 4792 (2024/25) L02 - 10

 About Jupyter Notebooks: Markdown—Text
(4/4)

● Common markup: Lists—Unordered

○ 1. Jupyter Notebooks
○ 2. Python 3
○ 3. Core Python Libraries

July 19, 2025 CSC 4792 (2024/25) L02 - 11

 About Jupyter Notebooks: Markdown—Code
(1/3)

● Python code, shell commands and magics are the most

common type of code
○ Python code is specified in its raw form in the cells

July 19, 2025 CSC 4792 (2024/25) L02 - 12

 About Jupyter Notebooks: Markdown—Code
(2/3)

● Python code, shell commands and magics are the most

common type of code
○ Shell commands are prefixed with “!”
○ Cell magics are prefixed with “%” or “%%”
○ Available magics specified with “%lsmagic”

July 19, 2025 CSC 4792 (2024/25) L02 - 13

 About Jupyter Notebooks: Markdown—Code
(3/3)
● Timing commands
○ %time—Measures execution time of a single line
○ %%time: Measures execution time of an entire cell
● Filesystem commands
○ All applicable bash commands (if on Linux)
● Other Languages
○ %%html—Renders a cell as HTML
○ %%javascript or %%js—Executes JavaScript code within a cell
○ %%markdown or %%md—Renders a cell as Markdown
○ %%latex—Renders a cell as LaTeX

July 19, 2025 CSC 4792 (2024/25) L02 - 14

 About Jupyter Notebooks: Markdown—
Visualisations

● Visualisations can be generated using plotting libraries like

matplotlib or using HTML via the “%%html” magic

July 19, 2025 CSC 4792 (2024/25) L02 - 15

 About Jupyter Notebooks: Markdown Guide
on Google Colab

[Link]

July 19, 2025 CSC 4792 (2024/25) L02 - 16

 Google Colab Interface (1/5)

● Google Colaboratory is a cloud-based alternative to Jupyter

Notebook

[Link]
July 19, 2025 CSC 4792 (2024/25) L02 - 17
 Google Colab Interface (2/5)

● Google Colaboratory is a cloud-based alternative to Jupyter

Notebook

[Link]
July 19, 2025 CSC 4792 (2024/25) L02 - 18
 Google Colab Interface (3/5)

● Google Colaboratory is a cloud-based alternative to Jupyter

Notebook

[Link]
July 19, 2025 CSC 4792 (2024/25) L02 - 19
 Google Colab Interface (4/5)

● Google Colaboratory is a cloud-based alternative to Jupyter

Notebook

[Link]
July 19, 2025 CSC 4792 (2024/25) L02 - 20
 Google Colab Interface (5/5)

● Google Colaboratory is a cloud-based alternative to Jupyter

Notebook

[Link]
July 19, 2025 CSC 4792 (2024/25) L02 - 21
 Lecture Series Outline

● Part I: Jupyter Notebooks and Google Colaboratory

● Part II: Datasets and Python Core Libraries
○ Datasets
○ Core Libraries

July 19, 2025 CSC 4792 (2024/25) L02 - 22

 Datasets: Predicting Student Learning
Outcomes (1/3)
● See “2024/25 CSC 4792” Moodle and/or Google Drive folder (
[Link]
o9ZrpLODX-KHsgL
)

July 19, 2025 CSC 4792 (2024/25) L02 - 23

 Datasets: Predicting Student Learning
Outcomes (2/3)

July 19, 2025 CSC 4792 (2024/25) L02 - 24

 Datasets: Predicting Student Learning
Outcomes (3/3)

July 19, 2025 CSC 4792 (2024/25) L02 - 25

 Datasets: University of Zambia Institutional
Repository Scholarly Research Output (1/3)
● See “2024/25 CSC 4792” Moodle and/or Google Drive folder (
[Link]
AhnR1LyBM/view?usp=sharing
)

July 19, 2025 CSC 4792 (2024/25) L02 - 26

 Datasets: University of Zambia Institutional
Repository Scholarly Research Output (2/3)

July 19, 2025 CSC 4792 (2024/25) L02 - 27

 Datasets: University of Zambia Institutional
Repository Scholarly Research Output (3/3)

July 19, 2025 CSC 4792 (2024/25) L02 - 28

 Pandas: What is Pandas?

● Pandas is a powerful, open-source Python library for data

manipulation and analysis
○ Name is derived from "Panel Data"
● Designed to work with structured (tabular) data
○ Data structure is similar to how data is organised in spreadsheets
or relational database management system tables
● The go-to tool for the most time-consuming part of data
science
○ data cleaning and exploration

July 19, 2025 CSC 4792 (2024/25) L02 - 29

 Pandas: Pandas vs Spreadsheets

● Imagine a spreadsheet
like Google Sheets
● The entire sheet (with
all its rows and columns)
is a DataFrame
● A single column from
that sheet is a Series
○ E.g. “StudentID” column
or “Quiz3” column

July 19, 2025 CSC 4792 (2024/25) L02 - 30

 Pandas: Core Objects—DataFrame and Series
(1/2)

● Series
○ A one-dimensional array-like object
○ Contains a sequence of values
○ Has an associated set of labels, called an index

July 19, 2025 CSC 4792 (2024/25) L02 - 31

 Pandas: Core Objects—DataFrame and Series
(2/2)

● Dataframe
○ A two-dimensional, tabular data structure
○ Has both a row and column index
○ Can be thought of as a dictionary of Series objects
○ Columns can be of different data types (number, text, date, etc.)

July 19, 2025 CSC 4792 (2024/25) L02 - 32

 Pandas: Getting Started in Your Environment

● Installation
○ Pandas is pre-installed in Google Colab and Anaconda
○ Manual installation in your local setup can be accomplished using
pip
pip install pandas
● The universal convention is to import Pandas with the alias
pd, prior to using pandas objects and methods
import pandas as pd

July 19, 2025 CSC 4792 (2024/25) L02 - 33

 Pandas: Loading Data Using The pd.read_*()
Family
● Pandas can read data from a huge variety of sources
● The functions follow a simple naming pattern: pd.read_*()
● Popular read functions
○ pd.read_csv() - The most common method
○ pd.read_excel() - For Microsoft Excel files
○ pd.read_html() - Reads tables directly from a webpage
○ pd.read_json() - For reading data from APIs
○ pd.read_sql() - For querying databases

July 19, 2025 CSC 4792 (2024/25) L02 - 34

 Pandas: Our Main Tool Shall be pd.read_csv()

● We will be working with two main datasets

○ db-unza25-csc4792-datasets-predicting_learning_outcomes.csv
○ db-unza25-csc4792-datasets-dspace_unza_zm.csv
● Example Code
Import pandas as pd
var_path = "path/to/your/[Link]"
var_learning_outcomes_df = pd.read_csv(var_path, sep='|')
[...]
help (pd.read_csv)

July 19, 2025 CSC 4792 (2024/25) L02 - 35

 Pandas: First Look at Your Data—Basic
Exploration
● Once data is ingested/loaded into a Pandas dataframe, there
is need to explore it
● Some commands you should always run
○ var_df.head() - Shows the first 5 rows
○ var_df.info() - Shows column data types and non-null counts. A
great way to spot missing data
○ var_df.describe() - Gives summary statistics (mean, median, std,
etc.) for numerical columns
○ var_df.shape - Shows the number of rows and columns

July 19, 2025 CSC 4792 (2024/25) L02 - 36

 Pandas: Asking Questions—Selection and
Filtering
● Selecting Columns (Getting a Series or smaller DataFrame)
○ Select one or more columns you are interested in
# Select the 'Gender' and 'FinalExamination' columns
var_subset_df = var_learning_outcomes_df[['Gender',
'FinalExamination']]
● Filtering Rows (Getting specific records)
○ Select rows that meet a certain condition
# Find all students whose minor is 'STEM'
var_stem_students_df = var_learning_outcomes_df[
var_learning_outcomes_df['MinorClassification'] == 'STEM'
]
July 19, 2025 CSC 4792 (2024/25) L02 - 37
 Pandas: Custom Operations Using .apply()
with Functions
● In instances where Pandas doesn't have built-in functions
○ You can create your own Python function and apply it to your
data.
○ The .apply() method is used to run a function on every value in a
Series (column).
○ Example Logic
# Define function
def fxn_classify_performance(var_score):
# Use .apply() to create a new column
var_df['Result'] =
var_df['FinalExamination'].apply(fxn_classify_performance)
July 19, 2025 CSC 4792 (2024/25) L02 - 38
 Pandas: Additional Operations for Common
Use Cases
● Some common operations
○ Converting to different file formats
○ df.to_csv([...])
○ df.to_excel([...])
○ df.to_sql([...])
○ df.to_*
○ Renaming columns
○ [Link](columns={[...]})
○ Aggregating data
○ [Link]([‘[...]’]).mean()
○ [Link](‘[...]’).max()

July 19, 2025 CSC 4792 (2024/25) L02 - 39

 Pandas: Summary and Hands-on
Walkthrough
● Pandas is the core library for working with structured data
● A DataFrame is like a spreadsheet, and a Series is like a
column
● We can load data from many sources using pd.read_*()
● Basic exploration is done with .head(), .info(), and .describe()
● We can select, filter, and transform our data
● See Google Colab notebook for hands-on walkthrough of all
these concepts at:
[Link]
rQ3iezGRF_uWZ3O6p?usp=drive_link
July 19, 2025 CSC 4792 (2024/25) L02 - 40
 Matplotlib: Why do we Visualise Data?

● A picture is worth a thousand

numbers
● Identify Patterns & Trends:
Can we see a difference in
performance between STEM
and Social Science students?
● Communicate Findings:
Which group, on average,
performed better on the final
exam?
July 19, 2025 CSC 4792 (2024/25) L02 - 41
 Matplotlib: What is Matplotlib

● The foundational and most widely-used plotting library for

Python
● It provides a low-level interface for creating a huge variety of
plots
● The Grandfather of Python Plotting
○ Many other libraries, like Seaborn, are built on top of Matplotlib
● Our focus will be on the pyplot module, which is the main
tool for creating figures and plots

July 19, 2025 CSC 4792 (2024/25) L02 - 42

 Matplotlib: Getting Started with the
Standard Import
● Environments like Google Colab and Anaconda come with
Matplotlib pre-installed
● If you have a basic Python setup, you will need to install it
yourself using pip and apt
○ Installation Command (in your terminal/command prompt):
○ Using pip
pip install matplotlib
○ Using apt
sudo apt-get install python3-matplotlib
○ To test the installation, open a Python interpreter and type:
import [Link]
July 19, 2025 CSC 4792 (2024/25) L02 - 43
 Matplotlib: Working with the Standard
Import
● Once installed, you must import the library into your script
or notebook
● The universal convention is to import the pyplot module
with the alias plt
import [Link] as plt
● You will also always need pandas to prepare the data for
plotting
import pandas as pd

July 19, 2025 CSC 4792 (2024/25) L02 - 44

 Matplotlib: The Anatomy of a
Plot/Graph/Chart (1/2)
● Figure
○ The entire window or
page that everything is
drawn on
● Axes
○ The actual plot or chart
area where you plot
your data
● Title
○ The main title of the
plot
July 19, 2025 CSC 4792 (2024/25) L02 - 45
 Matplotlib: The Anatomy of a
Plot/Graph/Chart (2/2)
● X-Axis Label
○ The label for the
horizontal axis
● Y-Axis Label
○ The label for the
vertical axis
● Legend
○ A key to identify
different data series on
the plot

July 19, 2025 CSC 4792 (2024/25) L02 - 46

 Matplotlib: Four-Step Plotting Heuristic

● Prepare the Data

○ Use Pandas to select, filter, and aggregate the data you want to
visualise
● Create the Plot
○ Call a plt function, like [Link]() or [Link](), passing in your
prepared data
● Customise
○ Add a title, axis labels, and other helpful context
● Show the Plot
○ Call [Link]() to display your final creation

July 19, 2025 CSC 4792 (2024/25) L02 - 47

 Matplotlib: Basic Plot Types—Bar Chart

● When to use it? To compare quantities across different

categories
● Example: Comparing the number of students sponsored by
different entities
● Core Function: [Link](categories, values)
# Data Preparation
var_sponsor_counts =
dataframe_learning_outcomes['Sponsor'].value_counts()

# Plotting
[Link](var_sponsor_counts.index, var_sponsor_counts.values)

July 19, 2025 CSC 4792 (2024/25) L02 - 48

 Matplotlib: Basic Plot Types—Line Plot

● When to use it? To show a trend or progression over a

continuous sequence, most commonly time
● Example: Tracking the number of publications in the UNZA
repository over the years
● Core Function: [Link](x_values, y_values)
# Data Preparation
var_publications_by_year =
dataframe_dspace['year'].value_counts().sort_index()

# Plotting
[Link](var_publications_by_year.index, var_publications_by_year.values)

July 19, 2025 CSC 4792 (2024/25) L02 - 49

 Matplotlib: Basic Plot Types—Histogram

● When to use it? To understand distribution variable

● Example: Visualising distribution of final examination scores
to see students scored in various ranges, e.g. 50-60
● Core Function: [Link](data_column, bins=...)
# Data Preparation
var_final_scores =
dataframe_learning_outcomes['FinalExamination'].dropna()

# Plotting
[Link](var_final_scores, bins=20)

July 19, 2025 CSC 4792 (2024/25) L02 - 50

 Matplotlib: Basic Plot Types—Scatter Plot

● When to use it? To investigate the relationship between two

numerical variables
● Example: Does a student's Test1 score have a relationship
with their FinalExamination score?
● Core Function: [Link](x_variable, y_variable)
# Plotting
[Link](
dataframe_learning_outcomes['Test1'],
dataframe_learning_outcomes['FinalExamination']
)

July 19, 2025 CSC 4792 (2024/25) L02 - 51

 Matplotlib: Summary and Hands-on
Walkthrough
● Visualisation is essential for finding insights in data.
● Matplotlib is the core plotting library in Python
● We can create fundamental plots like bar charts, line plots,
histograms, and scatter plots using a simple, four-step
process
● Customising plots with titles and labels is non-negotiable!
● See Google Colab notebook for hands-on walkthrough of all
these concepts at:
[Link]
HLNPq_SIxtvQirK3V
July 19, 2025 CSC 4792 (2024/25) L02 - 52
 Q & A Session

● Comments, concerns and complaints?

July 19, 2025 CSC 4792 (2024/25) L02 - 53

Bibliography

1. 2024/25 CSC 4792

[Link]
2. Daring Fireball: Markdown
[Link]
csc4792@[Link]
[Link]
YouTube Playlist ID: PLubb3AIPdPhYGt3ax8orjiDo3cuK0tGfx
 CSC 4792 (2024/25)
Data Mining and Warehousing
Lecture 2: Tools, Technologies & Services
for Data Mining and Warehousing
Lighton Phiri <[Link]@[Link]>
Department of Computing and Informatics
University of Zambia
[Link]