Data Analysis

Part 5

1
# Nodes: [Student1, Student2, Student3]
# Features: [Age, Grade], Node Feature Matrix (X)
node_features = [Link]([ Matrix Summation: sum(axis=1) sums
[15, 10], # Student 1 each row, giving the degree of each node.
[16, 11], # Student 2 sum(axis=0) would sum each column.
[14, 9] ]) # Student 3
X_nodes = [Link](node_features, columns=['Age', 'Grade'])
print("Node Feature Matrix (X):") Node Feature Matrix (X):
print(X_nodes) Age Grade
# Adjacency Matrix (A) representing friendships 0 15 10
1 16 11
# A_{ij} = 1 if friends, 0 otherwise.
2 14 9
adjacency_matrix = [Link]([
[0, 1, 1], # Student 1 is connected to 2 and 3 Adjacency Matrix (A):
[1, 0, 0], # Student 2 is connected only to 1 [[0 1 1]
[1, 0, 0] ]) # Student 3 is connected only to 1 [1 0 0]
[1 0 0]]
print("\nAdjacency Matrix (A):") Node Degrees (Number of
print(adjacency_matrix) friends): [2 1 1]
node_degrees = adjacency_matrix.sum(axis=1) # Sum across rows
print(f"\nNode Degrees (Number of friends): {node_degrees}")
# Student 1 has 2 friends, Students 2 and 3 have 1 each. 2
Example : Flight Network
•Nodes = Airports
•Edges = Direct flights between airports
Here:
•Karachi Airport is connected to Lahore and Dubai.
•Lahore is connected only to Karachi.
•Dubai is connected only to Karachi.
So, Karachi acts as a hub in this mini flight networ

3
Mathematical Primary Python
Python Code / Libraries
Concept Representation
Pandas
Data Matrix DataFrame, [Link](), [Link]()
NumPy Array
Label Vector y Pandas Series [Link]()
Descriptive Statistics Pandas Methods .mean(), .std(), .corr()
DataFrame/Series
Time Series Data pd.date_range(), [Link]
with DateTimeIndex

Adjacency Matrix A NumPy Array [Link]()

Node Feature
Pandas DataFrame [Link]()
Matrix X

4
Data and Information

5
Data and Information
•Journey from raw data → meaningful information has two
perspectives:

➢ Practical View → Meaning through context and

interpretation.

➢ Shannon’s Mathematical View → Quantifies uncertainty

in data.

Both perspectives are essential for understanding how data

becomes useful.

6
Practical View
Processed, organized, and structured data that carries
meaning and is useful for decision-making

Information = Processed + Organized + Structured Data that

is:
• Relevant
• Contextualized
• Interpreted for a specific purpose

Key Formula:
Information = Data + Context + Interpretation

7
•Information = Processed + Organized + Structured Data that
is:
• Relevant
• Contextualized
• Interpreted for a specific purpose
Key Formula:
Information = Data + Context + Interpretation
Raw Data: 38
With Context: 38°C (patient’s temperature)
With Interpretation: Fever → possible infection → doctor
takes action.
[ Raw Data ] → (Context & Interpretation) → [ Practical Information ]
[ Raw Data ] → (Entropy Measurement) → [ Shannon Information ]
8
Practical View
➢ Purpose-Driven → Directly supports decision-
making.
➢ Context-Sensitive → Without context, meaning is
lost.
➢ Interpretation-Dependent → Requires domain
knowledge.
Applications:
➢ Medicine: Interpreting lab results.
➢ Finance: Turning stock price trends into buy/sell
signals.
➢ Business: Analyzing customer feedback for service
improvement
9
Data vs Information: Two Perspectives
Why this matters:
•Data becomes valuable only after it gains meaning or is
quantified.
•Two complementary perspectives:
• Practical View – How raw data turns into useful insights.
• Shannon’s Mathematical View – How much uncertainty
is removed.

[ Raw Data ] → (Context & Interpretation) → [ Practical Information ]

[ Raw Data ] → (Entropy Measurement) → [ Shannon Information ]10
Shannon’s Mathematical View
Claude Shannon’s Information Theory:
•Information = Reduction of uncertainty.
•Quantified using entropy H(X):
Entropy (in Information
•H(X)=−∑p(x)log2p(x)H(X) Theory, introduced by
Claude Shannon) is a
Example: measure of uncertainty or
disorder in a system.
•Fair coin toss → 50-50 chance.
•Information per toss = 1 bit.
•Focuses on statistical unpredictability, not meaning.

11
 Entropy and knowledge
Entropy and Information Gain are super important in many
areas of machine learning, in particular, in the training of
Decision Trees.

Let’s say we have 3 buckets with 4 balls each. The balls have
the following colors:
Bucket 1: 4 red balls
Bucket 2: 3 red balls, and 1 blue ball
Bucket 3: 2 red balls, and 2 blue balls

12
[Link]
How much information we have on the color of a ball drawn at
random.
➢In the first bucket, we’ll know for sure that the ball coming out is
red.
➢In the second bucket, we know with 75% certainty that the ball
is red, and with 25% certainty that it’s blue.
➢In the third bucket, we know with 50% certainty that the ball is
red
Bucket 1 gives us the most amount of “knowledge” about what
ball we’ll draw (because we know for sure it’s red), that Bucket 2
gives us some knowledge, and that Bucket 3 will give us the least
amount of knowledge.
13
[Link]
Entropy is in some way, the opposite of knowledge.
➢ Bucket 1 has the least amount of entropy,
➢ Bucket 2 has medium entropy
➢ Bucket 3 has the greatest amount of entropy.
Entropy (in Information Theory, introduced by
Claude Shannon) is a measure of uncertainty or
disorder in a system.

14
[Link]
Mathematically, if we have possible outcomes with
probabilities p1,p2, pn, the entropy is:

➢If the outcome is certain (probability = 1 for one event),

entropy = 0.
➢If outcomes are equally likely, entropy is maximum.
➢Entropy is measured in bits (because of log base 2).

15
16
17
18
import numpy as np
def calculate_entropy(probabilities):
"""
Calculates the Shannon Entropy (in bits) for a given list of probabilities.
"""
# Convert the input to a numpy array
probs = [Link](probabilities)
# Filter out zero probabilities to avoid log2(0) which is undefined
probs = probs[probs > 0]
# Calculate the entropy: -sum(p_i * log2(p_i)) Fair Coin Entropy: 1.0000 bits
entropy = -[Link](probs * np.log2(probs)) Biased Coin Entropy: 0.7219 bits
return entropy Weather Entropy: 1.5000 bits
# Example : Fair Coin Toss
prob_fair_coin = [0.5, 0.5]
entropy_fair = calculate_entropy(prob_fair_coin)
print(f"Fair Coin Entropy: {entropy_fair:.4f} bits") # Output: 1.0000 bits
# Example : Biased Coin Toss
prob_biased_coin = [0.8, 0.2]
entropy_biased = calculate_entropy(prob_biased_coin)
print(f"Biased Coin Entropy: {entropy_biased:.4f} bits") # Output: ~0.7219 bits
# Example : Weather (Slide 13)
prob_weather = [0.5, 0.25, 0.25]
entropy_weather = calculate_entropy(prob_weather)
print(f"Weather Entropy: {entropy_weather:.4f} bits") # Output: 1.5000 bits 19
Apply to the Bucket Example: We have 3 buckets, each with 4 balls. We
want the entropy of "drawing a ball’s color".

20
Interpretation
➢ Bucket 1 → Entropy = 0 bits
→ No surprise. No new information is gained when drawing.
➢ Bucket 2 → Entropy = 0.81 bits
→ Some uncertainty. Each draw gives some information, but not
maximum.
➢ Bucket 3 → Entropy = 1 bit
→ Maximum uncertainty. Each draw gives the maximum possible new
information.
Linking to "Knowledge" and Decision Trees
➢ Low entropy = high knowledge (certainty).
Example: Bucket 1, no need to ask questions — we already know the
answer.
➢ High entropy = low knowledge (uncertainty).
Example: Bucket 3, we’re most uncertain, so learning the outcome gives
the most information gain.
This is why decision trees split data to reduce entropy — the goal is to
maximize information gain. 21
import numpy as np
def calculate_entropy(probabilities):
"""
Calculates the Shannon Entropy (in bits) for a given list of probabilities.
"""
# Convert the input to a numpy array
probs = [Link](probabilities)
# Filter out zero probabilities to avoid log2(0) which is undefined
probs = probs[probs > 0]
# Calculate the entropy: -sum(p_i * log2(p_i))
entropy = -[Link](probs * np.log2(probs))
return entropy

# Define the bucket compositions as probabilities

bucket1 = [1.0, 0.0] # p(red)=1.0, p(blue)=0.0
bucket2 = [0.75, 0.25] # p(red)=0.75, p(blue)=0.25
bucket3 = [0.5, 0.5] # p(red)=0.5, p(blue)=0.5
# Calculate their entropies
entropy_b1 = calculate_entropy(bucket1)
entropy_b2 = calculate_entropy(bucket2)
entropy_b3 = calculate_entropy(bucket3)
print("Bucket 1 Entropy:", entropy_b1, "bits") # 0.0 bits
print("Bucket 2 Entropy:", entropy_b2, "bits") # ~0.8113 bits
print("Bucket 3 Entropy:", entropy_b3, "bits") # 1.0 bits 22
# Let's visualize this relationship
import [Link] as plt

buckets = ['Bucket 1\n(4 Red, 0 Blue)', 'Bucket 2\n(3 Red, 1 Blue)', 'Bucket 3\n(2 Red, 2 Blue)']
entropies = [entropy_b1, entropy_b2, entropy_b3]

[Link](figsize=(8, 5))
[Link](buckets, entropies, color=['lightcoral', 'lightblue', 'lightgreen'])
[Link]('Entropy of Different Buckets')
[Link]('Entropy (bits)')
[Link]('Bucket Composition')
# Add value labels on top of each bar
for i, v in enumerate(entropies):
[Link](i, v + 0.01, f"{v:.2f}", ha='center', va='bottom')
[Link](0, 1.1)
[Link](axis='y', linestyle='--', alpha=0.7)
[Link]()

23
# Let's visualize this relationship
import [Link] as plt

buckets = ['Bucket 1\n(4 Red, 0 Blue)', 'Bucket 2\n(3 Red, 1 Blue)', 'Bucket 3\n(2 Red, 2 Blue)']
entropies = [entropy_b1, entropy_b2, entropy_b3]

[Link](figsize=(8, 5))
[Link](buckets, entropies, color=['lightcoral', 'lightblue', 'lightgreen'])
[Link]('Entropy of Different Buckets')
[Link]('Entropy (bits)')
[Link]('Bucket Composition')
# Add value labels on top of each bar
for i, v in enumerate(entropies):
[Link](i, v + 0.01, f"{v:.2f}", ha='center', va='bottom')
[Link](0, 1.1)
[Link](axis='y', linestyle='--', alpha=0.7)
[Link]()

24