Regex & Data Slicing

1. Regular Expressions (Regex)

Regular Expressions (Regex) are patterns used to match, search, and

manipulate strings. They allow identifying text formats such as emails, phone
numbers, dates, tokens, and specific character sequences. Regex uses
meta-characters, character classes, quantifiers, anchors, and groups to
define complex search patterns.
Common symbols include . (any character), * (0 or more), + (1 or more), []
(character set), () (grouping), ^ (start), $ (end), and predefined classes like \d
(digits), \w (alphanumeric), and \s (whitespace).
Regex is widely used in pattern matching, input validation, text mining,
compiler design, NLP, and string manipulation.

Example:

● Regex for email → ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$

● Extract numbers → \d+

Language Support:

● Python: Uses the re module.

● R: Uses functions like grep, grepl, sub, and gsub.

i. Core Components: Metacharacters

Metacharacters are characters with special meanings. They are the building
blocks of Regex.

Symbol Name Description Example Matches

. Dot Matches any single a.b acb, a@b, a9b

character (except newline).

^ Caret Anchors to the start of the ^Data Data Science (not

string. Big Data)
$ Dollar Anchors to the end of the py$ numpy, scipy
string.

* Star Matches 0 or more ab* a, ab, abbb

repetitions.

+ Plus Matches 1 or more ab+ ab, abbb (NOT a)

repetitions.

? Question Matches 0 or 1 repetition colou?r color, colour

(Optional).

[] Brackets A set of characters [a-z] Any lowercase

(Character Class). letter

` ` Pipe Logical `cat

OR.

\ Backslash Escape character. Treats \. A literal dot .

next char literally.

() Parentheses Grouping and capturing. (ab)+ ab, abab

ii. Special Sequences (Character Classes)

Shortcuts to represent common data types.

Sequence Meaning Python Equivalent

\d Any Digit (0-9). [0-9]

\D Any Non-digit. [^0-9]

\w Any Word character (a-z, A-Z, [a-zA-Z0-9_]

0-9, _).

\W Any Non-word character [^a-zA-Z0-9_]

(Symbols, spaces).

\s Any Whitespace (space, tab, [ \t\n]

newline).

\S Any Non-whitespace. --

\b Word Boundary. (Start/End of a e.g., \btest\b matches "test" but

word). not "testing".

iii. Python re Module Functions (The "Big 4")

In Data Analytics, you typically use these four functions:

A. [Link]() vs [Link]() (Exam Trap)

● [Link](pattern, string): Checks for a match ONLY at the beginning of
the string.
● [Link](pattern, string): Scans the ENTIRE string and returns the first
match found.

Python:
import re
text = "Learn Python for Data"

# match() fails because text doesn't start with "Python"

print([Link](r'Python', text)) # Output: None

# search() succeeds
print([Link](r'Python', text)) # Output: <Match object>
B. re.findall() (Most Used in Analytics)
Returns all non-overlapping matches as a list of strings.

● Use Case: Extracting all hashtags or email addresses from a document.

Python:
text = "Contact us at support@[Link] or help@[Link]"
emails = re.findall(r'\S+@\S+', text)
print(emails)
# Output: ['support@[Link]', 'help@[Link]']

C. [Link]() (Substitution)
Replaces matches with a replacement string.

● Use Case: Cleaning data (removing special characters).

Python:
phone = "(123)-456-7890"
# Replace anything that is NOT a digit (\D) with an empty string
clean_phone = [Link](r'\D', '', phone)
print(clean_phone)
# Output: '1234567890'

iv. R Language Comparison

Task Python Function R Function

Find if pattern exists [Link]() grepl() (returns logical

TRUE/FALSE)

Find indices of match [Link]().span() grep() (returns indices)

Replace first match [Link](count=1) sub()

Replace all matches [Link]() gsub()

Data Slicing
Data Slicing refers to extracting a subset of data from sequences like strings,
lists, tuples, or from structured collections like NumPy arrays and Pandas
DataFrames.
It uses the general syntax:

sequence[start : stop : step]

Here, the start index is inclusive, stop is exclusive, and step defines the jump.
Slicing supports positive and negative indexing and enables operations like
reversing sequences ([::-1]), selecting ranges, skipping elements, and extracting
rows/columns in arrays.

Data slicing is crucial in data preprocessing, data analysis, machine learning,

array manipulation, and feature extraction.

Examples:

● s[0:3] → first 3 characters

● arr[1:4] → subarray in NumPy
● [Link][:5] → first 5 rows of a DataFrame

i. The Python Indexing Rules

1. Zero-Based Indexing: The first element is at index 0.
2. Exclusive Stop: The range includes the start index but excludes the stop
index.
3. Negative Indexing: Python allows counting from the end. -1 is the last
item, -2 is the second to last.

ii. Slicing Sequences (Lists & Strings)

Given List: L = [10, 20, 30, 40, 50, 60, 70]

A. Basic Range
● L[1:4] Starts at index 1, stops before 4.
 ○ Result: [20, 30, 40]

B. Omitting Indices (Defaults)

● L[:3] Start (0) to 3. Result: [10, 20, 30]
● L[4:] 4 to End. Result: [50, 60, 70]
● L[:] Full copy of the list.

C. Step Slicing
● L[::2] Every 2nd element (0, 2, 4...).
○ Result: [10, 30, 50, 70]

D. Negative Slicing (Reversing)

● L[-3:] Last 3 elements. Result: [50, 60, 70]
● L[::-1] Reverses the list. Result: [70, 60, ... 10]

iii. Slicing in Pandas (DataFrames)

This is the most critical topic for the Data Analytics portion of the exam.
Pandas provides two distinct slicing methods.

A. .iloc[] (Integer Location)

● Logic: Purely integer-based indexing (like standard lists).
● Syntax: [Link][row_indices, column_indices]
● Rule: The stop index is exclusive.

Python
# Select rows 0 to 4 (exclusive) and columns 0 to 2 (exclusive)
[Link][0:4, 0:2]

B. .loc[] (Label Location)

● Logic: Label-based indexing (uses row names/index and column
headers).
● Syntax: [Link][row_labels, column_names]
● Rule (CRITICAL): The stop bound is INCLUSIVE in .loc.

Python
# Select rows with index 'A' through 'C' and specific columns
[Link]['A':'C', ['Name', 'Age']]
iv. Comparison Table:

Feature .iloc .loc

Input Integers only (0, 1, 2) Labels/Strings ('Name',

'2023-01-01')

Stop Exclusive (n-1) Inclusive (n)

Bound

Use Case "Give me the first 5 "Give me data for 'January'."

rows."

v.) Slicing in NumPy (Multi-dimensional)

Used for numerical data reshaping.

● Syntax: arr[row_slice, col_slice]

● Example: arr[:2, 1:3] (First 2 rows, columns 1 to 2).

vi.) R Slicing (The "Gotcha" for Python Users)

Since the syllabus mentions R, you must know the fundamental difference.

1. One-Based Indexing: R starts counting at 1, not 0.

2. Inclusive Logic: Slicing 1:3 in R includes element 1, 2, and 3.

Concept Python Code R Code

First Element list[0] vector[1]

Slice First 3 list[0:3] (Stop is vector[1:3] (Stop is inclusive)
exclusive)

Exclude Index 1 (Complex logic needed) vector[-1] (Negative sign removes

index)

vi.)Dictionary Slicing

Dictionaries don’t support slicing directly.

We slice keys/values:

d = {'a':1,'b':2,'c':3}
list([Link]())[0:2]

vii.) Dataframe Column Slicing

df[['Name','Age']]
 Lists, Dictionaries & Sets
Lists (Python)
1. Overview
A List is Python’s most versatile way to store a collection of items. Think of it as
a container that holds items in a specific order.

● It is Mutable (you can change the contents).

● It is Ordered (items stay in the position you put them).
● It creates a sequence using Square Brackets [].

2. Key Features & Concepts

A. Heterogeneous Nature

Unlike arrays in C or Java, a Python List can hold different types of data at the
same time.

● Example: my_list = [1, "SEBI", 3.5, True]

● It can even hold another list inside it (Nested List): nested = [1, [2, 3], 4]

B. Indexing (How to grab data)

Python gives every item a "seat number" (Index).

● Positive Indexing: Starts from 0 (left to right).

● Negative Indexing: Starts from -1 (right to left). This is useful to grab the
last item without knowing the list length.
● Example: L = ['A', 'B', 'C', 'D']
○ L[0] is 'A'
○ L[-1] is 'D'

C. Slicing (Extracting a chunk)

Slicing allows you to get a sub-list.

● Syntax: list[start : stop : step]

● Crucial Rule: The stop index is excluded (it stops before that index).
● Example: nums = [0, 1, 2, 3, 4, 5]
○ nums[1:4] [1, 2, 3] (Starts at index 1, stops before index 4).
 ○ nums[::-1] [5, 4, 3, 2, 1, 0] (Reverses the list).

3. Important List Methods (The Tools)

You must know the difference between these for the exam.

Method Description Example Result

append(x) Adds x to the end of the list. [Link](9) [..., 9]

insert(i, x) Puts x at specific index i. [Link](0, 9) [9, ...]

extend(iter) Joins a new list to the old one. [Link]([8,9]) [..., 8, 9]

pop(i) Removes & returns item at [Link](1) Removes

index i. If i is empty, removes item at index
last item. 1.

remove(x) Finds the first occurrence of [Link](5) Removes the

value x and deletes it. number 5.

count(x) Counts how many times x [Link](2) Returns

appears. integer.

sort() Sorts the list in-place [Link]() Ascending

(changes original). order.

4. Deep Dive: List Comprehension

Syntax: [expression for item in iterable if condition]
The Long Way:
squares = []
for x in range(5):
[Link](x**2)
# Result: [0, 1, 4, 9, 16]
The List Comprehension Way (Exam Preferred):
squares = [x**2 for x in range(5)]
# Result: [0, 1, 4, 9, 16]

5. Exam Trap: Aliasing vs. Cloning

The Trap (Aliasing):

If you assign one list to another variable using =, they point to the same object
in memory.

Python
A = [1, 2, 3]
B=A # B is just a nickname for A
B[0] = 99
print(A) # Output is [99, 2, 3] -> A changed too!

The Solution (Cloning):

To create a separate independent copy, you must "clone" it.

Python
A = [1, 2, 3]
B = [Link]() # OR use slicing: B = A[:]
B[0] = 99
print(A) # Output is [1, 2, 3] -> A is safe.
Dictionaries (Python)
1. Overview
A Dictionary is Python’s built-in "mapping" type. Instead of using numbers
(indexes) to find data like a List, Dictionaries use Keys (unique labels) to find
Values.
 ● Think of it like a real dictionary: You look up a word (Key) to find its
definition (Value).
● It is Mutable (you can change it).
● It is Unordered
● Syntax: Uses Curly Braces {} with a colon : separating the key and value.

2. Key Features & Concepts

A. The Structure (Key-Value Pairs)

Every item in a dictionary is a pair.

● Key: The label. It must be Unique (no duplicates) and Immutable

(unchangeable, like a String, Integer, or Tuple).
● Value: The data. It can be anything (Lists, Numbers, duplicates allowed).

B. Syntax Example

Python
student = {
"Name": "Rahul", # String Key
"Roll_No": 101, # Integer Key
"Marks": [90, 85, 88] # List as Value
}

C. Accessing Data

You don't use student[0]. You use the key name.

● print(student["Name"]) Output: "Rahul"

● print(student["Marks"]) Output: [90, 85, 88]

3. Important Dictionary Methods (The Tools)

These are the most tested methods in IT officer exams.

Method Description Example Result

keys() Returns a list-like view of all [Link]() ['Name',
Keys. 'Roll_No']

values() Returns a list-like view of all [Link]() ['Rahul', 101]

Values.

items() Returns a list of (Key, [Link]() [('Name',

Value) tuples. 'Rahul'), ...]

get(key) Safest way to fetch a value. [Link]("Age") None (Doesn't

Returns None if key is crash)
missing (no error).

update(othe Merges another dictionary [Link]({'Age': Adds 'Age' to d.

r) into the current one. 25})

pop(key) Removes the key and [Link]("Name") Removes

returns its value. "Name", returns
"Rahul".

4. Deep Dive: Why can't a List be a Key? (Hashing)

To find data quickly, Python takes the Key and runs it through a mathematical
formula called a Hash Function. This formula turns the Key into a unique ID
number (Hash).

● The Rule: To have a Hash ID, the object must be permanent (immutable).
● The Problem: A List is not permanent; you can add/remove items from it.
If the content changes, the Hash ID would change. Python would lose
track of where the data is stored.
● Conclusion: Therefore, Python forbids Lists (mutable) from being Keys.
You will get a TypeError: unhashable type: 'list'.
5. Exam Trap: The KeyError
The Trap:

Trying to access a key that doesn't exist using square brackets.

Python
data = {"A": 1, "B": 2}
print(data["C"]) # CRASH! Raises KeyError: 'C'

The Fix (Best Practice):

Always use .get() if you aren't sure the key exists.

Python
print([Link]("C")) # Output: None (Safe)
print([Link]("C", 0)) # Output: 0 (You can set a default value)

Sets (Python)
1. Overview
A Set is a collection of items that is unordered and unindexed.

● Think of it like a bag of marbles: You can reach in and grab a marble,
but there is no "first" or "last" marble, and you can't have two identical
marbles.
● It is mainly used for mathematical operations (like finding common
items) and removing duplicates.
● Syntax: Uses Curly Braces {}, just like dictionaries, but without Key-Value
pairs.

2. Key Features & Concepts

A. Uniqueness (No Duplicates)

This is the defining feature of a Set. It automatically deletes duplicates.

Example:
Python
my_set = {1, 2, 2, 3, 3, 3}
print(my_set)
# Output: {1, 2, 3}
B. Unordered & Unindexed

Because sets have no order, you cannot use an index to find items.

● my_set[0] Error! (TypeError: 'set' object is not subscriptable).

● You cannot slice a set.

C. Mutable

You can add or remove items from a set, but the items inside the set must be
immutable (like numbers, strings, or tuples). You cannot put a List inside a Set.

[Link] Set Operations

Operation Symbol Python Method

Union A∪B [Link](B) or `A

Intersection A∩B [Link](B) or A & B

Difference A–B [Link](B) or A - B

Symmetric Difference A Δ B A.symmetric_difference(B) or A

^ B
Subset A⊆B [Link](B)

Superset A⊇B [Link](B)

4. Important Set Methods (The Tools)

Method Description Example

add(x) Adds item x to the set. [Link](5)

update(iter) Adds multiple items from a list/set. [Link]([6, 7])

 remove(x) Removes x. Raises Error if x is missing. [Link](5)

discard(x) Removes x. Does NOT raise error if x is [Link](99)

missing. (Safe)

clear() Removes all items, leaving an empty set. [Link]()

5. Deep Dive: The "Speed" Secret

Why use a Set instead of a List? Speed.

● Scenario: You have a list of 1 million ID numbers, and you need to check
if ID 999 is in there.
○ List: Python must scan every single number one by one until it finds
999. This is slow ( ).
○ Set: Python hashes 999, calculates exactly where it should be in
memory, and goes straight there. It doesn't scan. This is instant (
).

Conclusion: If you need to check "Is X present in Y?" frequently, always convert
Y to a Set first.

6. Exam Trap: The Empty Set Confusion

This is a classic trick question.

The Trap:

Since both Dictionaries and Sets use curly braces {}, how do you make an empty
one?

Python
x = {} # This creates an empty DICTIONARY (default)
y = set() # This creates an empty SET

● If you write x = {} and then try [Link](1), you will get an error because
Dictionaries don't have an .add() method.
 DataFrames & Reshaping

DataFrames in Python

● Imagine a DataFrame as a table in a spreadsheet (like Excel or Google

Sheets). It has:
○ Rows: Each row is like one "record" or "item" (e.g., one person's
info).
○ Columns: Each column is like a "category" or "label" (e.g., Name,
Age, City).
● Why use it? Raw data is often messy (lists or files). A DataFrame makes it
structured, easy to read, filter, and analyze. It's perfect for data
analytics because you can do math, sort, or visualize it quickly.
● In Python, we use the pandas library to create DataFrames. (Pandas is
free and comes with most Python setups for data work.)
● Key idea: DataFrames hold labeled data – rows and columns have names,
so you don't get lost in numbers.

Real-life analogy: A DataFrame is like a class attendance sheet:

● Rows: Students (e.g., Alice, Bob).

● Columns: Details (Name, Age, Grade).

Why Use It?

● Handles messy real data (e.g., SEBI compliance logs).

● Easy import/export (CSV, Excel).
● Ties to syllabus: Slicing (select rows/cols), dictionaries (create from dicts),
lists (add data).

How to Set Up Python for DataFrames

Before we start, you need:

1. Install Python (free from [Link]).

2. Install pandas: Open your command prompt/terminal and type pip install
pandas.
3. Use a simple editor like Jupyter Notebook (free online via Google Colab)
to run code.
In your code, always start with:

import pandas as pd # 'pd' is a shortcut name for pandas

Creating DataFrames

Start simple—build from basics.

1. From Dictionaries (Ties to "Dictionaries" in Syllabus)

Dicts: Keys = column names, values = lists (rows)

import pandas as pd
data = {
'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35],
'Salary': [50000, 60000, 70000]
}
df = [Link](data)
print(df)

Output:

Name Age Salary

0 Alice 25 50000
1 Bob 30 60000
2 Charlie 35 70000

R Equivalent:

data <- [Link](

Name = c("Alice", "Bob", "Charlie"),
Age = c(25, 30, 35),
Salary = c(50000, 60000, 70000)
)
print(data)

2. From Lists of Lists (Ties to "Lists")

Each inner list = a row; add column names separately.

rows = [
['Alice', 25, 50000],
['Bob', 30, 60000],
['Charlie', 35, 70000]
]
df = [Link](rows, columns=['Name', 'Age', 'Salary'])
print(df) # Same output as above

3. Empty or Random Data (For Testing)

df_empty = [Link](columns=['A', 'B']) # Empty with cols
print(df_empty)
import numpy as np
df_random = [Link]([Link](3, 2), columns=['X', 'Y'])
print(df_random)

4. From Series (Single Column)

s = [Link](['Apple', 'Banana', 'Cherry'])
df = [Link]({'Fruit': s})
print(df)

Analytics Tip: Use dicts for structured data (e.g., JSON imports); lists for raw
logs.

Viewing and Inspecting DataFrames

Method What It Does Example Output

[Link](n) First n rows (default 5). Top 5 rows.

[Link](n) Last n rows. Bottom 5 rows.

[Link] (rows, cols) tuple. (3, 3)

[Link]() Data types, non-null Memory usage, types.

counts.
[Link]() Stats (mean, std, etc.) Table of
for nums. min/max/avg.
[Link] Column types (int, str, Name: object, Age:
etc.). int64

Example:

print([Link](2)) # First 2 rows

print([Link]) # (3, 3)
print([Link]()) # Stats on Age/Salary

R Equivalents:

Python R

[Link]() head(df)

[Link] dim(df)

[Link]( summary(df
) )

Accessing and Selecting Data

Slicing: Grab subsets like df[0:2] for rows.

1. By Column (Label-Based)

● Single: df['Column'] (Series) or [Link] (dot notation—avoid spaces).

● Multiple: df[['Col1', 'Col2']].

names = df['Name'] # Series: Alice, Bob, Charlie

subset = df[['Name', 'Salary']] # DataFrame with 2 cols
print(names)
print(subset)

2. By Row (Index-Based Slicing)

● [Link][0:2] (integer position, like lists).

● [Link][0:2] (label-based, inclusive end).

first_two = [Link][0:2] # Rows 0-1

print(first_two)
# With conditions (boolean slicing)
high_salary = df[df['Salary'] > 55000] # Rows where Salary > 55k
print(high_salary) # Bob and Charlie

3. By Row and Column

value = [Link][1, 'Age'] # Row 1, Age col: 30
sub_slice = [Link][0:2, 0:2] # Rows 0-1, cols 0-1
print(value)
print(sub_slice)

Analytics Tip: Use .loc for labels (e.g., dates as indices); .iloc for positions.
Boolean slicing shines for filtering (e.g., stocks > threshold).

Basic Operations on DataFrames

● Add Column: df['New'] = values.

df['Bonus'] = df['Salary'] * 0.1 # 10% bonus

print(df)

● Drop Column/Row: [Link]('Col', axis=1) or axis=0 for rows.

df_no_age = [Link]('Age', axis=1)

print(df_no_age)

● Sort: df.sort_values('Col').

sorted_df = df.sort_values('Salary', ascending=False)

print(sorted_df) # Charlie first

● Missing Data: Check [Link](), fill df.fillna(0).

[Link][0, 'Age'] = None # Add missing

print([Link]().sum()) # Counts NaNs per col
df_filled = df.fillna({'Age': 0}) # Fill Age with 0

R Equivalents:

Operation R
Add Col df$Bonus <- df$Salary * 0.1

Drop Col df[, -which(names(df) == "Age")]

Sort arrange(df, desc(Salary)) (dplyr)

Missing [Link](df); df[[Link](df$Age), "Age"] <- 0

Importing and Exporting DataFrames

Load/save like files—perfect for CSV/Excel in analytics.

Format Import Export

CSV pd.read_csv('fi[Link]') df.to_csv('fi[Link]', index=False)

Excel pd.read_excel('fi[Link]') df.to_excel('fi[Link]', index=False)

JSON pd.read_json('fi[Link]') df.to_json('fi[Link]')

Example (CSV):

# Assume '[Link]' exists with columns

df_from_csv = pd.read_csv('[Link]')
print(df_from_csv.head())
df.to_csv('[Link]', index=False) # No row indices

Options: read_csv(na_values=['?']) for custom missing; encoding='utf-8' for

special chars..

Data Reshaping in Python

What is Data Reshaping?

● Reshaping means reorganizing your table's structure:

○ Wide format: Many columns, fewer rows (e.g., months as separate
columns: Jan_Sales, Feb_Sales).
○ Long format: Fewer columns, more rows (e.g., one row per month:
Month column with 'Jan', 'Feb').
● Why do it?
 ○ Easier analysis: Long format is great for charts (one data point per
row) or stats (e.g., average sales over time).
○ Data cleaning: Raw files (CSVs) often come wide; models need
long.
○ Comparisons: Pivot to spread data for quick side-by-side views.
● Analogy: Imagine a recipe book. Wide: Ingredients listed as columns
(Flour: 2 cups, Sugar: 1 cup). Long: One row per ingredient (Ingredient:
Flour, Amount: 2 cups). Reshape to switch views!
● Key Warning: Reshaping doesn't change the total data – just the shape.
Always check with [Link] (rows, columns).

Common tools: pivot(), melt(), pivot_table(), stack(), unstack().

2. Pivot: Turn Long Data into Wide (Spread It Out)

● What it does: Like Excel's "Pivot Table" – rotates data so one column
becomes rows (index), another becomes new columns, and values fill in.
● When to use: Summarize groups (e.g., sales by product across months).
● Basic Syntax: [Link](index='row_col', columns='new_col_source',
values='fill_col')
○ index: What goes on the left (rows).
○ columns: What spreads across the top (new headers).
○ values: The numbers/text that go in cells.

Example 1: Simple Pivot (No Duplicates)

Start with long data (tall and skinny – repeated categories).

import pandas as pd

# Long data: Sales for fruits over months (one row per combo)

long_data = {

'Fruit': ['Apple', 'Apple', 'Banana', 'Banana', 'Orange', 'Orange'],

'Month': ['Jan', 'Feb', 'Jan', 'Feb', 'Jan', 'Feb'],

'Sales': [100, 150, 80, 120, 90, 110]

}
df_long = [Link](long_data)

print("Original Long Data:\n", df_long)

Output:

Fruit Month Sales

0 Apple Jan 100

1 Apple Feb 150

2 Banana Jan 80

3 Banana Feb 120

4 Orange Jan 90

5 Orange Feb 110

● Shape: (6 rows, 3 columns) – Lots of rows because months repeat.

Now pivot to wide (months as columns):

df_wide = df_long.pivot(index='Fruit', columns='Month', values='Sales')

print("Pivoted Wide Data:\n", df_wide)

Output:

Month Feb Jan

Fruit
Apple 150 100
Banana 120 80
Orange 110 90

● Shape: (3 rows, 3 columns) – Fewer rows, more columns. Easy to compare

Jan vs. Feb!
● What happened? Fruits are rows, months spread across, sales fill cells.

If Duplicates? Use Pivot_Table

● Pivot fails on duplicates (e.g., two 'Jan' for Apple). pivot_table()

averages/sums them.
 ● Syntax: df.pivot_table(index='...', columns='...', values='...', aggfunc='mean')
(or 'sum', 'count').

Quick Example (Add duplicate data):

# Add a duplicate row for Apple Jan

df_long_dup = [Link]([df_long, [Link]({'Fruit': ['Apple'], 'Month':

['Jan'], 'Sales': [105]})], ignore_index=True)

# Pivot table to average duplicates

df_wide_avg = df_long_dup.pivot_table(index='Fruit', columns='Month',

values='Sales', aggfunc='mean')

print(df_wide_avg)

Output (Apple Jan averages 100 and 105 → 102.5):

Month Feb Jan

Fruit
Apple 150.0 102.5
Banana 120.0 80.0
Orange 110.0 90.0

● Pro: Handles messiness. Con: Pick aggfunc wisely (mean for averages,
sum for totals).

Analogy: Pivot is like a calendar: Days (months) across top, events (fruits) down
side, notes (sales) in boxes.

3. Melt: Turn Wide Data into Long (Stack It Up)

● What it does: Opposite of pivot – "melts" columns into rows. Collapses

wide tables into long lists.
● When to use: Prep for graphs (e.g., line chart needs time in rows) or when
data is too spread out.
● Basic Syntax: [Link](id_vars=['keep_cols'], var_name='new_col_name',
value_name='new_value_col')
○ id_vars: Columns to keep unchanged (e.g., 'Fruit').
 ○ var_name: Name for the melted column headers (e.g., 'Month').
○ value_name: Name for the cell values (e.g., 'Sales').

Example: Melting the Wide Data from Above

Using our df_wide from pivot:

# Reset index so 'Fruit' is a column again (pivot made it row labels)

df_wide_reset = df_wide.reset_index()

# Now melt

df_melted = df_wide_reset.melt(id_vars=['Fruit'], var_name='Month',

value_name='Sales')

print("Melted Long Data:\n", df_melted)

Output:

Fruit Month Sales

0 Apple Feb 150
1 Banana Feb 120
2 Orange Feb 110
3 Apple Jan 100
4 Banana Jan 80
5 Orange Jan 90

● Shape: Back to (6 rows, 3 columns). Now it's long – perfect for time-series
analysis.
● What happened? Old columns ('Jan', 'Feb') became rows in 'Month';
values went to 'Sales'.

Analogy: Melting is like stacking pancakes flat into a pile – wide pan becomes
tall stack. Easy to add syrup (analysis) one by one.

4. Stack and Unstack: For Layered (Hierarchical) Data

● These are for multi-level tables (e.g., groups within groups, like Region >
City > Sales).
● Stack: Makes it taller/skinner – moves inner columns to rows.
 ● Unstack: Opposite – spreads inner rows to columns.
● When to use: Complex data with hierarchies (less common for beginners,
but powerful for reports).

Example: Multi-Level Data

# Create hierarchical index: Region > Fruit

data_hier = {

'Sales': [100, 150, 80, 120], # Values

'East': [100, 150], # Fake, but imagine regions

'West': [80, 120]

# Better: Use MultiIndex

index = [Link].from_tuples([('East', 'Apple'), ('East', 'Banana'), ('West',

'Apple'), ('West', 'Banana')],

names=['Region', 'Fruit'])

df_hier = [Link]({'Sales': [100, 150, 80, 120]}, index=index)

print("Hierarchical Data:\n", df_hier)

Output:

Sales
Region Fruit
East Apple 100
Banana 150
West Apple 80
Banana 120

● Columns: Just 'Sales', but rows have levels (Region > Fruit).

Stack (Taller – but here it's already stacked-ish; imagine more columns):
# If we had more columns, stack would move them down

stacked = df_hier.stack() # Stacks the columns level

print("Stacked:\n", stacked)

Output (Turns to Series if single column):

Region Fruit
East Apple 100
Banana 150
West Apple 80

Banana 120
dtype: int64

● Now even longer.

Unstack (Wider – spread the inner level):

unstacked = df_hier.unstack(level=0) # level=0 is outermost (Region to columns)

print("Unstacked:\n", unstacked)

Output:

Fruit Apple Banana

Region
East 100 150
West 80 120

● Fruits as rows, regions as columns – quick regional comparison!

 File Handling & Functions

File Handling in Python (File Management)

File handling means working with files on your computer using Python—like
reading data from a text file (e.g., a CSV with sales data), writing results to a
new file, or appending updates. In data analytics, this is key for
importing/exporting data (e.g., loading a dataset from a file and saving
cleaned data).

Python treats files like objects. You "open" a file, do stuff (read/write), and
"close" it to avoid errors.

Why Use It?

● Load real-world data (e.g., logs, reports).

● Save analysis results (e.g., charts data to CSV).
● Automate tasks without manual copy-paste.

Basic Steps for File Handling

1. Open the file: Use open() function.

2. Do operations: Read, write, etc.
3. Close the file: Use close()—always do this to free up memory and avoid
errors.

Pro Tip: Closing manually is clear for learning, but Python has a shortcut
(explained later) to auto-close.

Opening a File: Modes (Like Permissions)

When you open a file, specify a mode (what you want to do):

● 'r' (read): Default. Read existing file. Error if file doesn't exist.
● 'w' (write): Create new file or overwrite existing. Careful—erases old
content!
● 'a' (append): Add to end of existing file. Creates if missing.
● 'r+' (read+write): Both, but file must exist.
● 'x' (exclusive create): Create new file only; error if exists.
Example syntax:

file_object = open('fi[Link]', 'r') # 'r' for read

For binary files (e.g., images, executables), add 'b' to any mode. These treat
data as raw bytes (no text decoding). Exam tip: Binary modes are common in
questions about media handling.

Mode Meaning

'rb' Read binary: File must exist.

'wb' Write binary: Creates or overwrites.

'ab' Append binary: Adds to end (creates if missing).

'rb+' Read + write binary: File must exist.

'wb+' Write + read binary: Overwrites, then allows read.

'ab+' Append + read binary: Adds to end, allows read.

Reading from Files

Once opened, use methods to grab data. Remember to close after!

1. read(): Read entire file as one big string.

● Good for small files.

● Example:

file = open('[Link]', 'r') # '[Link]' has lines like

"Apple\nBanana\nCherry"

content = fi[Link]()

print(content) # Output: Apple\nBanana\nCherry

fi[Link]() # Always close!

2. readline(): Read one line at a time (includes \n newline).

● Loops well for line-by-line processing.

● Example:

file = open('[Link]', 'r')

line1 = fi[Link]() # "Apple\n"
print([Link]()) # strip() removes \n: "Apple"
fi[Link]()

3. readlines(): Read all lines into a list (each line a string with \n).

● Perfect for data lists in analytics.

● Example:

file = open('[Link]', 'r')

lines = fi[Link]()
print(lines) # ['Apple\n', 'Banana\n', 'Cherry\n']
for line in lines:
print([Link]()) # Clean output: Apple, Banana, Cherry
fi[Link]()

Analytics Tip: Combine with split(',') for CSV files. E.g., read sales data and split
into columns.

Handling Large Files

Don't use read() on huge files—it loads everything into memory. Use a loop:

file = open('bigfi[Link]', 'r')

for line in file: # Reads line-by-line efficiently

print([Link]())
fi[Link]()

Writing to Files

Use 'w' or 'a' mode.

1. write(): Write a string to file.

● Example (creates/overwrites '[Link]'):

file = open('[Link]', 'w')
fi[Link]("Hello, Data World!\n") # \n for new line
fi[Link]("Sales: 1000") # Appends on same line? No, add \n
fi[Link]()

2. writelines(): Write a list of strings.

● Example:

data = ['Apple\n', 'Banana\n', 'Cherry\n']

file = open('[Link]', 'w')
fi[Link](data)
fi[Link]()

Analytics Tip: After analyzing data (e.g., with pandas), export to CSV:
df.to_csv('[Link]')—but basics use write().

Flush, Close, and Detach Methods

These control how/when data hits the disk.

● flush(): Forces buffered data to write immediately (without closing). Useful

for real-time logs.

file = open('live_log.txt', 'w', buffering=1)

fi[Link]("Update: Processing...\n")

file.flush() # Writes now—don't wait for close

fi[Link]()

● close(): Saves any remaining buffer and releases the file. Always call it!
Calling twice is safe (ignores second).
● detach(): Advanced—for binary files, separates the buffer from the file
object (returns raw buffer). Rarely used, but exams might ask for buffer
management.

file = open('[Link]', 'wb')

buffer = fi[Link]() # Now file is detached; use buffer directly

# [Link](b'bytes')
Appending Data

Use 'a' to add without overwriting.

file = open('[Link]', 'a')

fi[Link]("New entry: 2025-11-27\n")
fi[Link]()

Checking File Position & Seeking

● tell(): Current position (byte number).

● seek(offset): Jump to position (e.g., seek(0) to start). Example:

file = open('[Link]', 'r')

print(fi[Link]()) # 0 (start)
fi[Link](5) # Jump 5 bytes
print(fi[Link]()) # Reads from there
fi[Link]()

Error Handling

Exception Meaning When It Happens

FileNotFoundError File/path doesn't exist. Trying to read missing

file.
PermissionError No access rights. Read/write protected file.

IOError General I/O failure (e.g., disk Hardware/network issues.

full).
EOFError End of file reached unexpectedly. input() on empty file.

IsADirectoryError Path is a folder, not file. Open folder as file.

NotADirectoryError Path should be folder but isn't. Rare, e.g., mkdir on file.

Example

try:

file = open('[Link]', 'r')

print(fi[Link]())

fi[Link]()

except FileNotFoundError:

print("File not found! Check the name.")

except PermissionError:

print("No permission—check access rights.")

except (IOError, EOFError):

print("I/O or end-of-file error.")

except (IsADirectoryError, NotADirectoryError):

print("Path issue—file vs. folder mismatch.")

Working with Different File Types

● Text (.txt): As above.

● CSV: Use built-in csv module for structured data.

import csv
file = open('[Link]', 'r')
reader = [Link](file)
for row in reader:
print(row) # List like ['Date', 'Amount']
fi[Link]()

● JSON: For dictionaries/sets

import json
data = {'name': 'SEBI', 'score': 100}
file = open('[Link]', 'w')
[Link](data, file) # Write dict to JSON
fi[Link]()

File Paths: Relative vs. Absolute

 ● Relative: From current folder (e.g., '[Link]'—looks in same directory as
your script).
● Absolute: Full path (e.g., '/Users/You/Documents/[Link]' on Mac, or
'C:\\Users\\You\\[Link]' on Windows—use double backslashes).
● Use os module for cross-platform:

import os
path = [Link]('folder', '[Link]') # Handles slashes automatically
file = open(path, 'r')
# ... read ...
fi[Link]()

Encoding: Handling Special Characters

Text files might have accents/emojis (e.g., ₹ symbol in Indian data). Default is
UTF-8, but specify if needed:

file = open('[Link]', 'r', encoding='utf-8') # Safe for global data

content = fi[Link]()
fi[Link]()

● Without it? Errors like UnicodeDecodeError. Always add for analytics with
diverse data.

Binary Files: For Images or Non-Text

Use 'rb' (read binary) or 'wb' (write binary)—treats file as bytes, not strings.

Example (copy image):

input_file = open('[Link]', 'rb')

output_file = open('[Link]', 'wb')
data = input_fi[Link]() # Bytes
output_fi[Link](data)
input_fi[Link]()
output_fi[Link]()

Analytics Tip: Load images for ML data, but use libraries like Pillow for editing.

Deleting, Renaming, or Checking Files (Using os Module)

File management isn't just read/write—use os for ops.

import os
# Check if exists
if [Link]('[Link]'):
print("File is there!")
# Rename
[Link]('[Link]', '[Link]')
# Delete
[Link]('[Link]')
# List files in folder
print([Link]('.')) # '.' means current folder

A Note on the with Statement

Once you're comfortable with manual open() and close(), learn with—it's a
Python shortcut that auto-closes the file (even on errors), making code shorter
and safer. Syntax:

with open('[Link]', 'r') as file: # Opens

content = fi[Link]() # Work
# Auto-closes here—no manual close!

Use it in real projects to prevent "file leak" bugs, especially with big data.

Functions in Python
What are Functions?

A function is like a mini-program inside your code—a reusable block that does a
specific job. You define it once, call it many times. In data analytics, functions
clean data, calculate stats, or plot graphs without repeating code.

Think of it as a recipe: Ingredients (inputs) → Steps (code) → Dish (output).

Why Use Functions?

● Avoid repetition (DRY: Don't Repeat Yourself).

● Easier to debug/fix.
 ● Modular: Break big code into small pieces.
● Ties to classes (next in syllabus)—functions are building blocks.

Defining a Function
Use def keyword.

Basic syntax:

def function_name(parameters): # Parameters: inputs (optional)

1. Simple Function (No Inputs/Outputs)

def greet():
print("Hello, SEBI Officer!")
# Call it:
greet() # Output: Hello, SEBI Officer!

2. Function with Parameters (Inputs)

● Positional: Order matters.

def add_numbers(a, b): # a and b are parameters

result = a + b
return result # Sends back the answer
sum_value = add_numbers(5, 3) # 5 and 3 are arguments
print(sum_value) # 8

● Keyword Arguments: Name them (order doesn't matter).

print(add_numbers(b=3, a=5)) # Still 8

3. Default Parameters (Optional Inputs)

Set defaults if no value given.

def greet_user(name="World"): # Default: "World"

print(f"Hello, {name}!")
greet_user() # Hello, World!
greet_user("Alice") # Hello, Alice!
4. Variable Arguments (*args & **kwargs)

For unknown number of inputs.

● *args: Extra positional as tuple.

● **kwargs: Extra keyword as dict.

def sum_all(*args): # args is tuple

return sum(args)

print(sum_all(1, 2, 3, 4)) # 10
def display_info(**kwargs): # kwargs is dict
for key, value in [Link]():
print(f"{key}: {value}")
display_info(name="Bob", age=30, role="Analyst") # name: Bob, etc.

Analytics Tip: Use *args for flexible data lists, like summing sales figures.

Returning Values

● return sends output. Function ends there.

● Multiple returns? Use if-else.

def check_even(num):
if num % 2 == 0:
return "Even"
return "Odd" # Only one runs
print(check_even(4)) # Even

● Return multiple: As tuple.

def stats(numbers):
return sum(numbers), len(numbers), sum(numbers)/len(numbers)
total, count, avg = stats([10, 20, 30]) # Unpack: 60, 3, 20.0

Scope: Where Variables Live

● Local: Inside function—gone after call.

 ● Global: Outside—accessible everywhere, but don't change globals inside
functions.

x = 10 # Global
def modify():
x = 5 # Local x, doesn't change global
print(x) # 5
modify()
print(x) # Still 10

● To change global: Use global x inside function .

Lambda Functions (Anonymous/One-Liners)

Short functions for quick tasks (e.g., sorting data).

Syntax: lambda parameters: expression

square = lambda num: num ** 2

print(square(5)) # 25
# With map() for lists (analytics fave):
numbers = [1, 2, 3]
squares = list(map(lambda n: n**2, numbers))
print(squares) # [1, 4, 9]

Recursion: Functions Calling Themselves

For repetitive tasks (e.g., factorial).

def factorial(n):
if n == 1: # Base case—stop!
return 1
return n * factorial(n-1) # Calls itself
print(factorial(5)) # 120 (5*4*3*2*1)

Note: Can crash with deep recursion—use loops for big data.

Passing Functions as Arguments (Higher-Order Functions)

Functions can be inputs to other functions—powerful for analytics (e.g., sorting
with custom rules).

def apply_operation(func, x, y): # func is a function!

return func(x, y)
def multiply(a, b):
return a * b
result = apply_operation(multiply, 4, 5) # 20

print(result)

Handling Errors in Functions

Wrap risky code in try-except inside functions.

def safe_divide(a, b):

try:
return a / b

except ZeroDivisionError:
return "Can't divide by zero!"
except TypeError:
return "Inputs must be numbers."
print(safe_divide(10, 0)) # Can't divide by zero!

Functions in Data Analytics Context

● Data Cleaning: Function to remove duplicates from lists.

def clean_data(data_list):

return list(set(data_list)) # Ties to "Sets" in syllabus

print(clean_data([1, 2, 2, 3])) # [1, 2, 3]

● Export Helper: Wrap file write.

def save_report(data, filename):

file = open(filename, 'w')

fi[Link](str(data))
 fi[Link]()

● For charts/graphs: Functions to generate plots (with matplotlib—import

it).

Pure vs. Impure Functions (Theoretical Concept)

● Pure: Same input → same output; no side effects (e.g., no prints, globals,
I/O). Predictable—great for testing/math.

def pure_add(a, b): # Pure: Just math

return a + b

● Impure: Depends on external state (e.g., time, files, random). Output

varies.

import random

def impure_rand(a): # Impure: Random each time

return a + [Link](1, 10)

 Data Import & Export
Data Import and Export is the process of reading data from external files (like
CSV, Excel, JSON) into a DataFrame for analysis, and then saving the
processed results back to a file. This relies heavily on the pandas library.

Why is Import/Export Critical?

● It's the bridge between raw, real-world data (e.g., SEBI compliance logs )
and the Python analysis environment.
● It allows for permanent storage of cleaned data and results.
● It supports various file formats required by different stakeholders and
systems.

1. Importing Data (Loading into a DataFrame)

The primary functions used for importing data begin with pd.read_. These
functions take the file path and various parameters to correctly parse the file's
content into a DataFrame.

2. Importing Data (Bringing It In)

Pandas has pd.read_*() functions. They turn files into DataFrames

automatically.

2.1 Importing CSV Files (Comma-Separated Values – Easiest)

● CSV: Like a table saved as text (e.g., Name,Age\nAlice,20).

● Syntax: pd.read_csv('fi[Link]')
● Options: Skip header rows, handle missing data.

Example: Imagine a '[Link]' file:

Product,Month,Sales
Apple,Jan,100
Banana,Feb,120

Code:
import pandas as pd
# Import the CSV
df = pd.read_csv('[Link]') # Assumes file in same folder
# Peek at it
print([Link]())
print([Link]) # (2 rows, 3 columns)

Output:

Product Month Sales

0 Apple Jan 100
1 Banana Feb 120

● If file not found? Error: Use full path like

pd.read_csv('/Users/you/Downloads/[Link]').
● Fancy Options:
○ No header: pd.read_csv('fi[Link]', header=None) → Columns
become 0,1,2.
○ Skip rows: pd.read_csv('fi[Link]', skiprows=1) (ignores first line).
○ Delimiter not comma? sep=';' for semicolons.

2.2 Importing Excel Files (.xlsx or .xls)

● Excel: Multi-sheets, formulas – but we import as data only.

● Syntax: pd.read_excel('fi[Link]')
● Needs: pip install openpyxl (if not in Colab).

Example: '[Link]' with sheet 'Sales':

Python
df = pd.read_excel('[Link]', sheet_name='Sales') # Or index=0 for first
sheet
print(df)

● Output: Same as CSV table.

● Options: sheet_name='Sheet2' or usecols=['A','B'] (only certain columns).

Analogy: CSV is a printed list; Excel is a binder with tabs (sheets) – pick the tab
you want.
2.3 Importing JSON Files (For Web/API Data)

● JSON: Like a dictionary (e.g., {"name": "Alice", "age": 20}) – nested,

flexible.
● Syntax: pd.json_normalize() for flat tables, or pd.read_json() for simple.

Example: '[Link]':

JSON
[
{"Product": "Apple", "Month": "Jan", "Sales": 100},
{"Product": "Banana", "Month": "Feb", "Sales": 120}
]

Code:

df = pd.read_json('[Link]')
print(df)

● Output: DataFrame like before.

● For nested: pd.json_normalize(data) where data is a dict.

2.4 Importing Text Files (TXT – Raw Data)

● TXT: Unstructured, but if comma-separated, treat as CSV.

● Syntax: pd.read_csv('fi[Link]', sep='\t') (for tabs) or pd.read_table().

Quick Example (Tab-separated):

Python
df = pd.read_csv('[Link]', sep='\t')
print(df)

2.5 Other Imports (Quick Mentions)

● From URL: pd.read_csv('[Link] – great for online

data.
● From Clipboard: pd.read_clipboard() – copy table from browser.
● SQL Database: pd.read_sql('SELECT * FROM table', connection) (needs
SQL setup).
After Import – Always Do This:

● [Link](): Check types, missing values.

● [Link](): Stats.
● Clean: [Link]() for missing rows.

3. Exporting Data (Saving It Out)

Pandas has df.to_*() methods. Super simple – your DataFrame becomes a file.

3.1 Exporting to CSV

● Syntax: df.to_csv('[Link]', index=False) # index=False skips row

numbers

Example (From our imported df):

Python
df.to_csv('new_sales.csv', index=False)
print("Saved to new_sales.csv!")

● Creates 'new_sales.csv' in your folder. Open in Excel to check.

3.2 Exporting to Excel

● Syntax: df.to_excel('[Link]', sheet_name='Data', index=False)

Example:

df.to_excel('[Link]', sheet_name='Sales Summary')

● Multiple sheets? Use ExcelWriter:

with [Link]('[Link]') as writer:

df.to_excel(writer, sheet_name='Sales', index=False)

[Link]().to_excel(writer, sheet_name='Stats', index=False)

3.3 Exporting to JSON

 ● Syntax: df.to_json('[Link]', orient='records') # 'records' for
list-of-dicts

Example:

df.to_json('[Link]', orient='records')

● Perfect for web apps.

3.4 Exporting to Text (TXT)

● Syntax: df.to_csv('[Link]', sep='\t', index=False) # Tab-separated

Quick Example:

df.to_csv('[Link]', sep='\t')

3.5 Other Exports

● To HTML: df.to_html('[Link]') – for web pages.

● To Clipboard: df.to_clipboard() – paste into Excel.
● Pickle (Binary, Fast): df.to_pickle('[Link]') – load with pd.read_pickle().

Analogy: Export is like printing your edited photo – choose format (CSV=plain
paper, Excel=fancy album).

Decision Table: Quick Format Guide

Format When to Code Snippet When to Pro Con

Import Export
CSV Simple pd.read_csv('fi[Link]') Sharing basic Fast, No
tables data universal formulas
Excel Multi-sheets pd.read_excel('fi[Link] Reports with Sheets, Bigger
x') formatting styles files
JSON Nested/web pd.read_json('fi[Link]' APIs/apps Flexible Harder to
data ) read
TXT Logs/raw pd.read_csv('fi[Link]', Quick dumps Plain Needs
sep='\t') delimiter
Data Mining and Charts & Graphs in Python for
Data Analytics
These notes provide a structured, professional overview of Data Mining and
Charts & Graphs, key components of data analytics as outlined in the syllabus.
The content is designed for beginners with no prior programming or statistical
knowledge, using clear, step-by-step explanations and Python code examples.
We assume familiarity with foundational concepts such as DataFrames (from
pandas) and basic data operations.

All examples use Python 3.x with libraries: pandas for data manipulation,
scikit-learn for data mining (install via pip install scikit-learn), and matplotlib
with seaborn for visualization (install via pip install matplotlib seaborn). Code is
executable in environments like Jupyter Notebook or Google Colab.

Import statements for all sections:

import pandas as pd
import [Link] as plt
import seaborn as sns
from [Link] import KMeans
from [Link] import StandardScaler
from mlxtend.frequent_patterns import apriori, association_rules
from [Link] import TransactionEncoder
Data Mining
Data Mining is the process of discovering patterns, insights, and hidden
knowledge from large datasets. It uses methods from machine learning,
statistics, and database systems to turn raw data into actionable information.

Data Mining is the key step where you move from basic descriptive statistics
(like mean, median) to predictive and prescriptive analytics.

Phases of Data Mining

Data mining generally follows a cyclical process (often referred to as CRISP-DM,
or Cross-Industry Standard Process for Data Mining):
 1. Business Understanding: Define the objective (e.g., predict stock
volatility, identify fraudulent transactions).
2. Data Understanding: Initial data collection, exploration, and verification.
3. Data Preparation (Ties to Syllabus): Cleaning, transformation, feature
selection, and reshaping. This is often the most time-consuming step.
4. Modeling (The Core Mining): Applying algorithms to the prepared data.
5. Evaluation: Assessing the model's performance and accuracy.
6. Deployment: Using the model to generate insights or predictions in a live
system.

Key Data Mining Techniques and Algorithms

The modeling phase involves several core tasks:

Technique Goal Python Syllabus Tie-In

Libraries

Classification Predicts a category Scikit-learn Classes and

or class (e.g., whether (SKlearn): Functions.
a loan applicant will Decision Tree,
default (Yes/No)). Logistic
Regression.

Regression Predicts a continuous SKlearn: Linear Dataframes,

value (e.g., predicting Regression, Functions.
the price of a stock Ridge
next month). Regression.

Clustering Groups similar data SKlearn: Sets, Data

points together K-Means, reshaping.
without prior training DBSCAN.
labels (unsupervised
learning) (e.g.,
 segmenting
customers).

Association Finds relationships MLxtend: Apriori Dictionaries

between items (e.g., algorithm. (key-value
"If a customer buys relationships).
product A, they often
buy product B").

Outlier Detection Identifies rare events SKlearn: Slicing (to

or anomalies (e.g., Isolation Forest. isolate
flagging suspicious anomalies).
trades in compliance
data).

Data Mining in Python

Python is the standard for data mining, primarily utilizing these powerful
libraries:

● Pandas: Used for Data Preparation and cleaning

● NumPy: Provides fast numerical computation for handling arrays and
mathematical operations.
● Scikit-learn (SKlearn): The comprehensive library for implementing
nearly all machine learning and data mining algorithms (Classification,
Regression, Clustering).

Charts and Graphs in Python

Charts and Graphs (Data Visualization) are essential for communicating
patterns, trends, and outliers discovered during Data Mining. Visualization
turns abstract data into a concise, easily understandable visual format.

Core Visualization Libraries

Library Purpose & Focus Analytics Use

Matplotlib The foundational library. Powerful for Used for initial data
creating basic plots and highly exploration and complex
customized static figures. scientific plots.

Seaborn Built on Matplotlib. Provides a Excellent for statistical

high-level interface for drawing summaries and
attractive and informative correlation analysis.
statistical graphics (e.g., heatmaps,
pair plots).

Plotly / Interactive visualization libraries. Used for dashboarding

Bokeh Create web-based, zoomable, and and sharing dynamic
hoverable charts. results with non-technical
teams.

Essential Chart Types for Data Analytics

Choosing the right chart depends on the relationship you want to display:

Chart Type Purpose Example Use Case (SEBI

Context)

Line Chart Shows trends over a Tracking the Nifty 50 index price
continuous period movement over the last year.
(time-series data).

Bar Chart Compares discrete Comparing the number of

categories or counts. compliance violations across
 different regulatory zones
(Categories).

Scatter Plot Shows the relationship Plotting Volume vs. Price to see
(correlation) between two if high volume correlates with
continuous variables. price changes.

Histogram Shows the distribution of Displaying the frequency

a single numerical distribution of daily stock
variable. returns to assess risk.

Box Plot Displays the five-number Comparing the volatility

summary (min, max, (spread) of different stocks and
median, quartiles) and identifying unusual trading
highlights outliers. days.

Heatmap Shows magnitude using Visualizing the correlation

color in a 2D matrix. matrix between 20 different
stock prices.

Creating a Simple Chart in Python

Creating a graph typically involves: (1) importing the data (often a DataFrame),
(2) selecting the relevant columns (Slicing), and (3) calling the plot function.

Example using Matplotlib and Pandas:

import [Link] as plt
import pandas as pd

# Assume df is a DataFrame with 'Date' and 'Stock_Price' columns

# df = pd.read_csv('historical_data.csv')
# 1. Prepare data (Slicing)
x_data = df['Date']
y_data = df['Stock_Price']

# 2. Plot the data

plt.figure(figsize=(10, 6)) # Define size of the figure
[Link](x_data, y_data, label='Stock Price')
[Link]('Stock Price Trend Over Time')
[Link]('Date')
[Link]('Price (INR)')
[Link]()
[Link](True)
[Link]() # Display the chart