Chapter-1

Advanced Python Concepts and Best

Practices

Review of Python essentials

Data Types
Data types define the type of data a variable can hold. Common Python data types include:

• Integer (int) – Whole numbers, e.g., 5, -10

• Float (float) – Decimal numbers, e.g., 3.14, -0.5

• String (str) – Sequence of characters, e.g., ”Hello”

• Boolean (bool) – Logical values, True or False

Data Structures
Data structures are ways to organize and store multiple values efficiently. Common Python
structures include:

• List (list) – Ordered, mutable collection, e.g., [1, 2, 3]

• Tuple (tuple) – Ordered, immutable collection, e.g., (1, 2, 3)

• Set (set) – Unordered, unique elements, e.g., {1, 2, 3}

• Dictionary (dict) – Key-value pairs, e.g., {”name”: ”Alice”, ”age”: 25}

Mutable Data Types

Mutable data types are those whose values can be changed after they are created. Examples:

• List (list) – e.g., mylist = [1, 2, 3] → mylist[0] = 10

• Dictionary (dict) – e.g., mydict = {"name": "Alice"} → mydict["name"] = "Bob"

• Set (set) – e.g., myset = {1, 2, 3} → [Link](4)

1
 2

Immutable Data Types

Immutable data types are those whose values cannot be changed once they are created. Any
modification creates a new object. Examples:

• Integer (int) – e.g., x = 5 → x + 1 creates a new integer object

• Float (float) – e.g., y = 3.14

• String (str) – e.g., s = "Hello" → [Link]() creates "HELLO"

• Tuple (tuple) – e.g., t = (1, 2, 3)

Sequence Types
Lists []
Ordered, mutable, heterogeneous collections.
1 fruits = [ ’ apple ’ , ’ banana ’ , ’ orange ’]
2 fruits . append ( ’ grape ’) # Add to end
3 fruits . insert (1 , ’ kiwi ’) # Insert at index
4 fruits . remove ( ’ banana ’) # Remove by value
5 popped = fruits . pop () # Remove and return last
6 sliced = fruits [1:3] # Slicing [ start : end : step ]

Tuples ()
Ordered, immutable, heterogeneous collections.
1 coordinates = (10 , 20)
2 x , y = coordinates # Tuple unpacking
3 single_item = (1 ,) # Single item needs trailing comma

Strings ’’ or ""
Ordered, immutable sequences of characters.
1 text = " Python Programming "
2 text . upper () # Returns new string
3 text . split () # Split into list
4 " " . join ([ ’ Python ’ , ’ Data ’ ]) # Join list into string

Ranges
Immutable sequences of numbers.
1 range (5) # 0, 1, 2, 3, 4
2 range (2 , 10 , 2) # 2, 4, 6, 8

Mapping Types
Dictionaries {}
Key-value pairs (Python 3.7+ preserves insertion order).

2
 3

1 person = { ’ name ’: ’ Alice ’ , ’ age ’: 30}

2 person . get ( ’ city ’ , ’ Unknown ’) # Safe access with default
3 person . keys () # View of keys
4 person . values () # View of values
5 person . items () # View of key - value pairs
6 person . setdefault ( ’ country ’ , ’ USA ’) # Set if key doesn ’t exist

defaultdict (from collections)

Provides default values for missing keys.
1 from collections import defaultdict
2 word_count = defaultdict ( int ) # Default 0 for missing keys
3 word_count [ ’ apple ’] += 1 # No KeyError
4
5 group_by_first = defaultdict ( list ) # Default empty list
6 group_by_first [ ’a ’ ]. append ( ’ apple ’) # Auto - creates list

OrderedDict (from collections)

Remembers insertion order (less needed in Python 3.7+).
1 from collections import OrderedDict
2 od = OrderedDict ()
3 od . move_to_end ( ’ key ’) # Move to beginning / end

Set Types
set
Unordered collection of unique, hashable objects.
1 set1 = {1 , 2 , 3 , 4}
2 set2 = {3 , 4 , 5 , 6}
3 set1 . add (5) # Add element
4 set1 . remove (3) # Remove ( raises KeyError if missing )
5 set1 . discard (10) # Remove ( no error if missing )
6
7 # Set operations
8 set1 . union ( set2 ) # {1 , 2 , 3 , 4 , 5 , 6}
9 set1 . intersection ( set2 ) # {3 , 4}
10 set1 . difference ( set2 ) # {1 , 2}
11 set1 . s y m m etric_difference ( set2 ) # {1 , 2 , 5 , 6}
12 set1 . issubset ( set2 ) # Check if subset

frozenset
Immutable set (hashable, can be dict key).
1 immutable_set = frozenset ([1 , 2 , 3])

NoneType
Represents absence of value.

• Single value None.

• Falsy in boolean contexts.

3
 4

• Default return of functions without a return statement.

• Use is None for comparison (not ==).

Control Flow
Conditional Statements
Conditional statements allow decision-making in code based on logical conditions.
1 # Basic if - elif - else
2 if condition1 :
3 # code block
4 elif condition2 :
5 # code block
6 else :
7 # code block
8
9 # Ternary expression
10 value = x if condition else y
11
12 # Pattern matching ( Python 3.10+)
13 match point :
14 case (0 , 0) :
15 print ( " Origin " )
16 case (x , 0) :
17 print ( f "X - axis at { x } " )
18 case (0 , y ) :
19 print ( f "Y - axis at { y } " )
20 case (x , y ) :
21 print ( f " Point at { x } , { y } " )

Loops
Loops allow repeated execution of code blocks.
For loops: Iterate over sequences.
1 for item in iterable :
2 # code block
3
4 # Enumerate for index and value
5 for i , value in enumerate ( iterable ) :
6 print ( f " Index { i }: { value } " )
7
8 # Zip for parallel iteration
9 for a , b in zip ( list1 , list2 ) :
10 print (a , b )
11
12 # Iterate over dictionary
13 for key , value in dictionary . items () :
14 print ( key , value )

While loops: Repeat while a condition is true.

1 while condition :
2 # code block
3 if exit_condition :
4 break
5 if skip_condition :
6 continue

Loop else clause: Executes only if loop completes without break.

4
 5

1 for item in items :

2 if condition ( item ) :
3 break
4 else :
5 print ( " No item matched condition " )

Comprehensions
Concise way to create collections from iterables.
1 # List comprehension
2 squares = [ x **2 for x in range (10) ]
3 evens = [ x for x in range (20) if x % 2 == 0]
4 matrix = [[ i * j for j in range (3) ] for i in range (3) ]
5
6 # Dictionary comprehension
7 square_dict = { x : x **2 for x in range (5) }
8 filtered_dict = { k : v for k , v in original . items () if v > 10}
9
10 # Set comprehension
11 unique_lengths = { len ( word ) for word in words }
12
13 # Generator expressions
14 sum_of_squares = sum ( x **2 for x in range (1000000) )
15 gen = ( x **2 for x in range (5) )
16 print ( list ( gen ) ) # [0 , 1 , 4 , 9 , 16]
17 print ( list ( gen ) ) # [] ( exhausted )

Functions
First-Class Functions
Functions can be assigned, passed, and returned.
1 def greet ( name ) :
2 return f " Hello , { name } "
3
4 say_hello = greet
5 print ( say_hello ( " Alice " ) )
6
7 def apply_twice ( func , arg ) :
8 return func ( func ( arg ) )
9
10 def add_one ( x ) :
11 return x + 1
12
13 result = apply_twice ( add_one , 5) # 7

Lambda Functions
Anonymous, single-expression functions.
1 square = lambda x : x **2
2 add = lambda x , y : x + y
3
4 pairs = [(1 , ’ one ’) , (3 , ’ three ’) , (2 , ’ two ’) ]
5 pairs . sort ( key = lambda pair : pair [0])
6 evens = list ( filter ( lambda x : x % 2 == 0 , numbers ) )
7 squared = list ( map ( lambda x : x **2 , numbers ) )

5
 6

*args and **kwargs

Handle variable positional and keyword arguments.
1 def sum_all (* args ) :
2 return sum ( args )
3
4 def print_info (** kwargs ) :
5 for key , value in kwargs . items () :
6 print ( f " { key }: { value } " )
7
8 def complex_function (a , b , * args , ** kwargs ) :
9 print ( f " a : { a } , b : { b } " )
10 print ( f " args : { args } " )
11 print ( f " kwargs : { kwargs } " )
12
13 # Unpacking arguments
14 def add (a , b , c ) :
15 return a + b + c
16
17 numbers = [1 , 2 , 3]
18 print ( add (* numbers ) ) # Unpack list
19
20 kwargs = { ’a ’: 1 , ’b ’: 2 , ’c ’: 3}
21 print ( add (** kwargs ) ) # Unpack dict

Decorators
Functions that modify behavior of other functions.
1 def timer ( func ) :
2 import time
3 def wrapper (* args , ** kwargs ) :
4 start = time . time ()
5 result = func (* args , ** kwargs )
6 end = time . time ()
7 print ( f " { func . __name__ } took { end - start :.4 f } seconds " )
8 return result
9 return wrapper
10
11 @timer
12 def slow_function () :
13 import time
14 time . sleep (1)
15 return " Done "
16
17 # Decorators with arguments
18 def repeat ( n_times ) :
19 def decorator ( func ) :
20 def wrapper (* args , ** kwargs ) :
21 for _ in range ( n_times ) :
22 result = func (* args , ** kwargs )
23 return result
24 return wrapper
25 return decorator
26
27 @repeat (3)
28 def greet ( name ) :
29 print ( f " Hello { name } " )

Closures
Functions capturing variables from outer scope.

6
 7

1 def outer_function ( message ) :

2 def inner_function () :
3 print ( message )
4 return inner_function
5
6 closure = outer_function ( " Hello World " )
7 closure () # Still remembers " Hello World "
8
9 def counter ( start =0) :
10 count = start
11 def increment () :
12 nonlocal count
13 count += 1
14 return count
15 return increment
16
17 c1 = counter (10)
18 print ( c1 () ) # 11
19 print ( c1 () ) # 12

Error Handling
try, except, else, finally
Handle exceptions gracefully.
1 try :
2 risky_operation ()
3 except SomeSpecificError as e :
4 print ( f " Error occurred : { e } " )
5 except Exception as e :
6 print ( f " Unexpected error : { e } " )
7 else :
8 print ( " Operation successful ! " )
9 finally :
10 clean up_resources ()

Custom Exceptions
Define specific errors for clarity.
1 class DataScienceError ( Exception ) : pass
2 class D a t aValidationError ( DataScienceError ) : pass
3
4 class Mo delTrainingError ( DataScienceError ) :
5 def __init__ ( self , message , model_name = None ) :
6 self . model_name = model_name
7 super () . __init__ ( f " { message } ( Model : { model_name }) " )
8
9 # Usage
10 def train_model ( data , model_name ) :
11 if not data :
12 raise DataValidationError ( " Empty dataset provided " )
13 try :
14 # training code
15 pass
16 except Exception as e :
17 raise ModelTrainingError ( f " Training failed : { e } " , model_name )
18
19 # Handling custom exceptions
20 try :

7
 8

21 train_model ([] , " RandomForest " )

22 except D a taValidationError as e :
23 print ( f " Validation issue : { e } " )
24 except Mo delTrainingError as e :
25 print ( f " Training issue with { e . model_name }: { e } " )
26 except DataScienceError as e :
27 print ( f " General data science error : { e } " )

Context Managers
Ensure resource management using with.
1 with open ( ’ file . txt ’ , ’r ’) as file :
2 content = file . read ()
3
4 # Custom context manager using contextlib
5 from contextlib import contextmanager
6
7 @contextmanager
8 def managed_file ( filename , mode = ’r ’) :
9 file = open ( filename , mode )
10 try :
11 yield file
12 finally :
13 file . close ()
14
15 # Multiple context managers
16 with open ( ’ input . txt ’) as infile , open ( ’ output . txt ’ , ’w ’) as outfile :
17 outfile . write ( infile . read () )
18
19 # Timing context manager
20 import time
21
22 @contextmanager
23 def timer () :
24 start = time . time ()
25 try :
26 yield
27 finally :
28 end = time . time ()
29 print ( f " Operation took { end - start :.4 f } seconds " )

Naming conventions
Naming conventions are a set of standardized rules and guidelines for choosing names for program
identifiers, such as variables, functions, classes, and constants. They ensure:

• Consistency – Uniform style across the codebase

• Readability – Code is easier to understand at a glance

• Maintainability – Easier to update and debug

By following naming conventions, identifiers convey meaning about their purpose and scope.

Common Python Conventions / PEP Guidelines

• PEP 8 – Style Guide for Python Code

8
 9

• PEP 257 – Docstring Conventions

• PEP 484 – Type Hints

• PEP 20 – The Zen of Python

PEP 8 Style Guide

Naming Conventions
• Variables and Functions: snake case

• Classes: CapWords / CamelCase

• Constants: UPPER CASE

• Internal use: single leading underscore

• Private attributes: double leading underscore

• Magic / special methods: dunder (e.g., init )

Code Layout
• 4 spaces per indentation level

• No tabs

• Maximum line length: 79 characters

• Maximum docstring length: 72 characters

• 2 blank lines between top-level functions/classes

• 1 blank line between class methods

• UTF-8 encoding for files

Imports
• One import per line

• Imports grouped in order: standard library → third-party → local

• Absolute imports preferred

• No wildcard imports (avoid from module import *)

Whitespace
• Space around binary operators: a + b, not a+b

• No spaces inside parentheses, brackets, or braces: func(a, b)

• Space after commas: [1, 2, 3]

• No space before commas

9
 10

• Space after colon in dictionaries: {’key’: value}

• No space before colon

Comments
• Inline comments start with #: # comment

• Block comments explain complex logic

• Comments should explain why, not what

String Quotes
• Consistency within a module

• Triple quotes for docstrings: """Docstring"""

Line Breaks
• Break before binary operators

• Align with opening delimiter or use hanging indent

PEP 8 Examples

Naming Conventions Programs

1 # Variables and Functions : snake_case

2 customer_name = " John Smith "
3 total_amount = 150.75
4 is_active_member = True
5

6 def c al cu la te _d is co un t ( price , discount_rate ) :

7 " " " Calculate discounted price . " " "
8 return price * (1 - discount_rate )
9
10 def fetch_user_data ( user_id ) :
11 " " " Retrieve user information from database . " " "
12 # Database query logic
13 return { " id " : user_id , " name " : " Alice " }
14
15 # Classes : CapWords / CamelCase
16 class CustomerAccount :
17 " " " Represents a customer account . " " "
18 def __init__ ( self , name , balance ) :
19 self . name = name
20 self . balance = balance
21
22 def deposit ( self , amount ) :
23 self . balance += amount
24
25 class DataProcessor :
26 " " " Process and transform data . " " "

10
 11

27 def process_csv ( self , filepath ) :

28 " " " Process CSV file . " " "
29 pass
30
31 # Constants : UPPER_CASE
32 M AX _L OG IN _A TT EM PT S = 5
33 D E F A U L T _ T I M E O U T _ S E C O N D S = 30
34 API_BASE_URL = " https :// api . example . com / v1 "
35 PI = 3.14159265359
36 INTEREST_RATE = 0.05

Code Layout Programs

1 # 4 spaces per indentation level

2 def process_orders ( orders ) :
3 " " " Process multiple orders . " " "
4 results = []
5 for order in orders :
6 customer_id = order . get ( ’ customer_id ’)
7 if customer_id :
8 items = order . get ( ’ items ’ , [])
9 for item in items :
10 price = item . get ( ’ price ’ , 0)
11 quantity = item . get ( ’ quantity ’ , 1)
12 total = price * quantity
13 results . append ( total )
14 return results
15
16 # No tabs - using spaces consistently
17 class OrderManager :
18 " " " Manage customer orders . " " "
19 def __init__ ( self , orders ) :
20 self . orders = orders
21 self . total = 0
22

23 def calculate_total ( self ) :

24 for order in self . orders :
25 for item in order [ ’ items ’ ]:
26 self . total += item [ ’ price ’] * item [ ’ quantity ’]
27 return self . total

Imports Programs

1 # Standard Library
2 import os
3 import sys
4 import json
5 from datetime import datetime
6 from collections import defaultdict , Counter
7
8 # Third - party
9 import numpy as np
10 import pandas as pd
11 import matplotlib . pyplot as plt

11
 12

12 from sklearn . model_selection import train_test_split

13

14 # Local imports
15 from myproject . config import settings
16 from myproject . database import get_connection
17 from myproject . models import Customer

Whitespace, Comments, Strings, Line Breaks

1 # Whitespace examples
2 x = 5 + 3
3 function_call ( arg1 , arg2 , arg3 )
4 user = { ’ name ’: ’ Alice ’ , ’ age ’: 30}
5
6 # Comments
7 def calculate_tax ( price , tax_rate ) :
8 " " " Calculate tax amount . " " "
9 # Adjust for tax - exempt items
10 if tax_rate < 0:
11 tax_rate = 0
12 return price * tax_rate
13
14 # Strings
15 name = " Alice "
16 message = f " { name } , welcome ! "
17 def multiply (a , b ) :
18 " " " Multiply two numbers . " " "
19 return a * b
20
21 # Line breaks
22 total = ( first_variable
23 + second_variable
24 - third_variable
25 * fourth_variable )

Complete Example: Data Processing Pipeline

1 """
2 data_pipeline . py
3 Complete data processing pipeline demonstrating PEP 8 conventions .
4 """
5 import csv
6 import json
7 import logging
8 from pathlib import Path
9 from typing import List
10 import numpy as np
11 import pandas as pd
12 from sklearn . preprocessing import StandardScaler
13 from . config import settings
14 from . validators import validate_data
15
16 MAX_FILE_SIZE_MB = 100
17 DEFAULT_ENCODING = ’utf -8 ’

12
 13

18
19 class DataPipeline :
20 def __init__ ( self , source_path : Path , destination_path : Path ) :
21 self . source_path = source_path
22 self . destination_path = destination_path
23 self . data = None
24 self . _processed = False
25

26 def load_data ( self ) -> pd . DataFrame :

27 if not self . source_path . exists () :
28 raise Fi leNotF oundEr ror ( f " { self . source_path } not found " )
29 self . data = pd . read_csv ( self . source_path , encoding =
DEFAULT_ENCODING )
30 return self . data
31
32 def process ( self , normalize_cols : List [ str ] = None ) -> pd . DataFrame :
33 self . _clean_data ()
34 if normalize_cols :
35 self . _ no r m al i z e_ f e at u r es ( normalize_cols )
36 self . _processed = True
37 return self . data
38
39 def _clean_data ( self ) :
40 self . data = self . data . drop_duplicates ()
41 for col in self . data . columns :
42 if self . data [ col ]. dtype in [ ’ int64 ’ , ’ float64 ’ ]:
43 self . data [ col ]. fillna ( self . data [ col ]. median () , inplace =
True )
44 else :
45 self . data [ col ]. fillna ( ’ Unknown ’ , inplace = True )
46
47 def _no r m al i z e_ f e at u r es ( self , columns : List [ str ]) :
48 scaler = StandardScaler ()
49 self . data [ columns ] = scaler . fit_transform ( self . data [ columns ])
50
51 def export_data ( self , format = ’ csv ’) :
52 if not self . _processed :
53 raise ValueError ( " No processed data to export " )
54 if format == ’ csv ’:
55 self . data . to_csv ( self . destination_path , encoding =
DEFAULT_ENCODING , index = False )

Types of Advanced Data Structures in Python

• Collections – Specialized container datatypes like namedtuple, defaultdict, Counter,
OrderedDict, deque, ChainMap.

• Iterators – Objects that implement the iterator protocol with iter () and next ()
methods.

• Generators – Functions using yield or generator expressions that produce values lazily.

• Decorators – Functions or classes that modify behavior of other functions or classes using
@decorator syntax.

13
 14

Collections Module

namedtuple
1 from collections import namedtuple
2
3 # Creating a namedtuple type
4 Point = namedtuple ( ’ Point ’ , [ ’x ’ , ’y ’ ])
5 p = Point (10 , 20)
6 print ( p .x , p . y ) # Access by name
7 print ( p [0] , p [1]) # Access by index also works

defaultdict
1 from collections import defaultdict
2
3 # int provides default value 0
4 word_counts = defaultdict ( int )
5 # list provides default empty list
6 grouped_data = defaultdict ( list )
7 # set provides default empty set
8 unique_sets = defaultdict ( set )

Counter
1 from collections import Counter
2
3 # Count elements in an iterable
4 freq = Counter ([ ’a ’ , ’b ’ , ’a ’ , ’c ’ , ’a ’ , ’b ’ ])
5 # Most common elements
6 top_two = freq . most_common (2)

OrderedDict
1 from collections import OrderedDict
2
3 od = OrderedDict ()
4 od [ ’ first ’] = 1
5 od [ ’ second ’] = 2
6 od . move_to_end ( ’ first ’) # Moves to end

deque
1 from collections import deque
2

3 dq = deque ( maxlen =3) # Fixed - size queue

4 dq . append (1) # Add to right
5 dq . appendleft (2) # Add to left
6 dq . pop () # Remove from right
7 dq . popleft () # Remove from left

14
 15

ChainMap

1 from collections import ChainMap

2
3 defaults = { ’ theme ’: ’ light ’ , ’ language ’: ’ en ’}
4 user_prefs = { ’ theme ’: ’ dark ’}
5 settings = ChainMap ( user_prefs , defaults )
6 print ( settings [ ’ theme ’ ]) # ’ dark ’
7 print ( settings [ ’ language ’ ]) # ’ en ’

Iterators

Iterator Protocol

1 class CountDown :
2 " " " Iterator that counts down from n to 1. " " "
3 def __init__ ( self , start ) :
4 self . current = start
5
6 def __iter__ ( self ) :
7 return self
8
9 def __next__ ( self ) :
10 if self . current <= 0:
11 raise StopIteration
12 self . current -= 1
13 return self . current + 1

Iterable vs Iterator

1 # List is iterable but not an iterator

2 my_list = [1 , 2 , 3]
3 my_iter = iter ( my_list )
4 print ( next ( my_iter ) ) # 1

Built-in Iterator Functions

1 # enumerate adds indices

2 for idx , val in enumerate ([ ’a ’ , ’b ’ , ’c ’ ]) :
3 print ( idx , val )
4
5 # zip combines iterables
6 for a , b in zip ([1 , 2] , [ ’x ’ , ’y ’ ]) :
7 print (a , b )

Infinite Iterators

15
 16

1 from itertools import count , cycle , repeat

2
3 # count ( start =0 , step =1) - infinite counter
4 for i in count (5 , 2) :
5 if i > 15: break
6
7 # cycle ( iterable ) - infinitely repeats
8 for item in cycle ([ ’ red ’ , ’ green ’ ]) :
9 pass

Generators

Generator Functions

1 def countdown ( n ) :
2 " " " Generator that counts down from n . " " "
3 while n > 0:
4 yield n
5 n -= 1
6
7 gen = countdown (5)
8 print ( next ( gen ) ) # 5
9 print ( next ( gen ) ) # 4

Generator Expressions

1 # List comprehension - creates entire list

2 squares_list = [ x **2 for x in range (1000) ]
3
4 # Generator expression - produces values on demand
5 squares_gen = ( x **2 for x in range (1000) )
6 total = sum ( x **2 for x in range (1000000) )

Memory Efficiency

1 def read_large_file ( file_path ) :

2 " " " Read file line by line without loading entire file . " " "
3 with open ( file_path ) as file :
4 for line in file :
5 yield line . strip ()
6

7 for line in read_large_file ( ’ huge_dataset . txt ’) :

8 process_line ( line )

Generator Pipeline

1 def read_data ( source ) :

2 for item in source :
3 yield item

16
 17

4
5 def filter_data ( stream , condition ) :
6 for item in stream :
7 if condition ( item ) :
8 yield item
9
10 def transform_data ( stream , func ) :
11 for item in stream :
12 yield func ( item )
13
14 pipeline = transform_data (
15 filter_data (
16 read_data ( data_source ) ,
17 lambda x : x > 0
18 ),
19 lambda x : x ** 2
20 )

yield from
Theory: The yield from expression allows a generator to delegate part of its operations to
another generator or iterable. It yields all values from the sub-iterator, making it easier to
compose generators and flatten nested generator structures.
1 def chain (* iterables ) :
2 " " " Chain multiple iterables together . " " "
3 for iterable in iterables :
4 yield from iterable
5
6 # Equivalent to :
7 def chain_alt (* iterables ) :
8 for iterable in iterables :
9 for item in iterable :
10 yield item

Generator State and Closing

Theory: Generators maintain their state between yields, including local variables and instruction
pointer. They can be closed using .close() which raises GeneratorExit inside the generator.
The yield expression can also receive values using .send().
1 def echo () :
2 " " " Generator that echoes received values . " " "
3 while True :
4 received = yield
5 print ( f " Received : { received } " )
6
7 gen = echo ()
8 next ( gen ) # Advance to first yield
9 gen . send ( ’ hello ’) # Send value to generator
10 gen . close () # Close generator

Function Decorators
Theory: Decorators are functions that modify the behavior of other functions without changing
their source code. They use the @decorator syntax and are applied at function definition time.

17
 18

1 def timer ( func ) :

2 " " " Decorator that measures execution time . " " "
3 import time
4 def wrapper (* args , ** kwargs ) :
5 start = time . time ()
6 result = func (* args , ** kwargs )
7 print ( f " { func . __name__ } took { time . time () - start :.4 f } s " )
8 return result
9 return wrapper
10
11 @timer
12 def slow_function () :
13 import time
14 time . sleep (1)

Decorator Syntax
Theory: The @decorator syntax is syntactic sugar for function = decorator(function).
1 # These are equivalent
2 @decorator
3 def func () :
4 pass
5
6 # Same as
7 def func () :
8 pass
9 func = decorator ( func )

Decorators with Arguments

Theory: Decorators can accept arguments by creating nested functions.
1 def repeat ( n_times ) :
2 " " " Decorator factory that repeats function n times . " " "
3 def decorator ( func ) :
4 def wrapper (* args , ** kwargs ) :
5 for _ in range ( n_times ) :
6 result = func (* args , ** kwargs )
7 return result
8 return wrapper
9 return decorator
10
11 @repeat (3)
12 def greet ( name ) :
13 print ( f " Hello { name } " )

Preserving Function Metadata

Theory: Use [Link] to preserve function metadata.
1 from functools import wraps
2
3 def logged ( func ) :
4 @wraps ( func )
5 def wrapper (* args , ** kwargs ) :
6 print ( f " Calling { func . __name__ } " )
7 return func (* args , ** kwargs )
8 return wrapper

18
 19

9
10 @logged
11 def add (x , y ) :
12 " " " Add two numbers . " " "
13 return x + y
14
15 print ( add . __name__ ) # ’ add ’
16 print ( add . __doc__ ) # ’ Add two numbers . ’

Class Decorators
Theory: Decorators can also be applied to classes to modify or enhance behavior.
1 def singleton ( cls ) :
2 " " " Decorator that ensures only one instance of class . " " "
3 instances = {}
4 def get_instance (* args , ** kwargs ) :
5 if cls not in instances :
6 instances [ cls ] = cls (* args , ** kwargs )
7 return instances [ cls ]
8 return get_instance
9
10 @singleton
11 class Da tabaseConnection :
12 def __init__ ( self ) :
13 self . connected = False

Property Decorator
Theory: The @property decorator transforms methods into managed attributes.
1 class Temperature :
2 def __init__ ( self , celsius ) :
3 self . _celsius = celsius
4
5 @property
6 def celsius ( self ) :
7 return self . _celsius
8
9 @property
10 def fahrenheit ( self ) :
11 return ( self . _celsius * 9/5) + 32
12
13 @celsius . setter
14 def celsius ( self , value ) :
15 if value < -273.15:
16 raise ValueError ( " Temperature below absolute zero " )
17 self . _celsius = value

Common Decorator Patterns

Theory: Patterns include logging, timing, memoization, access control, retry, and validation.
1 def memoize ( func ) :
2 " " " Caching decorator for expensive functions . " " "
3 cache = {}
4 @wraps ( func )
5 def wrapper (* args ) :
6 if args in cache :

19
 20

7 return cache [ args ]

8 result = func (* args )
9 cache [ args ] = result
10 return result
11 return wrapper
12
13 def retry ( max_attempts =3) :
14 " " " Retry decorator for unreliable operations . " " "
15 def decorator ( func ) :
16 @wraps ( func )
17 def wrapper (* args , ** kwargs ) :
18 for attempt in range ( max_attempts ) :
19 try :
20 return func (* args , ** kwargs )
21 except Exception :
22 if attempt == max_attempts - 1:
23 raise
24 return None
25 return wrapper
26 return decorator

Stacked Decorators
Theory: Multiple decorators can be stacked; execution order is bottom to top.
1 @timer
2 @logged
3 @memoize
4 def fibonacci ( n ) :
5 " " " Calculate nth Fibonacci number . " " "
6 if n < 2:
7 return n
8 return fibonacci (n -1) + fibonacci (n -2)

Decorator Factories
Theory: Factories parameterize decorators.
1 def rate_limit ( max_calls , period ) :
2 " " " Decorator factory for rate limiting . " " "
3 def decorator ( func ) :
4 calls = []
5 @wraps ( func )
6 def wrapper (* args , ** kwargs ) :
7 from time import time
8 now = time ()
9 calls [:] = [ c for c in calls if now - c < period ]
10 if len ( calls ) >= max_calls :
11 raise Exception ( " Rate limit exceeded " )
12 calls . append ( now )
13 return func (* args , ** kwargs )
14 return wrapper
15 return decorator
16
17 @rate_limit ( max_calls =5 , period =60)
18 def api_call () :
19 " " " Limited to 5 calls per minute . " " "
20 return " API response "

20
 21

Integration Examples

1 from collections import defaultdict , Counter , deque

2 from functools import wraps
3 import time
4
5 def timing ( func ) :
6 @wraps ( func )
7 def wrapper (* args , ** kwargs ) :
8 start = time . time ()
9 result = func (* args , ** kwargs )
10 print ( f " { func . __name__ }: { time . time () - start :.3 f } s " )
11 return result
12 return wrapper
13
14 def read_chunks ( file_path , chunk_size =1024) :
15 " " " Generator to read file in chunks . " " "
16 with open ( file_path , ’ rb ’) as f :
17 while True :
18 chunk = f . read ( chunk_size )
19 if not chunk :
20 break
21 yield chunk
22
23 class RollingWindow :
24 " " " Iterator that yields rolling windows over data . " " "
25 def __init__ ( self , data , window_size ) :
26 self . data = data
27 self . window_size = window_size
28 self . index = 0
29
30 def __iter__ ( self ) :
31 return self
32
33 def __next__ ( self ) :
34 if self . index + self . window_size > len ( self . data ) :
35 raise StopIteration
36 result = self . data [ self . index : self . index + self . window_size ]
37 self . index += 1
38 return result
39
40 @timing
41 def process_data ( data_stream ) :
42 " " " Process data using collections and generators . " " "
43 recent = deque ( maxlen =10)
44 word_freq = Counter ()
45 word_groups = defaultdict ( list )
46
47 for chunk in data_stream :
48 words = chunk . decode () . split ()
49 word_freq . update ( words )
50
51 for word in words :
52 word_groups [ word [0]]. append ( word )
53 recent . append ( word )
54
55 yield {
56 ’ unique_words ’: len ( word_freq ) ,
57 ’ most_common ’: word_freq . most_common (3) ,
58 ’ recent_words ’: list ( recent ) ,
59 ’ group_counts ’: { k : len ( v ) for k , v in word_groups . items () }
60 }

21
 22

Functions

Function Definition and Structure

Functions are reusable blocks of code that perform specific tasks. In Python, functions are first-
class objects, meaning they can be assigned to variables, passed as arguments, and returned
from other functions. The def keyword defines a function, followed by a name, parameters in
parentheses, and a colon. The function body must be indented.
1 def fibonacci ( N ) :
2 " " " Generate first N Fibonacci numbers . " " "
3 L = []
4 a, b = 0, 1
5 while len ( L ) < N :
6 a, b = b, a + b
7 L . append ( a )
8 return L
9
10 # Function call
11 result = fibonacci (10)
12 print ( result ) # [1 , 1 , 2 , 3 , 5 , 8 , 13 , 21 , 34 , 55]

Key Characteristics:

• Functions are defined with def statement

• Can take any number of arguments

• Return any Python object (simple or compound)

• No type declarations required

Return Values
Python functions can return multiple values by packing them into a tuple. Functions without a
return statement return None.
1 def real_imag_conj ( val ) :
2 " " " Return real part , imaginary part , and conjugate . " " "
3 return val . real , val . imag , val . conjugate ()
4
5 r , i , c = real_imag_conj (3 + 4 j )
6 print (r , i , c ) # 3.0 4.0 (3 -4 j )
7
8 def print_message ( msg ) :
9 print ( msg )
10
11 result = print_message ( " Hello " )
12 print ( result ) # None

Default Argument Values

Default arguments allow functions to be called with fewer arguments than defined. Mutable
default arguments can persist across calls—use None as default to avoid this.
1 def fibonacci (N , a =0 , b =1) :
2 L = []
3 while len ( L ) < N :
4 a, b = b, a + b
5 L . append ( a )

22
 23

6 return L
7
8 print ( fibonacci (10) ) # Defaults
9 print ( fibonacci (10 , 0 , 2) ) # Positional override
10 print ( fibonacci (10 , b =3 , a =1) ) # Keyword override
11
12 def add_item ( item , L =[]) : # Dangerous !
13 L . append ( item )
14 return L
15
16 def add_item_safe ( item , L = None ) :
17 if L is None :
18 L = []
19 L . append ( item )
20 return L

Positional and Keyword Arguments

Arguments can be passed positionally or by keyword. Keyword arguments must come after
positional arguments.
1 def greet ( name , greeting = " Hello " , punctuation = " ! " ) :
2 return f " { greeting } , { name }{ punctuation } "
3
4 print ( greet ( " Alice " , " Hi " , " ? " ) )
5 print ( greet ( name = " Bob " , punctuation = " !!! " ) )
6 print ( greet ( " Charlie " , greeting = " Hey " ) )

Variable-Length Arguments

*args - Variable Positional Arguments

Theory: Collects extra positional arguments into a tuple.
1 def catch_all (* args ) :
2 print ( " args = " , args )
3
4 catch_all (1 , 2 , 3)
5 catch_all ( ’a ’ , ’b ’ , ’c ’ , ’d ’)
6
7 def sum_all (* args ) :
8 return sum ( args )
9
10 print ( sum_all (1 , 2 , 3 , 4 , 5) )

**kwargs - Variable Keyword Arguments

Collects extra keyword arguments into a dictionary.
1 def catch_all_kwargs (** kwargs ) :
2 print ( " kwargs = " , kwargs )
3
4 catch_all_kwargs ( a =4 , b =5)
5 catch_all_kwargs ( name = " Alice " , age =30)
6
7 def configure (** kwargs ) :
8 for key , value in kwargs . items () :
9 print ( f " Setting { key } = { value } " )
10

23
 24

11 configure ( host = " localhost " , port =8080 , debug = True )

Combining *args and **kwargs

Functions can accept both variable positional and keyword arguments.
1 def catch_both (* args , ** kwargs ) :
2 print ( " args = " , args )
3 print ( " kwargs = " , kwargs )
4
5 catch_both (1 , 2 , 3 , a =4 , b =5)
6
7 def add (a , b , c ) :
8 return a + b + c
9
10 numbers = [1 , 2 , 3]
11 print ( add (* numbers ) )
12
13 kwargs = { ’a ’: 10 , ’b ’: 20 , ’c ’: 30}
14 print ( add (** kwargs ) )

Positional-Only and Keyword-Only Arguments

Python 3.8+ supports positional-only (before /) and keyword-only (after *).
1 def precise_function (a , b , / , c , d , * , e , f ) :
2 return a + b + c + d + e + f
3
4 print ( precise_function (1 , 2 , 3 , 4 , e =5 , f =6) )
5 print ( precise_function (1 , 2 , c =3 , d =4 , e =5 , f =6) )

Lambda Expressions

Lambda Syntax and Theory

Lambda expressions create anonymous, single-expression functions.
1 square = lambda x : x ** 2
2 print ( square (5) )
3
4 add = lambda x , y : x + y
5 constant = lambda : 42

Use Cases: map, filter, reduce, sorting

Lambdas are useful for sorting, transformation, filtering, and reducing iterables.
1 # Sorting
2 data = [{ ’ first ’: ’ Guido ’} , { ’ first ’: ’ Grace ’} , { ’ first ’: ’ Alan ’ }]
3 sorted_by_first = sorted ( data , key = lambda item : item [ ’ first ’ ])
4
5 # map
6 numbers = [1 , 2 , 3]
7 squared = list ( map ( lambda x : x **2 , numbers ) )
8
9 # filter
10 evens = list ( filter ( lambda x : x % 2 == 0 , range (1 ,11) ) )

24
 25

11
12 # reduce
13 from functools import reduce
14 total = reduce ( lambda x , y : x +y , [1 ,2 ,3 ,4 ,5])

Nested Lambda Functions

Lambdas can return other lambdas for higher-order functionality.
1 adder = lambda x : ( lambda y : x + y )
2 add5 = adder (5)
3 print ( add5 (10) )

Higher-Order Functions

Functions as First-Class Objects

Functions can be passed, returned, and assigned.
1 def greet ( name ) : return f " Hello , { name } "
2 say_hello = greet
3 print ( say_hello ( " Alice " ) )
4
5 def apply_twice ( func , arg ) : return func ( func ( arg ) )

Function Composition
Combine functions to form new functions.
1 def compose (f , g ) :
2 return lambda x : f ( g ( x ) )
3
4 add_then_square = compose ( lambda x : x **2 , lambda x : x +1)
5 print ( add_then_square (3) )

Function Decorators

Basic Decorator Pattern

Decorators modify function behavior without changing its code.
1 def timer ( func ) :
2 import time
3 def wrapper (* args , ** kwargs ) :
4 start = time . time ()
5 result = func (* args , ** kwargs )
6 end = time . time ()
7 print ( f " { func . __name__ } took { end - start :.4 f } seconds " )
8 return result
9 return wrapper
10
11 @timer
12 def slow_function () :
13 import time
14 time . sleep (1)
15 return " Done "

25
 26

Scoping Rules

LEGB Rule
Python resolves names using Local, Enclosing, Global, Built-in scopes.
1 x = " global "
2
3 def outer () :
4 x = " enclosing "
5 def inner () :
6 x = " local "
7 print ( " inner : " , x )
8 inner ()
9 print ( " outer : " , x )
10
11 outer ()
12 print ( " global : " , x )
13
14 # nonlocal example
15 def counter () :
16 count = 0
17 def increment () :
18 nonlocal count
19 count += 1
20 return count
21 return increment
22
23 c = counter ()
24 print ( c () )
25 print ( c () )

Summary
Concept Purpose Syntax Best Used For
def functions Named, reusable code def name(params): Complex logic, reuse
Lambda Anonymous, single-expression lambda params: expr Simple callbacks, key functions
*args Variable positional args def f(*args): Wrappers, variable inputs
**kwargs Variable keyword args def f(**kwargs): Configuration, wrappers
Default args Optional parameters def f(x=default): Common-case values
Higher-order Functions as objects def f(): return g Composition, decorators

Why OOP in Data Science?

Object-oriented programming transforms data science work from exploratory scripts into main-
tainable, scalable code. It enables:

• Encapsulation: Bundling data and operations into cohesive units

• State Management: Tracking transformations and preserving fitted parameters
• Code Organization: Moving from scattered code to structured components
• Reusability: Building components that can be shared across projects

Core Class Structure

26
 27

1 class DataProcessor :
2 " " " Template for processing data with configurable operations . " " "
3
4 def __init__ ( self , name : str , scale_factor : float = 1.0) :
5 """
6 Initialize processor with configuration .
7
8 Args :
9 name : Identifier for the processor instance
10 scale_factor : Multiplier to apply during processing
11 """
12 self . name = name # Public attribute
13 self . _scale_factor = scale_factor # Protected by convention
14 self . __processed_data = None # Private ( name mangling )
15 self . status = " initialized "
16 self . raw_data = None
17
18 def load_data ( self , data ) :
19 " " " Load data into the processor . " " "
20 print ( f " { self . name }: Loading data ... " )
21 self . raw_data = data
22 self . status = " loaded "
23
24 def process ( self ) :
25 " " " Apply configured transformations . " " "
26 if self . raw_data is not None :
27 print ( f " Processing with scale factor : { self . _scale_factor } " )
28 self . __processed_data = [ x * self . _scale_factor for x in self .
raw_data ]
29 self . status = " processed "
30 return self . __processed_data
31 else :
32 raise ValueError ( " No data loaded . Call load_data () first . " )
33
34 def get_results ( self ) :
35 " " " Safely access processed data . " " "
36 return self . __processed_data if self . __processed_data is not None else
None
37
38 def reset ( self ) :
39 " " " Reset processor state . " " "
40 self . __processed_data = None
41 self . raw_data = None
42 self . status = " reset "
43
44 def __repr__ ( self ) :
45 " " " Provide readable representation . " " "
46 return f " DataProcessor ( name = ’{ self . name } ’ , status = ’{ self . status } ’) "
47
48 def __len__ ( self ) :
49 " " " Return length of raw data if loaded . " " "
50 return len ( self . raw_data ) if self . raw_data else 0
51
52 # Usage Example
53 processor = DataProcessor ( " test_processor " , scale_factor =2.5)
54 print ( processor )
55
56 data = [1 , 2 , 3 , 4 , 5]
57 processor . load_data ( data )
58 result = processor . process ()
59 print ( f " Processed result : { result } " )
60 print ( f " Data length : { len ( processor ) } " )

27
 28

Encapsulation and Data Hiding

Encapsulation protects internal state and controls data access. This is crucial for maintaining
data integrity in pipelines.
1 class FeatureScaler :
2 " " " Scale features while protecting fitted parameters . " " "
3
4 def __init__ ( self , method = ’ standard ’) :
5 self . method = method
6 self . _fitted = False
7 self . _parameters = {} # Protected - internal use only
8 self . __cache = {} # Private - for internal caching
9
10 def fit ( self , data ) :
11 " " " Learn scaling parameters from data . " " "
12 if not data :
13 raise ValueError ( " Empty data provided " )
14
15 if self . method == ’ standard ’:
16 self . _parameters [ ’ mean ’] = sum ( data ) / len ( data )
17 self . _parameters [ ’ std ’] = (
18 sum (( x - self . _parameters [ ’ mean ’ ]) ** 2 for x in data ) / len (
data )
19 ) ** 0.5
20
21 elif self . method == ’ minmax ’:
22 self . _parameters [ ’ min ’] = min ( data )
23 self . _parameters [ ’ max ’] = max ( data )
24
25 self . _fitted = True
26 self . __cache = {} # Clear cache on new fit
27 return self
28
29 def transform ( self , data ) :
30 " " " Apply scaling using fitted parameters . " " "
31 if not self . _fitted :
32 raise RuntimeError ( " Scaler must be fitted before transform " )
33
34 # Check cache first
35 data_tuple = tuple ( data )
36 if data_tuple in self . __cache :
37 return self . __cache [ data_tuple ]. copy ()
38
39 # Apply transformation
40 if self . method == ’ standard ’:
41 result = [
42 ( x - self . _parameters [ ’ mean ’ ]) / self . _parameters [ ’ std ’]
43 for x in data
44 ]
45
46 elif self . method == ’ minmax ’:
47 range_val = self . _parameters [ ’ max ’] - self . _parameters [ ’ min ’]
48 if range_val == 0:
49 result = [0.5] * len ( data )
50 else :
51 result = [
52 ( x - self . _parameters [ ’ min ’ ]) / range_val
53 for x in data
54 ]
55
56 # Cache result
57 self . __cache [ data_tuple ] = result . copy ()

28
 29

58 return result
59
60 def fit_transform ( self , data ) :
61 " " " Convenience method combining fit and transform . " " "
62 self . fit ( data )
63 return self . transform ( data )
64
65 @property
66 def parameters ( self ) :
67 " " " Read - only access to parameters . " " "
68 return self . _parameters . copy ()
69
70 @property
71 def is_fitted ( self ) :
72 " " " Check if scaler has been fitted . " " "
73 return self . _fitted
74
75 # Usage Example
76 scaler = FeatureScaler ( method = ’ standard ’)
77 data = [10 , 20 , 30 , 40 , 50]
78
79 # Fit and transform
80 scaled_data = scaler . fit_transform ( data )
81 print ( f " Scaled data : { scaled_data } " )
82 print ( f " Parameters : { scaler . parameters } " )
83
84 # Transform new data using same parameters
85 new_data = [15 , 25 , 35]
86 transformed = scaler . transform ( new_data )
87 print ( f " Transformed new data : { transformed } " )

Inheritance and Polymorphism

Inheritance enables code reuse and establishes clear hierarchies. Polymorphism allows different
objects to respond to the same interface.
1 from abc import ABC , abstractmethod
2 import math
3
4 class BaseModel ( ABC ) :
5 " " " Abstract base class for all models . " " "
6
7 def __init__ ( self , name ) :
8 self . name = name
9 self . _fitted = False
10 self . _metrics = {}
11
12 @abstractmethod
13 def fit ( self , X , y ) :
14 " " " Train the model on data . " " "
15 pass
16
17 @abstractmethod
18 def predict ( self , X ) :
19 " " " Make predictions for input data . " " "
20 pass
21
22 def score ( self , X , y ) :
23 " " " Calculate prediction accuracy . " " "
24 predictions = self . predict ( X )
25 correct = sum (1 for p , t in zip ( predictions , y ) if p == t )
26 accuracy = correct / len ( y )

29
 30

27 self . _metrics [ ’ accuracy ’] = accuracy

28 return accuracy
29
30 def get_metrics ( self ) :
31 " " " Return model performance metrics . " " "
32 return self . _metrics . copy ()
33
34 def __repr__ ( self ) :
35 return f " { self . __class__ . __name__ }( name = ’{ self . name } ’ , fitted ={ self .
_fitted }) "
36
37
38 class Lo gisticRegression ( BaseModel ) :
39 " " " Logistic regression classifier . " " "
40
41 def __init__ ( self , name , learning_rate =0.01 , iterations =1000) :
42 super () . __init__ ( name )
43 self . learning_rate = learning_rate
44 self . iterations = iterations
45 self . weights = None
46 self . bias = None
47
48 def _sigmoid ( self , z ) :
49 " " " Sigmoid activation function . " " "
50 return 1 / (1 + math . exp ( - z ) )
51
52 def fit ( self , X , y ) :
53 " " " Train using gradient descent . " " "
54 n_samples = len ( X )
55 n_features = len ( X [0])
56
57 # Initialize parameters
58 self . weights = [0.0] * n_features
59 self . bias = 0.0
60
61 # Gradient descent
62 for _ in range ( self . iterations ) :
63 # Forward pass
64 linear_predictions = [
65 sum ( w * x for w , x in zip ( self . weights , xi ) ) + self . bias
66 for xi in X
67 ]
68 predictions = [ self . _sigmoid ( z ) for z in linear_predictions ]
69
70 # Gradient calculation
71 dw = [0.0] * n_features
72 db = 0.0
73
74 for i in range ( n_samples ) :
75 error = predictions [ i ] - y [ i ]
76 for j in range ( n_features ) :
77 dw [ j ] += (1/ n_samples ) * error * X [ i ][ j ]
78 db += (1/ n_samples ) * error
79
80 # Update parameters
81 for j in range ( n_features ) :
82 self . weights [ j ] -= self . learning_rate * dw [ j ]
83 self . bias -= self . learning_rate * db
84
85 self . _fitted = True
86 return self
87
88 def predict ( self , X ) :

30
 31

89 " " " Make binary predictions . " " "

90 if not self . _fitted :
91 raise RuntimeError ( " Model must be fitted before prediction " )
92
93 predictions = []
94 for xi in X :
95 linear = sum ( w * x for w , x in zip ( self . weights , xi ) ) + self . bias
96 prob = self . _sigmoid ( linear )
97 predictions . append (1 if prob >= 0.5 else 0)
98
99 return predictions
100
101
102 class DecisionTree ( BaseModel ) :
103 " " " Simple decision tree classifier . " " "
104
105 class Node :
106 " " " Internal tree node . " " "
107 def __init__ ( self , feature = None , threshold = None , left = None , right = None ,
value = None ) :
108 self . feature = feature
109 self . threshold = threshold
110 self . left = left
111 self . right = right
112 self . value = value # For leaf nodes
113
114 def __init__ ( self , name , max_depth =5) :
115 super () . __init__ ( name )
116 self . max_depth = max_depth
117 self . tree = None
118
119 def _gini ( self , groups , classes ) :
120 " " " Calculate Gini impurity . " " "
121 n_instances = sum ( len ( group ) for group in groups )
122 gini = 0.0
123
124 for group in groups :
125 size = len ( group )
126 if size == 0:
127 continue
128
129 score = 0.0
130 for class_val in classes :
131 proportion = [ row [ -1] for row in group ]. count ( class_val ) / size
132 score += proportion ** 2
133
134 gini += (1 - score ) * ( size / n_instances )
135
136 return gini
137
138 def _split ( self , data , feature , threshold ) :
139 " " " Split data based on feature and threshold . " " "
140 left = [ row for row in data if row [ feature ] <= threshold ]
141 right = [ row for row in data if row [ feature ] > threshold ]
142 return left , right
143
144 def _get_best_split ( self , data ) :
145 " " " Find best feature and threshold to split on . " " "
146 class_values = list ( set ( row [ -1] for row in data ) )
147 best_feature , best_threshold , best_score , best_groups = None , None ,
float ( ’ inf ’) , None
148
149 for feature in range ( len ( data [0]) - 1) :

31
 32

150 thresholds = list ( set ( row [ feature ] for row in data ) )

151
152 for threshold in thresholds :
153 groups = self . _split ( data , feature , threshold )
154 gini = self . _gini ( groups , class_values )
155
156 if gini < best_score :
157 best_feature , best_threshold , best_score , best_groups = (
158 feature , threshold , gini , groups
159 )
160
161 return {
162 ’ feature ’: best_feature ,
163 ’ threshold ’: best_threshold ,
164 ’ groups ’: best_groups
165 }
166
167 def _build_tree ( self , data , depth ) :
168 " " " Recursively build decision tree . " " "
169 classes = list ( set ( row [ -1] for row in data ) )
170
171 # Check stopping conditions
172 if len ( classes ) == 1 or depth >= self . max_depth :
173 # Create leaf node with majority class
174 class_counts = { c : [ row [ -1] for row in data ]. count ( c ) for c in
classes }
175 leaf_value = max ( class_counts , key = class_counts . get )
176 return self . Node ( value = leaf_value )
177
178 # Find best split
179 split = self . _get_best_split ( data )
180
181 if split [ ’ groups ’] is None :
182 class_counts = { c : [ row [ -1] for row in data ]. count ( c ) for c in
classes }
183 leaf_value = max ( class_counts , key = class_counts . get )
184 return self . Node ( value = leaf_value )
185
186 # Recursively build children
187 left = self . _build_tree ( split [ ’ groups ’ ][0] , depth + 1)
188 right = self . _build_tree ( split [ ’ groups ’ ][1] , depth + 1)
189
190 return self . Node (
191 feature = split [ ’ feature ’] ,
192 threshold = split [ ’ threshold ’] ,
193 left = left ,
194 right = right
195 )
196
197 def fit ( self , X , y ) :
198 " " " Build decision tree from training data . " " "
199 # Combine X and y for tree building
200 data = [ list ( x ) + [ y [ i ]] for i , x in enumerate ( X ) ]
201 self . tree = self . _build_tree ( data , 0)
202 self . _fitted = True
203 return self
204
205 def _predict_row ( self , node , row ) :
206 " " " Predict class for a single row . " " "
207 if node . value is not None :
208 return node . value
209
210 if row [ node . feature ] <= node . threshold :

32
 33

211 return self . _predict_row ( node . left , row )

212 else :
213 return self . _predict_row ( node . right , row )
214
215 def predict ( self , X ) :
216 " " " Make predictions for input data . " " "
217 if not self . _fitted :
218 raise RuntimeError ( " Model must be fitted before prediction " )
219
220 return [ self . _predict_row ( self . tree , row ) for row in X ]
221
222
223 # Usage Example - Polymorphism in action
224 def evaluate_model ( model , X_train , y_train , X_test , y_test ) :
225 " " " Function that works with any model implementing fit / predict . " " "
226 print ( f " \ nEvaluating { model . name }: " )
227
228 # Fit model
229 model . fit ( X_train , y_train )
230 print ( f " Model fitted : { model . _fitted } " )
231
232 # Make predictions
233 predictions = model . predict ( X_test )
234 print ( f " Predictions : { predictions } " )
235
236 # Calculate accuracy
237 accuracy = model . score ( X_test , y_test )
238 print ( f " Accuracy : { accuracy :.2 f } " )
239 print ( f " Metrics : { model . get_metrics () } " )
240
241 return accuracy
242
243 # Sample data
244 X_train = [[1 , 2] , [2 , 3] , [3 , 4] , [4 , 5]]
245 y_train = [0 , 0 , 1 , 1]
246 X_test = [[1.5 , 2.5] , [3.5 , 4.5]]
247 y_test = [0 , 1]
248
249 # Both models can be used interchangeably
250 log_reg = LogisticRegression ( " LR_Model " , learning_rate =0.1)
251 tree = DecisionTree ( " DT_Model " , max_depth =3)
252
253 # Same function works for both
254 evaluate_model ( log_reg , X_train , y_train , X_test , y_test )
255 evaluate_model ( tree , X_train , y_train , X_test , y_test )

Composition over Inheritance

Composition builds complex objects by combining simpler ones, often providing more flexibility
than deep inheritance.
1 class DataLoader :
2 " " " Handle data loading from various sources . " " "
3
4 def __init__ ( self ) :
5 self . source = None
6 self . loaded_data = None
7
8 def from_csv ( self , filepath ) :
9 " " " Load data from CSV file . " " "
10 print ( f " Loading CSV from { filepath } " )
11 # Simulated CSV loading

33
 34

12 self . loaded_data = [
13 [1 , 2 , 3] ,
14 [4 , 5 , 6] ,
15 [7 , 8 , 9]
16 ]
17 self . source = filepath
18 return self . loaded_data
19
20 def from_json ( self , filepath ) :
21 " " " Load data from JSON file . " " "
22 print ( f " Loading JSON from { filepath } " )
23 # Simulated JSON loading
24 self . loaded_data = [
25 { ’ feature1 ’: 1 , ’ feature2 ’: 2} ,
26 { ’ feature1 ’: 3 , ’ feature2 ’: 4} ,
27 { ’ feature1 ’: 5 , ’ feature2 ’: 6}
28 ]
29 self . source = filepath
30 return self . loaded_data
31
32
33 class DataCleaner :
34 " " " Handle data cleaning operations . " " "
35
36 def __init__ ( self ) :
37 self . cleaning_log = []
38
39 def remove_nulls ( self , data ) :
40 " " " Remove rows with null values . " " "
41 if not data :
42 return data
43
44 # Handle both list of lists and list of dicts
45 if isinstance ( data [0] , dict ) :
46 cleaned = [ row for row in data if all ( v is not None for v in row .
values () ) ]
47 else :
48 cleaned = [ row for row in data if all ( v is not None for v in row ) ]
49
50 self . cleaning_log . append ( f " Removed nulls : { len ( data ) - len ( cleaned ) }
rows " )
51 return cleaned
52
53 def normalize_types ( self , data ) :
54 " " " Ensure consistent data types . " " "
55 if not data or isinstance ( data [0] , dict ) :
56 return data
57
58 # Convert all to float
59 cleaned = []
60 for row in data :
61 cleaned . append ([ float ( x ) if x is not None else 0.0 for x in row ])
62
63 self . cleaning_log . append ( " Normalized types to float " )
64 return cleaned
65
66 def get_log ( self ) :
67 " " " Return cleaning operation log . " " "
68 return self . cleaning_log . copy ()
69
70
71 class FeatureEngineer :
72 " " " Create and transform features . " " "

34
 35

73
74 def __init__ ( self ) :
75 self . features_created = []
76
77 def add_interaction ( self , data , col1 , col2 ) :
78 " " " Add interaction feature . " " "
79 if isinstance ( data [0] , dict ) :
80 for row in data :
81 row [ f ’ interaction_ { col1 } _ { col2 } ’] = row [ col1 ] * row [ col2 ]
82 else :
83 for row in data :
84 row . append ( row [ col1 ] * row [ col2 ])
85
86 self . features_created . append ( f ’ interaction_ { col1 } _ { col2 } ’)
87 return data
88
89 def add_polynomial ( self , data , col , degree =2) :
90 " " " Add polynomial features . " " "
91 if isinstance ( data [0] , dict ) :
92 for d in range (2 , degree + 1) :
93 for row in data :
94 row [ f ’{ col }^{ d } ’] = row [ col ] ** d
95 else :
96 for d in range (2 , degree + 1) :
97 for i , row in enumerate ( data ) :
98 data [ i ] = row + [ row [ col ] ** d ]
99
100 self . features_created . append ( f ’ poly_ { col } _deg { degree } ’)
101 return data
102
103
104 class ModelTrainer :
105 " " " Train and evaluate models . " " "
106
107 def __init__ ( self , model ) :
108 self . model = model
109 self . training_history = []
110
111 def train ( self , X , y , validation_split =0.2) :
112 " " " Train model with validation . " " "
113 split_idx = int ( len ( X ) * (1 - validation_split ) )
114
115 X_train , X_val = X [: split_idx ] , X [ split_idx :]
116 y_train , y_val = y [: split_idx ] , y [ split_idx :]
117
118 # Train model
119 self . model . fit ( X_train , y_train )
120
121 # Validate
122 train_score = self . model . score ( X_train , y_train )
123 val_score = self . model . score ( X_val , y_val )
124
125 self . training_history . append ({
126 ’ train_score ’: train_score ,
127 ’ val_score ’: val_score ,
128 ’ samples ’: len ( X )
129 })
130
131 return self . model
132
133 def get_history ( self ) :
134 " " " Return training history . " " "
135 return self . training_history . copy ()

35
 36

136
137
138 class D a t aSciencePipeline :
139 " " " Complete pipeline using composition . " " "
140
141 def __init__ ( self , name ) :
142 self . name = name
143 self . loader = DataLoader ()
144 self . cleaner = DataCleaner ()
145 self . engineer = FeatureEngineer ()
146 self . trainer = None
147 self . data = None
148 self . labels = None
149
150 def set_model ( self , model ) :
151 " " " Set the model to use . " " "
152 self . trainer = ModelTrainer ( model )
153 return self
154
155 def load_data ( self , source_type , filepath ) :
156 " " " Load data from source . " " "
157 if source_type == ’ csv ’:
158 self . data = self . loader . from_csv ( filepath )
159 elif source_type == ’ json ’:
160 self . data = self . loader . from_json ( filepath )
161 else :
162 raise ValueError ( f " Unknown source type : { source_type } " )
163
164 # Assume last column is target for simplicity
165 if self . data and not isinstance ( self . data [0] , dict ) :
166 self . labels = [ row [ -1] for row in self . data ]
167 self . data = [ row [: -1] for row in self . data ]
168
169 return self
170
171 def clean_data ( self ) :
172 " " " Apply cleaning operations . " " "
173 if self . data :
174 self . data = self . cleaner . remove_nulls ( self . data )
175 self . data = self . cleaner . normalize_types ( self . data )
176 return self
177
178 def engineer_features ( self ) :
179 " " " Create new features . " " "
180 if self . data and len ( self . data [0]) >= 2:
181 self . data = self . engineer . add_interaction ( self . data , 0 , 1)
182 self . data = self . engineer . add_polynomial ( self . data , 0 , degree =2)
183 return self
184
185 def run ( self ) :
186 " " " Execute the complete pipeline . " " "
187 if not self . trainer :
188 raise RuntimeError ( " No model set . Call set_model () first . " )
189
190 print ( f " \ nRunning pipeline : { self . name } " )
191 print ( " = " * 40)
192
193 # Train model
194 self . trainer . train ( self . data , self . labels )
195
196 # Report results
197 print ( f " Cleaning log : { self . cleaner . get_log () } " )
198 print ( f " Features created : { self . engineer . features_created } " )

36
 37

199 print ( f " Training history : { self . trainer . get_history () } " )

200
201 return self . trainer . model
202
203 # Usage Example
204 pipeline = DataSciencePipeline ( " Experiment_1 " )
205 pipeline . set_model ( LogisticRegression ( " LR " , learning_rate =0.01) )
206 pipeline . load_data ( ’ csv ’ , ’ data . csv ’)
207 pipeline . clean_data ()
208 pipeline . engineer_features ()
209 model = pipeline . run ()

Magic Methods for Data Science

Special methods allow custom objects to integrate seamlessly with Python’s syntax and built-in
functions.
1 class DataFrame :
2 " " " Custom DataFrame implementation with magic methods . " " "
3
4 def __init__ ( self , data , columns = None ) :
5 """
6 Initialize DataFrame .
7
8 Args :
9 data : List of lists or list of dicts
10 columns : Column names ( for list of lists )
11 """
12 self . _data = []
13 self . columns = columns
14 self . _shape = None
15 self . _dtypes = {}
16
17 if data :
18 self . _initialize_data ( data )
19
20 def _initialize_data ( self , data ) :
21 " " " Initialize internal data structure . " " "
22 if isinstance ( data [0] , dict ) :
23 # List of dictionaries
24 self . columns = list ( data [0]. keys () )
25 self . _data = [[ row [ col ] for col in self . columns ] for row in data ]
26 else :
27 # List of lists
28 self . _data = [ list ( row ) for row in data ]
29 if not self . columns :
30 self . columns = [ f " col_ { i } " for i in range ( len ( data [0]) ) ]
31
32 self . _infer_dtypes ()
33 self . _update_shape ()
34
35 def _infer_dtypes ( self ) :
36 " " " Infer column data types . " " "
37 if not self . _data :
38 return
39
40 for i , col in enumerate ( self . columns ) :
41 col_data = [ row [ i ] for row in self . _data if row [ i ] is not None ]
42 if all ( isinstance (x , int ) for x in col_data ) :
43 self . _dtypes [ col ] = ’ int ’
44 elif all ( isinstance (x , float ) for x in col_data ) :
45 self . _dtypes [ col ] = ’ float ’

37
 38

46 elif all ( isinstance (x , str ) for x in col_data ) :

47 self . _dtypes [ col ] = ’ str ’
48 else :
49 self . _dtypes [ col ] = ’ object ’
50
51 def _update_shape ( self ) :
52 " " " Update shape information . " " "
53 self . _shape = ( len ( self . _data ) , len ( self . columns ) if self . _data else 0)
54
55 # Container Magic Methods
56 def __len__ ( self ) :
57 " " " Return number of rows . " " "
58 return len ( self . _data )
59
60 def __getitem__ ( self , key ) :
61 """
62 Support indexing and slicing .
63 - Integer : returns row
64 - String : returns column
65 - Slice : returns DataFrame subset
66 """
67 if isinstance ( key , int ) :
68 # Row indexing
69 return dict ( zip ( self . columns , self . _data [ key ]) )
70
71 elif isinstance ( key , str ) :
72 # Column indexing
73 if key not in self . columns :
74 raise KeyError ( f " Column ’{ key } ’ not found " )
75
76 col_idx = self . columns . index ( key )
77 return [ row [ col_idx ] for row in self . _data ]
78
79 elif isinstance ( key , slice ) :
80 # Row slicing
81 new_data = self . _data [ key ]
82 return DataFrame ( new_data , self . columns . copy () )
83
84 elif isinstance ( key , list ) :
85 # Multiple columns
86 if all ( isinstance (k , str ) for k in key ) :
87 col_indices = [ self . columns . index ( k ) for k in key ]
88 new_data = [[ row [ i ] for i in col_indices ] for row in self . _data ]
89 return DataFrame ( new_data , key . copy () )
90
91 raise TypeError ( f " Invalid key type : { type ( key ) } " )
92
93 def __setitem__ ( self , key , value ) :
94 " " " Support column assignment . " " "
95 if isinstance ( key , str ) :
96 if key in self . columns :
97 # Update existing column
98 col_idx = self . columns . index ( key )
99 for i , row in enumerate ( self . _data ) :
100 row [ col_idx ] = value [ i ] if hasattr ( value , ’ __getitem__ ’)
else value
101 else :
102 # Add new column
103 self . columns . append ( key )
104 for i , row in enumerate ( self . _data ) :
105 val = value [ i ] if hasattr ( value , ’ __getitem__ ’) else value
106 row . append ( val )
107

38
 39

108 self . _infer_dtypes ()

109 self . _update_shape ()
110
111 def __delitem__ ( self , key ) :
112 " " " Support column deletion . " " "
113 if key in self . columns :
114 col_idx = self . columns . index ( key )
115 self . columns . pop ( col_idx )
116 for row in self . _data :
117 row . pop ( col_idx )
118 self . _update_shape ()
119
120 # Numeric Magic Methods
121 def __add__ ( self , other ) :
122 " " " Element - wise addition . " " "
123 result = DataFrame ([] , self . columns . copy () )
124
125 if isinstance ( other , ( int , float ) ) :
126 # Scalar addition
127 result . _data = [[ x + other for x in row ] for row in self . _data ]
128
129 elif isinstance ( other , DataFrame ) :
130 # DataFrame addition
131 if self . _shape != other . _shape :
132 raise ValueError ( " DataFrames must have same shape " )
133
134 result . _data = [
135 [ self . _data [ i ][ j ] + other . _data [ i ][ j ] for j in range ( len ( self .
columns ) ) ]
136 for i in range ( len ( self . _data ) )
137 ]
138
139 return result
140
141 def __sub__ ( self , other ) :
142 " " " Element - wise subtraction . " " "
143 result = DataFrame ([] , self . columns . copy () )
144
145 if isinstance ( other , ( int , float ) ) :
146 result . _data = [[ x - other for x in row ] for row in self . _data ]
147
148 elif isinstance ( other , DataFrame ) :
149 if self . _shape != other . _shape :
150 raise ValueError ( " DataFrames must have same shape " )
151
152 result . _data = [
153 [ self . _data [ i ][ j ] - other . _data [ i ][ j ] for j in range ( len ( self .
columns ) ) ]
154 for i in range ( len ( self . _data ) )
155 ]
156
157 return result
158
159 def __mul__ ( self , other ) :
160 " " " Element - wise multiplication . " " "
161 result = DataFrame ([] , self . columns . copy () )
162
163 if isinstance ( other , ( int , float ) ) :
164 result . _data = [[ x * other for x in row ] for row in self . _data ]
165
166 elif isinstance ( other , DataFrame ) :
167 if self . _shape != other . _shape :
168 raise ValueError ( " DataFrames must have same shape " )

39
 40

169
170 result . _data = [
171 [ self . _data [ i ][ j ] * other . _data [ i ][ j ] for j in range ( len ( self .
columns ) ) ]
172 for i in range ( len ( self . _data ) )
173 ]
174
175 return result
176
177 # Comparison Magic Methods
178 def __eq__ ( self , other ) :
179 " " " Element - wise equality comparison . " " "
180 result = DataFrame ([] , self . columns . copy () )
181
182 if isinstance ( other , ( int , float , str ) ) :
183 result . _data = [[ x == other for x in row ] for row in self . _data ]
184
185 return result
186
187 def __gt__ ( self , other ) :
188 " " " Element - wise greater than comparison . " " "
189 result = DataFrame ([] , self . columns . copy () )
190
191 if isinstance ( other , ( int , float ) ) :
192 result . _data = [[ x > other for x in row ] for row in self . _data ]
193
194 return result
195
196 def __lt__ ( self , other ) :
197 " " " Element - wise less than comparison . " " "
198 result = DataFrame ([] , self . columns . copy () )
199
200 if isinstance ( other , ( int , float ) ) :
201 result . _data = [[ x < other for x in row ] for row in self . _data ]
202
203 return result
204
205 # String Representation
206 def __repr__ ( self ) :
207 " " " Detailed string representation . " " "
208 if not self . _data :
209 return " Empty DataFrame "
210
211 lines = []
212 lines . append ( f " DataFrame : { self . _shape [0]} rows { self . _shape [1]}
columns " )
213 lines . append ( f " Columns : { ’ , ’. join ( self . columns ) } " )
214 lines . append ( f " Types : { self . _dtypes } " )
215
216 # Show first few rows
217 lines . append ( " \ nFirst 5 rows : " )
218 header = " " . join ( f " { col : <10} " for col in self . columns [:5])
219 lines . append ( header )
220 lines . append ( " -" * len ( header ) )
221
222 for i , row in enumerate ( self . _data [:5]) :
223 row_str = " " . join ( f " { str ( x ) : <10} " [:10] for x in row [:5])
224 lines . append ( row_str )
225
226 if len ( self . _data ) > 5:
227 lines . append ( " ... " )
228
229 return " \ n " . join ( lines )

40
 41

230
231 def __str__ ( self ) :
232 " " " Simple string representation . " " "
233 return f " DataFrame ({ self . _shape [0]} x { self . _shape [1]}) "
234
235 # Callable Magic Method
236 def __call__ ( self , * args , ** kwargs ) :
237 " " " Make DataFrame callable for common operations . " " "
238 if args and callable ( args [0]) :
239 # Apply function to all elements
240 func = args [0]
241 result = DataFrame ([] , self . columns . copy () )
242 result . _data = [[ func ( x ) for x in row ] for row in self . _data ]
243 return result
244 return self
245
246 # Context Manager Magic Methods
247 def __enter__ ( self ) :
248 " " " Enter context manager . " " "
249 print ( f " Entering DataFrame context : { self . _shape } " )
250 self . _backup = [ row . copy () for row in self . _data ]
251 return self
252
253 def __exit__ ( self , exc_type , exc_val , exc_tb ) :
254 " " " Exit context manager . " " "
255 if exc_type is not None :
256 print ( f " Error occurred : { exc_val } " )
257 print ( " Restoring backup ... " )
258 self . _data = self . _backup
259 else :
260 print ( " Exiting DataFrame context normally " )
261 del self . _backup
262 return False # Don ’t suppress exceptions
263
264 # Iterator Magic Methods
265 def __iter__ ( self ) :
266 " " " Iterate over rows . " " "
267 self . _iter_index = 0
268 return self
269
270 def __next__ ( self ) :
271 " " " Get next row . " " "
272 if self . _iter_index >= len ( self . _data ) :
273 raise StopIteration
274
275 row = dict ( zip ( self . columns , self . _data [ self . _iter_index ]) )
276 self . _iter_index += 1
277 return row
278
279 # Utility methods
280 @property
281 def shape ( self ) :
282 " " " Return DataFrame shape . " " "
283 return self . _shape
284
285 @property
286 def dtypes ( self ) :
287 " " " Return column data types . " " "
288 return self . _dtypes . copy ()
289
290 def head ( self , n =5) :
291 " " " Return first n rows . " " "
292 return DataFrame ( self . _data [: n ] , self . columns . copy () )

41
 42

293
294 def tail ( self , n =5) :
295 " " " Return last n rows . " " "
296 return DataFrame ( self . _data [ - n :] , self . columns . copy () )
297
298 def describe ( self ) :
299 " " " Generate descriptive statistics . " " "
300 stats = {}
301 for i , col in enumerate ( self . columns ) :
302 col_data = [ row [ i ] for row in self . _data if isinstance ( row [ i ] , ( int ,
float ) ) ]
303 if col_data :
304 stats [ col ] = {
305 ’ count ’: len ( col_data ) ,
306 ’ mean ’: sum ( col_data ) / len ( col_data ) ,
307 ’ min ’: min ( col_data ) ,
308 ’ max ’: max ( col_data ) ,
309 ’ std ’: ( sum (( x - sum ( col_data ) / len ( col_data ) ) **2 for x in
col_data ) / len ( col_data ) ) **0.5
310 }
311 return stats
312
313 # Usage Example
314 print ( " = " * 50)
315 print ( " DEMONSTRATING MAGIC METHODS " )
316 print ( " = " * 50)
317
318 # Create DataFrame
319 data = [
320 [1 , 2 , 3] ,
321 [4 , 5 , 6] ,
322 [7 , 8 , 9] ,
323 [10 , 11 , 12]
324 ]
325 df = DataFrame ( data , columns =[ ’A ’ , ’B ’ , ’C ’ ])
326 print ( df )
327 print ()
328
329 # Container operations
330 print ( " Row 0: " , df [0])
331 print ( " Column A : " , df [ ’A ’ ])
332 print ( " First 2 rows :\ n " , df [:2])
333 print ()
334
335 # Assignment
336 df [ ’D ’] = [10 , 20 , 30 , 40]
337 print ( " After adding column D :\ n " , df . head (2) )
338 print ()
339
340 # Arithmetic operations
341 df2 = df + 10
342 print ( " df + 10:\ n " , df2 . head (2) )
343 print ()
344
345 # Comparisons
346 print ( " df > 5:\ n " , ( df > 5) . head (2) )
347 print ()
348
349 # Iteration
350 print ( " Iterating over rows : " )
351 for row in df . head (2) :
352 print ( row )
353 print ()

42
 43

354
355 # Context manager
356 with df as d :
357 d [ ’E ’] = [100 , 200 , 300 , 400]
358 print ( " Inside context :\ n " , d . head (2) )
359 print ()
360
361 # Callable
362 squared = df ( lambda x : x **2)
363 print ( " Squared values :\ n " , squared . head (2) )
364 print ()
365
366 # Statistics
367 print ( " Descriptive statistics : " )
368 stats = df . describe ()
369 for col , stat in stats . items () :
370 print ( f " { col }: { stat } " )

Properties for Controlled Access

Properties provide getter/setter logic while maintaining simple attribute syntax.
1 class TimeSeriesData :
2 " " " Time series data with validation and automatic alignment . " " "
3
4 def __init__ ( self , dates , values , name = " Series " ) :
5 self . _dates = list ( dates )
6 self . _values = list ( values )
7 self . _name = name
8 self . _freq = None
9 self . _cached_stats = {}
10
11 # Validate on initialization
12 self . _validate_data ()
13
14 def _validate_data ( self ) :
15 " " " Validate time series data . " " "
16 if len ( self . _dates ) != len ( self . _values ) :
17 raise ValueError ( " Dates and values must have same length " )
18
19 if len ( self . _dates ) == 0:
20 raise ValueError ( " Empty time series " )
21
22 # Check date order
23 for i in range (1 , len ( self . _dates ) ) :
24 if self . _dates [ i ] <= self . _dates [i -1]:
25 raise ValueError ( " Dates must be strictly increasing " )
26
27 @property
28 def name ( self ) :
29 " " " Get series name . " " "
30 return self . _name
31
32 @name . setter
33 def name ( self , value ) :
34 " " " Set series name with validation . " " "
35 if not isinstance ( value , str ) :
36 raise TypeError ( " Name must be a string " )
37 if not value :
38 raise ValueError ( " Name cannot be empty " )
39 self . _name = value
40 self . _cached_stats . clear () # Invalidate cache

43
 44

41
42 @property
43 def values ( self ) :
44 " " " Get values ( read - only copy ) . " " "
45 return self . _values . copy ()
46
47 @property
48 def dates ( self ) :
49 " " " Get dates ( read - only copy ) . " " "
50 return self . _dates . copy ()
51
52 @property
53 def length ( self ) :
54 " " " Get series length . " " "
55 return len ( self . _values )
56
57 @property
58 def start_date ( self ) :
59 " " " Get first date . " " "
60 return self . _dates [0]
61
62 @property
63 def end_date ( self ) :
64 " " " Get last date . " " "
65 return self . _dates [ -1]
66
67 @property
68 def date_range ( self ) :
69 " " " Get date range as tuple . " " "
70 return ( self . start_date , self . end_date )
71
72 @property
73 def freq ( self ) :
74 " " " Infer frequency from dates . " " "
75 if self . _freq is None and len ( self . _dates ) > 1:
76 # Simple frequency inference
77 diffs = [ self . _dates [ i ] - self . _dates [i -1] for i in range (1 , len (
self . _dates ) ) ]
78 if all ( d == diffs [0] for d in diffs ) :
79 self . _freq = diffs [0]
80 else :
81 self . _freq = None
82 return self . _freq

Exception Handling

Theory

Exceptions are errors detected during program execution. Python provides a robust mechanism
to handle these errors gracefully so that programs do not crash unexpectedly. Exception handling
allows developers to detect errors and respond to them in a controlled manner.
try block: Encloses code that might raise an exception.
except block: Catches and handles specific exceptions.
else block: Executes if no exception occurs in the try block.
finally block: Always executes regardless of whether an exception occurred. It is ideal for
cleanup tasks such as closing files or releasing resources.
raise: Used to manually trigger an exception.

44
 45

Common Built-in Exceptions

• ZeroDivisionError: Raised when division by zero occurs.

• ValueError: Raised when a function receives an invalid value.

• TypeError: Raised when an operation is performed on an inappropriate data type.

• FileNotFoundError: Raised when a specified file does not exist.

• KeyError: Raised when a dictionary key is not found.

• IndexError: Raised when an invalid index is used in a list.

Example Programs
Basic try/except with else and finally

1 def divide (a , b ) :
2 try :
3 result = a / b
4 except ZeroDivisionError :
5 print ( " Error : Cannot divide by zero . " )
6 result = None
7 else :
8 print ( " Division successful . " )
9 finally :
10 print ( " Cleanup actions ( if any ) go here . " )
11 return result
12
13 print ( divide (10 , 2) ) # Output : 5.0
14 print ( divide (10 , 0) ) # Error : Cannot divide by zero . None

Handling Multiple Exception Types

1 def read_file ( filename ) :

2 try :
3 with open ( filename , ’r ’) as f :
4 return f . read ()
5 except FileNotFoundError :
6 print ( f " File ’{ filename } ’ not found . " )
7 except PermissionError :
8 print ( f " Permission denied for ’{ filename } ’. " )
9 except Exception as e :
10 print ( f " An unexpected error occurred : { e } " )

Raising Exceptions Manually

Sometimes programmers need to deliberately trigger an exception when invalid conditions occur.
Python provides the raise keyword to manually generate exceptions. This is useful for enforcing
constraints and validating input data.
1 def set_age ( age ) :
2 if age < 0:
3 raise ValueError ( " Age cannot be negative " )
4 print ( f " Age set to { age } " )

45
 46

Using else and finally Together

The else block executes only when no exception occurs in the try block. The finally block
always executes regardless of whether an exception occurred or not. This combination is useful
for performing operations after successful execution while still ensuring cleanup activities are
performed.
1 def process_data ( data ) :
2 try :
3 value = int ( data )
4 except ValueError :
5 print ( " Invalid data : not an integer " )
6 else :
7 print ( f " Processing integer : { value } " )
8 finally :
9 print ( " Data processing attempt finished . " )

Debugging
Debugging is the process of identifying, analyzing, and fixing errors (bugs) in a program. It is an
essential skill in software development because even well-written programs may contain logical,
syntax, or runtime errors. Python provides several tools and techniques to assist in debugging.
Print Debugging: One of the simplest debugging techniques is inserting print() state-
ments in the code to track variable values and program execution flow.
Assertions: Assertions are used to check assumptions in the program. The syntax assert
condition, message verifies that the condition is true. If the condition evaluates to false,
Python raises an AssertionError.
Using pdb (Python Debugger): Python includes a built-in interactive debugger called
pdb. It allows developers to pause execution and inspect variables and program flow.

• [Link] trace(): Sets a breakpoint in the program.

• n: Execute the next line.

• c: Continue execution until the next breakpoint.

• p variable: Print the value of a variable.

• q: Quit the debugger.

Post-mortem Debugging: This technique starts the debugger automatically after an

exception occurs. For example, in IPython the command %pdb enables automatic debugging
after errors.
IDE Debuggers: Modern development environments such as VS Code and PyCharm pro-
vide graphical debugging tools including breakpoints, step execution, variable inspection, and
watch expressions.

Example Programs
Using Assertions

1 def average ( numbers ) :

2 assert len ( numbers ) > 0 , " List must not be empty "
3 return sum ( numbers ) / len ( numbers )
4
5 # average ([]) # Would raise AssertionError

46
 47

Inserting a Breakpoint with pdb

1 import pdb
2
3 def buggy_function ( x ) :
4 y = x + 1
5 pdb . set_trace () # Execution pauses here
6 z = y * 2
7 return z
8
9 result = buggy_function (5)
10
11 # At the ( Pdb ) prompt , you can inspect variables :
12 # p y
13 # p x
14 # then continue with :
15 # c

Post-mortem Debugging in IPython

1 # Enable automatic debugger on exception

2 % pdb on
3
4 def faulty () :
5 return 1 / 0
6
7 faulty () # After ZeroDivisionError , the debugger opens automatically

Using breakpoint() (Python 3.7+)

1 def compute (a , b ) :
2 breakpoint () # Same as pdb . set_trace () but more convenient
3 return a / b

Logging
Logging is the preferred way to record runtime information in software applications. It is more
flexible and configurable than using simple print() statements. Logging helps developers mon-
itor program behavior, diagnose problems, and maintain a record of application events.
Logging Levels (in increasing severity):

• DEBUG: Detailed diagnostic information useful for debugging.

• INFO: Confirmation that the program is functioning as expected.

• WARNING: Indicates a potential problem or unusual situation.

• ERROR: A serious issue that prevents part of the program from working.

• CRITICAL: A severe error indicating the program may not continue running.

Basic Configuration: The function [Link]() is used to configure the log-

ging system. It allows the developer to specify the logging level, output format, and destination
(such as console or file).
Components of Logging:

47
 48

• Logger: The object responsible for generating log messages.

• Handler: Determines where the log messages are sent (e.g., console or file).

• Formatter: Defines the structure and appearance of log messages.

Best Practices:

• Use loggers named after modules using name .

• Avoid using print() statements in production code; use logging instead.

• Configure logging once at the start of the application.

Example Programs

Basic Logging to Console

1 import logging
2
3 logging . basicConfig ( level = logging . INFO ,
4 format = ’ %( asctime ) s - %( levelname ) s - %( message ) s ’)
5
6 def process ( data ) :
7 logging . info ( " Starting processing " )
8 try :
9 result = data / 2
10 except Exception as e :
11 logging . error ( f " Error : { e } " )
12 else :
13 logging . debug ( f " Result : { result } " ) # Won ’t appear ( level = INFO )
14 return result
15
16 process (10)

Logging to a File with Custom Format

1 import logging
2
3 logging . basicConfig (
4 filename = ’ app . log ’ ,
5 level = logging . DEBUG ,
6 format = ’ %( asctime ) s - %( name ) s - %( levelname ) s - %( message ) s ’
7 )
8
9 def read_data ( filepath ) :
10 logging . info ( f " Attempting to read { filepath } " )
11 try :
12 with open ( filepath ) as f :
13 data = f . read ()
14 logging . debug ( f " Read { len ( data ) } characters " )
15 return data
16 except FileNotFoundError :
17 logging . error ( f " File { filepath } not found " )

48
 49

Using Multiple Handlers (Console + File)

1 import logging
2
3 # Create logger
4 logger = logging . getLogger ( ’ my_app ’)
5 logger . setLevel ( logging . DEBUG )
6
7 # Console handler ( only warnings and above )
8 c_handler = logging . StreamHandler ()
9 c_handler . setLevel ( logging . WARNING )
10 c_format = logging . Formatter ( ’ %( name ) s - %( levelname ) s - %( message ) s ’)
11 c_handler . setFormatter ( c_format )
12
13 # File handler ( all levels )
14 f_handler = logging . FileHandler ( ’ app . log ’)
15 f_handler . setLevel ( logging . DEBUG )
16 f_format = logging . Formatter ( ’ %( asctime ) s - %( name ) s - %( levelname ) s - %( message
) s ’)
17 f_handler . setFormatter ( f_format )
18
19 # Add handlers
20 logger . addHandler ( c_handler )
21 logger . addHandler ( f_handler )
22
23 # Test
24 logger . debug ( ’ Debug message ( file only ) ’)
25 logger . warning ( ’ Warning message ( both ) ’)

Logging in a Data Processing Context

1 import logging
2 import pandas as pd
3
4 logging . basicConfig (
5 level = logging . INFO ,
6 filename = ’ pipeline . log ’ ,
7 format = ’ %( asctime ) s - %( message ) s ’
8 )
9
10 def load_and_clean ( csv_file ) :
11 logging . info ( f " Loading { csv_file } " )
12 df = pd . read_csv ( csv_file )
13 initial = len ( df )
14 df . dropna ( inplace = True )
15 dropped = initial - len ( df )
16 logging . info ( f " Dropped { dropped } rows with missing values " )
17 return df

Summary for Quick Revision

Exception Handling
Key Elements:

• try, except, else, and finally blocks

• raise keyword for manually triggering exceptions

• Built-in exceptions such as ZeroDivisionError, ValueError, TypeError, and FileNotFoundError

49
50

Common Pitfalls:
• Catching exceptions too broadly (e.g., using a generic except)
• Forgetting to use finally for cleanup operations such as closing files

Debugging
Key Elements:
• print() statements for tracing program execution
• assert statements for validating assumptions
• pdb debugger and breakpoint() for interactive debugging
• IDE debugging tools such as breakpoints, step execution, and variable inspection
Common Pitfalls:
• Excessive reliance on print() debugging
• Leaving debugging statements in production code

Logging
Key Elements:
• Python logging module
• Logging levels: DEBUG, INFO, WARNING, ERROR, CRITICAL
• Handlers for directing logs to files or console
• Formatters for customizing log message structure
• Proper configuration using [Link]()
Common Pitfalls:
• Using print() instead of logging in real applications
• Not setting appropriate logging levels
• Misconfiguring handlers or logging formats

Working with Modules and Packages in Python

Modules and packages are fundamental for organizing and reusing code in Python. They allow
developers to divide large programs into smaller, manageable components that can be reused
across different projects.

1.1 What is a Module?

A module is a single .py file that contains Python definitions and statements. It can define
functions, classes, variables, and may also include executable code.
Modules provide the following advantages:
• Help break large programs into smaller, manageable components.
• Promote code reuse across multiple programs.
• Improve maintainability and readability of software projects.

50
 51

1.2 What is a Package?

A package is a collection of modules organized in directories. A directory must contain a special
file named init .py to be recognized as a package.

• The init .py file can be empty or contain initialization code.

• Packages can include subpackages (nested directories).

• Packages help organize large codebases into logical groups.

1.3 Importing Modules

Python provides multiple ways to import modules.

• import module name – imports the whole module. Access items using module [Link].

• from module name import item1, item2 – imports specific items into the current names-
pace.

• from module name import * – imports all public items (defined by all ). This ap-
proach should generally be avoided in production code.

• import module name as alias – imports a module with an alias.

• from package import module – imports a module from a package.

• from [Link] import item – imports a specific item from a module inside a
package.

1.4 Module Search Path ([Link])

When Python imports a module, it searches in the following locations:

• The directory containing the script or the current working directory.

• Directories listed in the PYTHONPATH environment variable.

• Installation-dependent default paths such as the standard library and site-packages.

Although it is possible to modify [Link] at runtime, this practice is generally discouraged.

1.5 name and main

Every Python module contains a special variable named name .

• When a module is executed directly using python [Link], name is set to " main ".

• When the module is imported, name is set to the module’s actual name.

A common pattern is:

1 if __name__ == " __main__ " :
2 # Code that runs only when the file is executed directly

51
 52

1.6 init .py and Package Initialization

The file init .py is executed when a package is imported.

• It may be empty or contain initialization logic.

• It can define the variable all to control exported names.

• It may import submodules to make them accessible directly from the package.

1.7 all

all is a list of strings that defines which names are exported when using:
1 from module import *

If all is not defined, Python imports all names that do not start with an underscore.

1.8 Relative Imports

Within a package, relative imports can be used.

• from . import module – import from the same package.

• from .. import module – import from the parent package.

• from .subpackage import module

Relative imports work only inside packages and not in top-level scripts.

2. Example Programs

2.1 Creating and Using a Simple Module

File: [Link]
1 # mymath . py
2 def add (a , b ) :
3 return a + b
4
5 def subtract (a , b ) :
6 return a - b
7
8 PI = 3.14159

Using the module

1 import mymath
2
3 print ( mymath . add (5 , 3) )
4 print ( mymath . PI )
5
6 from mymath import subtract
7 print ( subtract (10 , 4) )

52
 53

2.2 Creating a Package

Directory Structure
1 mycalculator /
2 __init__ . py
3 basic . py
4 advanced /
5 __init__ . py
6 stats . py
7 trig . py

File: mycalculator/ init .py

1 from . basic import add , subtract

File: mycalculator/[Link]
1 def add (a , b ) :
2 return a + b
3
4 def subtract (a , b ) :
5 return a - b

File: mycalculator/advanced/[Link]
1 def mean ( data ) :
2 return sum ( data ) / len ( data )

Using the package

1 import mycalculator
2 from mycalculator . basic import add
3 from mycalculator . advanced . stats import mean
4
5 print ( mycalculator . add (2 , 3) )
6 print ( add (10 , 5) )
7 print ( mean ([1 , 2 , 3 , 4]) )

2.3 Relative Imports Inside a Package

1 # mycalculator / advanced / trig . py

2 from . stats import mean
3
4 def some_function () :
5 pass

2.4 Using all to Control Import

1 __all__ = [ ’ add ’ , ’ PI ’]
2
3 def add (a , b ) :
4 return a + b
5
6 def subtract (a , b ) :
7 return a - b
8
9 PI = 3.14159
10 SECRET = " hidden "

53
 54

2.5 Running a Module as a Script

1 def say_hello ( name ) :

2 return f " Hello , { name }! "
3
4 if __name__ == " __main__ " :
5 print ( say_hello ( " Exam Student " ) )

3. Summary for Quick Revision

Modules
Key Elements

• .py files

• import, from ... import, as

• name and if name == " main "

Common Pitfalls

• Circular imports between modules

• Modifying [Link] unnecessarily

• Forgetting the name guard causing code to run during import

Packages
Key Elements

• Directory containing init .py

• Subpackages

• all

• Relative imports using . and ..

Common Pitfalls

• Missing init .py (in older Python versions)

• Incorrect relative import syntax

• Not exposing submodules in init .py when needed

Import System
Key Elements

• [Link]

• Search order

• PYTHONPATH

• Import caching using compiled .pyc files

54
 55

Common Pitfalls

• Naming conflicts with standard library modules

• Importing from unexpected paths

• Relying on the current working directory

55
Chapter-2
Data Sources and APIs

Data sources refer to the origins, locations, or systems from which data is obtained for pro-
cessing, analysis, and interpretation. They encompass a wide spectrum of storage mechanisms,
formats, and delivery methods, each with distinct characteristics that influence how data must
be accessed, extracted, and prepared for analysis.

Classification of Data Sources

Based on reference literature, data sources can be classified along several dimensions.

By Structure
• Structured Data Sources: Data with a predefined schema, typically organized in tables
with rows and columns. Examples include relational databases and CSV files.
• Semi-structured Data Sources: Data with a flexible structure that uses tags, markers,
or key-value pairs to organize information. Examples include JSON, XML, and HTML.
• Unstructured Data Sources: Data without a predefined structure. Examples include
text files, images, audio, and video.

By Access Method
• File-based Sources: Data stored in files that can be accessed locally or over a network,
such as CSV, Excel, or text files.
• Database Sources: Data stored in databases and accessed using query languages such
as SQL or through NoSQL systems.
• API Sources: Data obtained through web service endpoints using protocols such as
REST or SOAP.
• Stream Sources: Continuous data flows generated in real time, such as sensor data or
system logs.

By Storage Location
• Local Sources: Data stored on a local machine or device.
• Network Sources: Data accessed through network resources such as shared servers or
distributed systems.
• Cloud Sources: Data stored and managed through cloud-based services such as Amazon
S3, Google Cloud Storage, or Microsoft Azure.

1
2

By Data Origin
• Primary Sources: Data collected firsthand for a specific purpose, such as surveys,
experiments, or sensor measurements.

• Secondary Sources: Data obtained from existing repositories or previously collected

datasets, such as public datasets or institutional databases.

• Third-party Sources: Data provided by external organizations, such as commercial

data providers or public APIs.

Reading and Writing Structured and Unstructured Data

CSV Files (Comma-Separated Values)

Definition and Characteristics
CSV files represent tabular data where each line corresponds to a row, and fields within each
row are separated by commas. This format originated in early computing as a simple way to
exchange data between different systems and applications.

Historical Context
CSV emerged as a portable format when transferring data between incompatible database
systems. Its simplicity became its greatest strength—any system that can read and write text
can process CSV files.

Structural Components
• Records: Each line represents one record or row of data.

• Fields: Individual data elements separated by delimiters.

• Header Row (optional): First row containing column names.

• Footer Row (optional): Last row containing summaries or metadata.

Important Theoretical Considerations

Delimiters
Although commas are standard delimiters, other separators may be used:

• Tabs (TSV – Tab-Separated Values)

• Pipes (|)

• Semicolons (;)

• Spaces in fixed-width formatted exports

Headers
The first row often contains column names, although this is not mandatory. When headers are
absent, columns are referenced using index positions (0, 1, 2, ...).

2
 3

Quoting Mechanisms

• QUOTE ALL

• QUOTE MINIMAL

• QUOTE NONNUMERIC

• QUOTE NONE

Best Practices for Working with CSV

• Inspect the first few rows before loading the full dataset.

• Check delimiter consistency.

• Specify encoding such as UTF-8.

• Handle missing values explicitly.

• Use proper error handling.

JSON (JavaScript Object Notation)

Theoretical Background
JSON is a widely used data interchange format, especially for APIs and web services. Although
derived from JavaScript syntax, it is now language-independent.

Structural Components
Objects

Objects contain key-value pairs enclosed in curly braces.

{"name": "Alice", "age": 30}

Arrays

Arrays are ordered collections enclosed in square brackets.

["apple", "banana", 42, true]

Advantages of JSON
• Human-readable format

• Lightweight structure

• Flexible schema

• Language independent

3
4

Excel Files
File Format Evolution
• XLS – legacy binary format

• XLSX – XML-based modern format

• XLSM – macro-enabled format

Unique Challenges
• Multiple sheets

• Mixed data types

• Hidden rows and columns

• Formatting interference

Text Files
Conceptual Framework
Text files represent sequences of characters encoded in a specific encoding format.

Classification of Text Files

Structured Text
• CSV

• TSV

• Fixed-width files

• XML or HTML

Semi-structured Text
• Log files

• Configuration files

• JSON

Unstructured Text
• Emails

• Articles

• Social media posts

• Transcripts

4
 5

Common Text File Formats

• LOG

• INI/CFG

• Markdown

• reStructuredText

• LaTeX

1.1 Reading CSV Files using CSV module

A. Reading as Lists ([Link])

1 import csv
2
3 # Basic CSV reading - returns rows as lists
4 with open ( ’ employees . csv ’ , ’r ’ , encoding = ’utf -8 ’) as file :
5 csv_reader = csv . reader ( file )
6
7 # Get the header row ( first row )
8 header = next ( csv_reader )
9 print ( f " Header : { header } " )
10
11 # Read and process data rows
12 print ( " \ nEmployee Data : " )
13 for row_num , row in enumerate ( csv_reader , start =2) :
14 print ( f " Row { row_num }: { row } " )
15 # Access by index : row [0] = name , row [1] = age , etc .

B. Reading as Dictionaries ([Link])

1 import csv
2
3 # Reading CSV with DictReader - rows as dictionaries
4 with open ( ’ employees . csv ’ , ’r ’ , encoding = ’utf -8 ’) as file :
5 csv_reader = csv . DictReader ( file )
6
7 # Access field names
8 print ( f " Columns : { csv_reader . fieldnames } " )
9
10 # Read and process data rows
11 for row in csv_reader :
12 print ( f " Name : { row [ ’ name ’]} , Age : { row [ ’ age ’]} , "
13 f " Department : { row [ ’ department ’]} " )

1.2 Writing CSV Files

A. Writing with [Link]

1 import csv
2
3 # Data to write : header row + data rows
4 data = [
5 [ ’ name ’ , ’ age ’ , ’ department ’ , ’ salary ’] , # Header

5
 6

6 [ ’ Alice Brown ’ , 32 , ’ Engineering ’ , 82000] ,

7 [ ’ Charlie Wilson ’ , 41 , ’ Sales ’ , 91000] ,
8 [ ’ Diana Prince ’ , 29 , ’ Marketing ’ , 67000]
9 ]
10
11 # Write to CSV file
12 with open ( ’ output . csv ’ , ’w ’ , newline = ’ ’ , encoding = ’utf -8 ’) as file :
13 writer = csv . writer ( file )
14
15 # Write header
16 writer . writerow ( data [0])
17
18 # Write all data rows at once
19 writer . writerows ( data [1:])
20
21 print ( " File written successfully " )

B. Writing with [Link]

1 import csv
2
3 # Data as list of dictionaries
4 employees = [
5 { ’ name ’: ’ Eve Adams ’ , ’ age ’: 27 , ’ department ’: ’ Engineering ’ , ’ salary ’:
72000} ,
6 { ’ name ’: ’ Frank Miller ’ , ’ age ’: 38 , ’ department ’: ’ Sales ’ , ’ salary ’:
85000} ,
7 { ’ name ’: ’ Grace Lee ’ , ’ age ’: 31 , ’ department ’: ’ Marketing ’ , ’ salary ’:
68000}
8 ]
9
10 # Write using DictWriter
11 with open ( ’ employees_dict . csv ’ , ’w ’ , newline = ’ ’ , encoding = ’utf -8 ’) as file :
12 fieldnames = [ ’ name ’ , ’ age ’ , ’ department ’ , ’ salary ’]
13 writer = csv . DictWriter ( file , fieldnames = fieldnames )
14
15 # Write header
16 writer . writeheader ()
17
18 # Write all rows
19 writer . writerows ( employees )

Using pandas

2.1 Reading CSV Files with pandas

A. Basic Reading

1 import pandas as pd
2
3 # Simple CSV read
4 df = pd . read_csv ( ’ employees . csv ’)
5
6 print ( " DataFrame from CSV : " )
7 print ( df . head () ) # First 5 rows
8
9 print ( f " \ nShape : { df . shape } " )
10 print ( f " \ nData Types :\ n { df . dtypes } " )

6
 7

B. Reading with Common Parameters

1 import pandas as pd
2
3 # Reading with various options
4 df = pd . read_csv ( ’ employees . csv ’ ,
5 index_col =0 ,
6 header =0 ,
7 names =[ ’ name ’ , ’ age ’ , ’ dept ’ , ’ salary ’] ,
8 skiprows =[1 ,2] ,
9 nrows =100 ,
10 usecols =[ ’ name ’ , ’ salary ’] ,
11 dtype ={ ’ age ’: int , ’ salary ’: float } ,
12 parse_dates =[ ’ hire_date ’] ,
13 na_values =[ ’ NA ’ , ’ Missing ’] ,
14 encoding = ’utf -8 ’)
15
16 print ( df . head () )

C. Handling Missing Values

1 import pandas as pd
2 import numpy as np
3
4 df = pd . read_csv ( ’ data_with_missing . csv ’ ,
5 na_values =[ ’ ’ , ’ NA ’ , ’ NULL ’ , ’ -999 ’] ,
6 keep_default_na = True )
7
8 print ( " Missing values per column : " )
9 print ( df . isnull () . sum () )

2.2 Writing CSV Files with pandas

A. Basic Writing

1 import pandas as pd
2
3 data = {
4 ’ name ’: [ ’ Alice ’ , ’ Bob ’ , ’ Charlie ’ , ’ Diana ’] ,
5 ’ age ’: [25 , 30 , 35 , 28] ,
6 ’ department ’: [ ’ HR ’ , ’ IT ’ , ’ IT ’ , ’ HR ’] ,
7 ’ salary ’: [50000 , 60000 , 65000 , 52000]
8 }
9
10 df = pd . DataFrame ( data )
11
12 df . to_csv ( ’ output_pandas . csv ’ ,
13 index = False ,
14 encoding = ’utf -8 ’)

B. Writing with Common Parameters

1 df . to_csv ( ’ output_formatted . csv ’ ,

2 index = False ,
3 header = True ,
4 columns =[ ’ name ’ , ’ salary ’] ,
5 sep = ’; ’ ,
6 na_rep = ’ NULL ’ ,

7
 8

7 float_format = ’ %.2 f ’ ,
8 encoding = ’utf -8 - sig ’ ,
9 date_format = ’%Y -% m -% d ’)

Operation csv Module pandas

Read [Link]() [Link] csv()
[Link]()
Write [Link]() [Link] csv()
[Link]()
Parameters delimiter, quotechar sep, encoding, dtype
Output Control writerow(), writerows() index, header, columns

Table 1: Comparison between Python csv module and pandas for CSV operations

Database Access with Relational and Non-Relational Databases

Introduction to Database Access
Database access in Python follows a standard convention where data is represented as sequences
of tuples. All database interactions occur through SQL queries, with each input or output row
represented as a tuple.
Key Components:

• Connection Objects: Represent the communication channel between Python and the
database, handling authentication, network communication, and session management.

• Cursor Objects: Execute SQL statements and fetch results, providing query execution
and result set iteration.

• Parameterized Queries: Using placeholders (?, %s, or named parameters) instead of

string formatting to prevent SQL injection attacks.

• Transactions: Groups of operations that execute as a single unit with commit or rollback
capabilities.

Relational Databases
Definition:
A relational database is a collection of tables containing data organized by relations between
data items. Relationships can be established between rows in different tables or between columns
inside a table.
Core Concepts:

Concept Description
Table Collection of related data organized in rows and columns
Row (Record) Single entry in a table representing one instance
Column (Field) Specific attribute of the data
Primary Key Uniquely identifies each row in a table
Foreign Key Establishes relationships between tables
Index Improves query performance
View Virtual table based on SELECT query
Schema Defines table structure and relationships

8
 9

ACID Properties:

• Atomicity: Transactions are all-or-nothing; if any part fails, entire transaction rolls back.

• Consistency: Transactions maintain database rules and constraints.

• Isolation: Concurrent transactions do not interfere with each other.

• Durability: Committed transactions persist even after system failures.

SQL Categories:

Category Purpose Examples

DDL Define structure CREATE, ALTER, DROP
DML Manipulate data SELECT, INSERT, UPDATE, DELETE
DCL Control access GRANT, REVOKE
TCL Manage transactions COMMIT, ROLLBACK

Common Relational Databases:

• SQLite (embedded, serverless)

• MySQL / MariaDB

• PostgreSQL

• Oracle

• Microsoft SQL Server

Non-Relational Databases (NoSQL)

Definition:
Not Only SQL (NoSQL) databases are designed for big data and web applications. They
allow data storage without fixed schemas and are optimized for specific data models and access
patterns.
Types of NoSQL Databases:

• Document Stores

– Description: Data stored as JSON-like documents

– Examples: MongoDB, CouchDB
– Use Cases: Content management, catalogs

• Key-Value Stores

– Description: Simple key-value pairs

– Examples: Redis, DynamoDB
– Use Cases: Caching, session storage

• Column-Family Stores

– Description: Data stored in columns rather than rows

– Examples: Cassandra, HBase
– Use Cases: Time-series data, analytics

9
 10

• Graph Databases

– Description: Data stored as nodes and edges representing relationships

– Examples: Neo4j, ArangoDB
– Use Cases: Social networks, recommendation systems

Key Characteristics:

• Schema-less Design: Documents in the same collection can have different fields.

• Horizontal Scaling: Designed to distribute across many servers.

• BASE Properties: Basically Available, Soft state, Eventually consistent.

• Denormalization: Data duplication accepted for query performance.

• Embedded Documents: Related data can be nested rather than split across tables.

BASE vs ACID:

ACID BASE
Strong consistency Weak consistency (eventual)
Isolation Availability first
Focus on commit Focus on continue
Pessimistic Optimistic
Complex Simple

RELATIONAL DATABASE PROGRAMS

SQLite Programs
A. Basic SQLite Operations

1 import sqlite3
2
3 # Connect to database
4 conn = sqlite3 . connect ( ’ company . db ’)
5 print ( " Connected to database successfully " )
6
7 # Create a cursor
8 cursor = conn . cursor ()
9
10 # Create tables
11 cursor . execute ( ’ ’ ’
12 CREATE TABLE IF NOT EXISTS departments (
13 dept_id INTEGER PRIMARY KEY ,
14 dept_name TEXT NOT NULL ,
15 location TEXT
16 )
17 ’ ’ ’)
18
19 cursor . execute ( ’ ’ ’
20 CREATE TABLE IF NOT EXISTS employees (
21 emp_id INTEGER PRIMARY KEY ,
22 name TEXT NOT NULL ,
23 age INTEGER ,
24 salary REAL ,
25 dept_id INTEGER ,

10
 11

26 hire_date TEXT ,
27 FOREIGN KEY ( dept_id ) REFERENCES departments ( dept_id )
28 )
29 ’ ’ ’)
30 conn . commit ()
31 print ( " Tables created successfully " )
32
33 # Insert data
34 departments = [
35 (1 , ’ Engineering ’ , ’ Building A ’) ,
36 (2 , ’ Marketing ’ , ’ Building B ’) ,
37 (3 , ’ Sales ’ , ’ Building C ’)
38 ]
39 cursor . executemany ( ’ INSERT INTO departments VALUES (? , ? , ?) ’ , departments )
40
41 employees = [
42 (101 , ’ Alice Brown ’ , 32 , 82000 , 1 , ’ 2022 -03 -15 ’) ,
43 (102 , ’ Bob Smith ’ , 45 , 95000 , 1 , ’ 2021 -06 -01 ’) ,
44 (103 , ’ Charlie Wilson ’ , 38 , 72000 , 2 , ’ 2023 -01 -10 ’) ,
45 (104 , ’ Diana Prince ’ , 29 , 68000 , 3 , ’ 2023 -08 -22 ’)
46 ]
47 cursor . executemany ( ’ INSERT INTO employees VALUES (? , ? , ? , ? , ? , ?) ’ , employees
)
48 conn . commit ()
49 print ( f " Inserted { cursor . rowcount } rows " )

B. Querying Data

1 import sqlite3
2
3 conn = sqlite3 . connect ( ’ company . db ’)
4 cursor = conn . cursor ()
5
6 # SELECT all employees
7 cursor . execute ( ’ SELECT * FROM employees ’)
8 all_employees = cursor . fetchall ()
9 print ( " All Employees : " )
10 for emp in all_employees :
11 print ( f " ID : { emp [0]} , Name : { emp [1]} , Salary : $ { emp [4]} " )
12
13 # SELECT with condition
14 cursor . execute ( ’ SELECT name , salary FROM employees WHERE dept_id = ? ’ , (1 ,) )
15 engineers = cursor . fetchall ()
16 print ( " \ nEngineering Department Employees : " )
17 for name , salary in engineers :
18 print ( f " { name }: $ { salary } " )
19
20 # SELECT with JOIN
21 cursor . execute ( ’ ’ ’
22 SELECT e . name , e . salary , d . dept_name , d . location
23 FROM employees e
24 JOIN departments d ON e . dept_id = d . dept_id
25 WHERE salary > ?
26 ’ ’ ’ , (70000 ,) )
27 high_earners = cursor . fetchall ()
28 print ( " \ nEmployees earning > $70 ,000: " )
29 for name , salary , dept , loc in high_earners :
30 print ( f " { name } ({ dept }) : $ { salary } " )
31
32 # Aggregation queries
33 cursor . execute ( ’ ’ ’
34 SELECT d . dept_name , COUNT (*) as emp_count , AVG ( e . salary ) as avg_salary

11
 12

35 FROM employees e
36 JOIN departments d ON e . dept_id = d . dept_id
37 GROUP BY d . dept_name
38 ’ ’ ’)
39 stats = cursor . fetchall ()
40 print ( " \ nDepartment Statistics : " )
41 for dept , count , avg in stats :
42 print ( f " { dept }: { count } employees , Avg Salary : $ { avg :.2 f } " )

C. Update and Delete Operations

1 import sqlite3
2
3 conn = sqlite3 . connect ( ’ company . db ’)
4 cursor = conn . cursor ()
5
6 # UPDATE operation
7 cursor . execute ( ’ ’ ’
8 UPDATE employees
9 SET salary = salary * 1.10
10 WHERE dept_id = ?
11 ’ ’ ’ , (2 ,) )
12 conn . commit ()
13 print ( f " Updated { cursor . rowcount } rows in Marketing department " )
14
15 # DELETE operation
16 cursor . execute ( ’ DELETE FROM employees WHERE name = ? ’ , ( ’ Bob Smith ’ ,) )
17 conn . commit ()
18 print ( f " Deleted { cursor . rowcount } rows " )
19
20 # Parameterized queries
21 def u p d a t e _em pl oy ee _sa la ry ( emp_id , new_salary ) :
22 cursor . execute ( ’ ’ ’
23 UPDATE employees
24 SET salary = ?
25 WHERE emp_id = ?
26 ’ ’ ’ , ( new_salary , emp_id ) )
27 conn . commit ()
28 return cursor . rowcount
29
30 u p d a t e _ e m pl oy ee _sa la ry (101 , 85000)
31 print ( " Employee salary updated " )
32
33 conn . close ()

D. Transaction Management

1 import sqlite3
2
3 def trans fer_employee ( emp_id , new_dept_id ) :
4 conn = sqlite3 . connect ( ’ company . db ’)
5 cursor = conn . cursor ()
6
7 try :
8 # Check if employee exists
9 cursor . execute ( ’ SELECT name FROM employees WHERE emp_id = ? ’ , ( emp_id ,)
)
10 if not cursor . fetchone () :
11 raise Exception ( " Employee not found " )
12

12
 13

13 # Check if department exists

14 cursor . execute ( ’ SELECT dept_name FROM departments WHERE dept_id = ? ’ , (
new_dept_id ,) )
15 if not cursor . fetchone () :
16 raise Exception ( " Department not found " )
17
18 # Update employee department
19 cursor . execute ( ’ ’ ’
20 UPDATE employees
21 SET dept_id = ?
22 WHERE emp_id = ?
23 ’ ’ ’ , ( new_dept_id , emp_id ) )
24
25 conn . commit ()
26 print ( f " Employee { emp_id } transferred successfully " )
27
28 except Exception as e :
29 conn . rollback ()
30 print ( f " Transaction failed : { e } " )
31
32 finally :
33 conn . close ()
34
35 # Test the transaction
36 trans fer_employee (101 , 3)

E. Using pandas with SQLite

1 import sqlite3
2 import pandas as pd
3
4 conn = sqlite3 . connect ( ’ company . db ’)
5
6 # Read SQL query into DataFrame
7 df_employees = pd . read_sql_query ( " SELECT * FROM employees " , conn )
8 print ( " Employees DataFrame : " )
9 print ( df_employees . head () )
10
11 # Read with conditions
12 df_high_salary = pd . read_sql_query ( ’ ’ ’
13 SELECT e . name , e . salary , d . dept_name
14 FROM employees e
15 JOIN departments d ON e . dept_id = d . dept_id
16 WHERE salary > 70000
17 ’ ’ ’ , conn )
18 print ( " \ nHigh Salary Employees : " )
19 print ( df_high_salary )
20
21 # Write DataFrame to SQL table
22 new_employees = pd . DataFrame ({
23 ’ name ’: [ ’ Eve Adams ’ , ’ Frank Miller ’] ,
24 ’ age ’: [27 , 41] ,
25 ’ salary ’: [72000 , 89000] ,
26 ’ dept_id ’: [1 , 3] ,
27 ’ hire_date ’: [ ’ 2024 -01 -15 ’ , ’ 2024 -02 -01 ’]
28 })
29 new_employees . to_sql ( ’ employees ’ , conn , if_exists = ’ append ’ , index = False )
30 print ( " \ nNew employees added to database " )
31
32 conn . close ()

13
 14

MySQL/PostgreSQL Programs
A. MySQL Connection and Operations

1 import mysql . connector

2 from mysql . connector import Error
3
4 try :
5 # Establish connection
6 conn = mysql . connector . connect (
7 host = ’ localhost ’ ,
8 database = ’ company_db ’ ,
9 user = ’ root ’ ,
10 password = ’ password ’ ,
11 port =3306
12 )
13
14 if conn . is_connected () :
15 print ( " Connected to MySQL database " )
16 cursor = conn . cursor ()
17
18 # Create table
19 cursor . execute ( ’ ’ ’
20 CREATE TABLE IF NOT EXISTS products (
21 product_id INT AUTO_INCREMENT PRIMARY KEY ,
22 product_name VARCHAR (100) NOT NULL ,
23 price DECIMAL (10 ,2) ,
24 quantity INT ,
25 category VARCHAR (50)
26 )
27 ’ ’ ’)
28
29 # Insert data
30 insert_query = ’ ’ ’
31 INSERT INTO products ( product_name , price , quantity , category )
32 VALUES (% s , %s , %s , % s )
33 ’’’
34 products = [
35 ( ’ Laptop ’ , 999.99 , 10 , ’ Electronics ’) ,
36 ( ’ Mouse ’ , 24.99 , 50 , ’ Electronics ’) ,
37 ( ’ Desk Chair ’ , 199.99 , 15 , ’ Furniture ’)
38 ]
39 cursor . executemany ( insert_query , products )
40 conn . commit ()
41 print ( f " Inserted { cursor . rowcount } rows " )
42
43 # Query with conditions
44 cursor . execute ( ’ ’ ’
45 SELECT product_name , price , quantity
46 FROM products
47 WHERE category = % s AND price > % s
48 ’ ’ ’ , ( ’ Electronics ’ , 500) )
49
50 results = cursor . fetchall ()
51 for row in results :
52 print ( f " Product : { row [0]} , Price : $ { row [1]} " )
53
54 except Error as e :
55 print ( f " Error : { e } " )
56 finally :
57 if conn . is_connected () :
58 cursor . close ()
59 conn . close ()

14
 15

60 print ( " MySQL connection closed " )

B. PostgreSQL Connection

1 import psycopg2
2 from psycopg2 import sql
3
4 # Connect to PostgreSQL
5 conn = psycopg2 . connect (
6 host = " localhost " ,
7 database = " company_db " ,
8 user = " postgres " ,
9 password = " password " ,
10 port = " 5432 "
11 )
12
13 cursor = conn . cursor ()
14
15 # Create table with SERIAL for auto - increment
16 cursor . execute ( ’ ’ ’
17 CREATE TABLE IF NOT EXISTS customers (
18 customer_id SERIAL PRIMARY KEY ,
19 name VARCHAR (100) NOT NULL ,
20 email VARCHAR (100) UNIQUE ,
21 joined_date DATE DEFAULT CURRENT_DATE
22 )
23 ’ ’ ’)
24
25 # Insert using parameterized queries
26 cursor . execute ( ’ ’ ’
27 INSERT INTO customers ( name , email )
28 VALUES (% s , % s ) RETURNING customer_id
29 ’ ’ ’ , ( ’ John Doe ’ , ’ john@email . com ’) )
30 customer_id = cursor . fetchone () [0]
31 conn . commit ()
32 print ( f " Inserted customer with ID : { customer_id } " )
33
34 cursor . close ()
35 conn . close ()

MongoDB (Document Database)

A. Basic MongoDB Operations

1 from pymongo import MongoClient

2 from pymongo . errors import ConnectionFailure
3
4 # Connect to MongoDB
5 try :
6 client = MongoClient ( ’ mongodb :// localhost :27017/ ’ , se r v e rS e l e ct i o n Ti m e o ut M S
=5000)
7 client . admin . command ( ’ ping ’) # Test connection
8 print ( " Connected to MongoDB successfully " )
9
10 db = client [ ’ company_db ’]
11 e m p l o yees_collection = db [ ’ employees ’]
12
13 # Insert single document
14 employee = {
15 ’ emp_id ’: 101 ,

15
 16

16 ’ name ’: ’ Alice Brown ’ ,

17 ’ age ’: 32 ,
18 ’ skills ’: [ ’ Python ’ , ’ SQL ’ , ’ MongoDB ’] ,
19 ’ address ’: { ’ street ’: ’ 123 Main St ’ , ’ city ’: ’ New York ’ , ’ zip ’: ’ 10001 ’
},
20 ’ department ’: ’ Engineering ’ ,
21 ’ salary ’: 82000 ,
22 ’ is_active ’: True ,
23 ’ join_date ’: ’ 2022 -03 -15 ’
24 }
25
26 result = employees_collection . insert_one ( employee )
27 print ( f " Inserted document with ID : { result . inserted_id } " )
28
29 # Insert multiple documents
30 new_employees = [
31 { ’ emp_id ’: 102 , ’ name ’: ’ Bob Smith ’ , ’ age ’: 45 , ’ skills ’: [ ’ Java ’ , ’ AWS
’] ,
32 ’ department ’: ’ Engineering ’ , ’ salary ’: 95000} ,
33 { ’ emp_id ’: 103 , ’ name ’: ’ Charlie Wilson ’ , ’ age ’: 38 , ’ skills ’: [ ’
Marketing ’ , ’ SEO ’] ,
34 ’ department ’: ’ Marketing ’ , ’ salary ’: 72000}
35 ]
36 result = employees_collection . insert_many ( new_employees )
37 print ( f " Inserted { len ( result . inserted_ids ) } documents " )
38
39 except ConnectionFailure as e :
40 print ( f " Could not connect to MongoDB : { e } " )

B. Querying MongoDB

1 from pymongo import MongoClient

2
3 client = MongoClient ( ’ mongodb :// localhost :27017/ ’)
4 db = client [ ’ company_db ’]
5 employees = db [ ’ employees ’]
6
7 # Find one document
8 alice = employees . find_one ({ ’ name ’: ’ Alice Brown ’ })
9 print ( " Found employee : " )
10 print ( alice )
11
12 # Find all documents
13 all_employees = employees . find ()
14 print ( " \ nAll employees : " )
15 for emp in all_employees :
16 print ( f " { emp [ ’ name ’]} - { emp [ ’ department ’]} " )
17
18 # Find with conditions
19 high_salary = employees . find ({ ’ salary ’: { ’ $gt ’: 80000}})
20 print ( " \ nEmployees with salary > $80 ,000: " )
21 for emp in high_salary :
22 print ( f " { emp [ ’ name ’]}: $ { emp [ ’ salary ’]} " )
23
24 # Multiple conditions
25 engineers_young = employees . find ({
26 ’ $and ’: [{ ’ department ’: ’ Engineering ’} , { ’ age ’: { ’ $lt ’: 40}}]
27 })
28 print ( " \ nEngineers under 40: " )
29 for emp in engineers_young :
30 print ( f " { emp [ ’ name ’]} - Age : { emp [ ’ age ’]} " )
31

16
 17

32 # Projection ( specific fields )

33 names_only = employees . find ({} , { ’ name ’: 1 , ’ department ’: 1 , ’ _id ’: 0})
34 print ( " \ nNames and departments only : " )
35 for emp in names_only :
36 print ( f " { emp [ ’ name ’]} works in { emp [ ’ department ’]} " )
37
38 # Count documents
39 count = employees . count_documents ({ ’ department ’: ’ Engineering ’ })
40 print ( f " \ nNumber of engineers : { count } " )
41
42 # Sort results
43 sorted_emps = employees . find () . sort ( ’ salary ’ , -1) . limit (3)
44 print ( " \ nTop 3 highest paid : " )
45 for emp in sorted_emps :
46 print ( f " { emp [ ’ name ’]}: $ { emp [ ’ salary ’]} " )

C. Update and Delete in MongoDB

1 from pymongo import MongoClient

2
3 client = MongoClient ( ’ mongodb :// localhost :27017/ ’)
4 db = client [ ’ company_db ’]
5 employees = db [ ’ employees ’]
6
7 # Update one document
8 result = employees . update_one (
9 { ’ name ’: ’ Alice Brown ’} ,
10 { ’ $set ’: { ’ salary ’: 90000 , ’ skills ’: [ ’ Python ’ , ’ SQL ’ , ’ MongoDB ’ , ’ Data
Science ’ ]}}
11 )
12 print ( f " Modified { result . modified_count } document " )
13
14 # Update multiple documents
15 result = employees . update_many (
16 { ’ department ’: ’ Engineering ’} ,
17 { ’ $inc ’: { ’ salary ’: 5000}}
18 )
19 print ( f " Modified { result . modified_count } documents " )
20
21 # Add new field to all documents
22 employees . update_many ({} , { ’ $set ’: { ’ active_status ’: ’ Active ’ }})
23
24 # Delete one document
25 result = employees . delete_one ({ ’ name ’: ’ Bob Smith ’ })
26 print ( f " Deleted { result . deleted_count } document " )
27
28 # Delete many documents
29 result = employees . delete_many ({ ’ department ’: ’ Marketing ’ })
30 print ( f " Deleted { result . deleted_count } documents " )
31
32 # Drop collection
33 employees . drop ()
34 print ( " Collection dropped " )

D. Advanced MongoDB Operations

1 from pymongo import MongoClient

2 from datetime import datetime
3
4 client = MongoClient ( ’ mongodb :// localhost :27017/ ’)

17
 18

5 db = client [ ’ company_db ’]
6 orders = db [ ’ orders ’]
7
8 # Insert orders with nested documents
9 orders_data = [
10 {
11 ’ order_id ’: 1001 ,
12 ’ customer ’: { ’ name ’: ’ John Smith ’ , ’ email ’: ’ john@email . com ’ ,
13 ’ address ’: { ’ city ’: ’ New York ’ , ’ state ’: ’ NY ’}} ,
14 ’ items ’: [{ ’ product ’: ’ Laptop ’ , ’ price ’: 999.99 , ’ quantity ’: 1} ,
15 { ’ product ’: ’ Mouse ’ , ’ price ’: 24.99 , ’ quantity ’: 2}] ,
16 ’ total ’: 1049.97 ,
17 ’ date ’: datetime . now ()
18 },
19 {
20 ’ order_id ’: 1002 ,
21 ’ customer ’: { ’ name ’: ’ Jane Doe ’ , ’ email ’: ’ jane@email . com ’ ,
22 ’ address ’: { ’ city ’: ’ Boston ’ , ’ state ’: ’ MA ’}} ,
23 ’ items ’: [{ ’ product ’: ’ Desk Chair ’ , ’ price ’: 199.99 , ’ quantity ’: 1}] ,
24 ’ total ’: 199.99 ,
25 ’ date ’: datetime . now ()
26 }
27 ]
28 orders . insert_many ( orders_data )
29
30 # Query nested documents
31 ny_orders = orders . find ({ ’ customer . address . city ’: ’ New York ’ })
32 print ( " Orders from New York : " )
33 for order in ny_orders :
34 print ( f " Order { order [ ’ order_id ’]} - Customer : { order [ ’ customer ’][ ’ name ’]} " )
35
36 # Query array elements
37 large_orders = orders . find ({ ’ items ’: { ’ $elemMatch ’: { ’ price ’: { ’ $gt ’: 500}}}})
38 print ( " \ nOrders with items > $500 : " )
39 for order in large_orders :
40 print ( f " Order { order [ ’ order_id ’]} - Total : $ { order [ ’ total ’]} " )
41
42 # Aggregation pipeline
43 pipeline = [
44 { ’ $unwind ’: ’ $items ’} ,
45 { ’ $group ’: {
46 ’ _id ’: ’ $items . product ’ ,
47 ’ total_quantity ’: { ’ $sum ’: ’ $items . quantity ’} ,
48 ’ total_revenue ’: { ’ $sum ’: { ’ $multiply ’: [ ’ $items . price ’ , ’ $items .
quantity ’ ]}}
49 }} ,
50 { ’ $sort ’: { ’ total_revenue ’: -1}}
51 ]
52
53 results = orders . aggregate ( pipeline )
54 print ( " \ nProduct sales summary : " )
55 for result in results :
56 print ( f " Product : { result [ ’ _id ’]} " )
57 print ( f " Quantity : { result [ ’ total_quantity ’]} " )
58 print ( f " Revenue : $ { result [ ’ total_revenue ’]:.2 f } " )
59
60 client . close ()

Redis (Key-Value Store)

1 import redis
2

18
 19

3 # Connect to Redis
4 r = redis . Redis ( host = ’ localhost ’ , port =6379 , db =0 , decode_responses = True )
5
6 # Test connection
7 try :
8 r . ping ()
9 print ( " Connected to Redis " )
10 except redis . ConnectionError :
11 print ( " Could not connect to Redis " )
12
13 # Set key - value pairs
14 r . set ( ’ user :1001: name ’ , ’ Alice Brown ’)
15 r . set ( ’ user :1001: email ’ , ’ alice@email . com ’)
16 r . set ( ’ user :1001: age ’ , 32)
17
18 # Get values
19 name = r . get ( ’ user :1001: name ’)
20 email = r . get ( ’ user :1001: email ’)
21 print ( f " User : { name } , Email : { email } " )
22
23 # Set multiple keys at once
24 r . mset ({
25 ’ user :1002: name ’: ’ Bob Smith ’ ,
26 ’ user :1002: email ’: ’ bob@email . com ’ ,
27 ’ user :1002: age ’: 45
28 })
29
30 # Check if key exists
31 if r . exists ( ’ user :1001: name ’) :
32 print ( " User 1001 exists " )
33
34 # Set with expiration (10 seconds )
35 r . setex ( ’ temp : session ’ , 10 , ’ session_data ’)
36
37 # Working with lists
38 r . lpush ( ’ recent_searches :1001 ’ , ’ laptop ’ , ’ mouse ’ , ’ keyboard ’)
39 searches = r . lrange ( ’ recent_searches :1001 ’ , 0 , -1)
40 print ( f " Recent searches : { searches } " )
41
42 # Working with hashes
43 r . hset ( ’ user :1003 ’ , mapping ={
44 ’ name ’: ’ Charlie Wilson ’ ,
45 ’ email ’: ’ charlie@email . com ’ ,
46 ’ age ’: 38 ,
47 ’ department ’: ’ Engineering ’
48 })
49
50 # Get all fields from hash
51 user_data = r . hgetall ( ’ user :1003 ’)
52 print ( f " User 1003 data : { user_data } " )
53
54 # Get specific fields
55 user_name = r . hget ( ’ user :1003 ’ , ’ name ’)
56 print ( f " Name : { user_name } " )
57
58 # Delete keys
59 r . delete ( ’ temp : session ’)
60 print ( " Deleted temporary key " )
61
62 # Get all keys matching pattern
63 keys = r . keys ( ’ user :* ’)
64 print ( f " All user keys : { keys } " )
65

19
 20

66 # Close connection
67 r . close ()

Cassandra (Column-Family Store)

1 from cassandra . cluster import Cluster

2 from cassandra . query import SimpleStatement
3 from datetime import datetime
4 from uuid import uuid4
5
6 # Connect to Cassandra
7 cluster = Cluster ([ ’ [Link] ’ ])
8 session = cluster . connect ()
9
10 # Create keyspace
11 session . execute ( " " "
12 CREATE KEYSPACE IF NOT EXISTS company_keyspace
13 WITH replication = { ’ class ’: ’ SimpleStrategy ’, ’ replication_factor ’: 1}
14 """)
15
16 # Use keyspace
17 session . set_keyspace ( ’ company_keyspace ’)
18
19 # Create table
20 session . execute ( " " "
21 CREATE TABLE IF NOT EXISTS employee_timeline (
22 emp_id UUID ,
23 event_time timestamp ,
24 event_type text ,
25 description text ,
26 PRIMARY KEY ( emp_id , event_time )
27 ) WITH CLUSTERING ORDER BY ( event_time DESC )
28 """)
29
30 # Insert data
31 emp_id = uuid4 ()
32 session . execute ( " " "
33 INSERT INTO employee_timeline ( emp_id , event_time , event_type , description )
34 VALUES (% s , %s , %s , % s )
35 " " " , ( emp_id , datetime . now () , ’ LOGIN ’ , ’ User logged into system ’) )
36
37 # Batch insert
38 batch = []
39 for i in range (5) :
40 batch . append (( emp_id , datetime . now () , ’ ACTIVITY ’ , f ’ Performed action { i } ’) )
41
42 prepared = session . prepare ( " " "
43 INSERT INTO employee_timeline ( emp_id , event_time , event_type , description )
44 VALUES (? , ? , ? , ?)
45 """)
46 for item in batch :
47 session . execute ( prepared , item )
48
49 # Query data
50 rows = session . execute ( " " "
51 SELECT * FROM employee_timeline
52 WHERE emp_id = % s
53 LIMIT 10
54 " " " , ( emp_id ,) )
55
56 print ( " Employee timeline : " )
57 for row in rows :

20
 21

58 print ( f " { row . event_time }: { row . event_type } - { row . description } " )

59
60 # Close connection
61 cluster . shutdown ()

COMPARISON AND BEST PRACTICES

When to Use Each Database Type
• Structured data with relationships: Recommended Database: Relational (SQLite,
PostgreSQL) Reason: ACID compliance, joins, constraints

• Unstructured/semi-structured data: Recommended Database: MongoDB Reason:

Flexible schema, nested documents

• Caching, session management: Recommended Database: Redis Reason: Ultra-fast

key-value access

• Time-series data, analytics: Recommended Database: Cassandra Reason: High write

throughput, wide columns

• Complex relationships: Recommended Database: Neo4j Reason: Graph traversal op-

timization

SUMMARY OF DATABASE OPERATIONS

• Connect:

– SQLite: [Link]()
– MySQL/PostgreSQL: [Link]() / [Link]()
– MongoDB: MongoClient()
– Redis: [Link]()

• Insert:

– SQLite: [Link](’INSERT ...’)

– MySQL/PostgreSQL: Same as SQLite (parameterized query)
– MongoDB: [Link] one()
– Redis: [Link]()

• Query:

– SQLite: [Link](’SELECT ...’)

– MySQL/PostgreSQL: Same as SQLite
– MongoDB: [Link]()
– Redis: [Link]()

• Update:

– SQLite: [Link](’UPDATE ...’)

– MySQL/PostgreSQL: Same as SQLite
– MongoDB: [Link] one()

21
22

– Redis: [Link]()

• Delete:

– SQLite: [Link](’DELETE ...’)

– MySQL/PostgreSQL: Same as SQLite
– MongoDB: [Link] one()
– Redis: [Link]()

• With pandas:

– SQLite: [Link] sql query()

– MySQL/PostgreSQL: Same as SQLite
– MongoDB: [Link](list([Link]()))
– Redis: Not applicable

Accessing and Processing Data from APIs (REST, SOAP)

Introduction to APIs
Definition: An API (Application Programming Interface) is a set of protocols, routines, and
tools that allows different software applications to communicate with each other. In the context
of web services, APIs enable programs to request data or services from remote servers over
HTTP, providing a standardized way to access functionality without needing to understand the
internal implementation of the service.
Purpose: APIs have become a primary method for acquiring data from online sources.
They allow developers to leverage existing services, access remote data, and build integrated
systems without understanding the internal workings of the service provider.

REST APIs
Definition: REST (Representational State Transfer) is an architectural style for designing web
services. A RESTful API uses HTTP requests to GET, PUT, POST, and DELETE data. It is built
on the concept of resources, where each resource is identified by a URL.
Core Principles of REST:

• Stateless: Each request from client to server must contain all information needed to
understand the request. The server does not store any client context between requests.
This improves scalability because the server doesn’t need to maintain session state.

• Client-Server: Separation of concerns between the user interface (client) and data stor-
age (server), allowing them to evolve independently. Clients don’t need to know about
data storage, and servers don’t need to know about the user interface.

• Cacheable: Responses must implicitly or explicitly define themselves as cacheable or non-

cacheable. This improves performance by allowing clients to reuse responses for equivalent
requests.

• Uniform Interface: Resources are identified in requests, manipulated through represen-

tations, and include self-descriptive messages. This simplifies the overall system architec-
ture.

22
 23

• Layered System: Intermediary servers (proxies, gateways, load balancers) can exist
between client and server without either knowing. This improves scalability and enables
security policies.
• Code on Demand (optional): Servers can temporarily extend client functionality by
transferring executable code (like JavaScript).

HTTP Methods in REST

• GET Purpose: Retrieve resource
Safe: Yes
Idempotent: Yes
Description: Request data from server; never changes server state
• POST Purpose: Create resource
Safe: No
Idempotent: No
Description: Submit data to server to create a new resource
• PUT Purpose: Update/replace resource
Safe: No
Idempotent: Yes
Description: Update entire resource; multiple identical requests have same effect
• PATCH Purpose: Partial update
Safe: No
Idempotent: No
Description: Partially modify resource
• DELETE Purpose: Remove resource
Safe: No
Idempotent: Yes
Description: Delete resource; multiple identical requests have same effect
• HEAD Purpose: GET without body
Safe: Yes
Idempotent: Yes
Description: Retrieve headers only, useful for checking resource existence
• OPTIONS Purpose: Available methods
Safe: Yes
Idempotent: Yes
Description: Discover allowed operations on a resource

HTTP Status Codes

• Success (2xx)
– 200 OK – Request succeeded; response body contains requested data
– 201 Created – Resource created successfully; typically returned after POST
– 204 No Content – Success but no response body; common for DELETE operations
• Redirection (3xx)
– 301 Moved Permanently – Resource has new URL; client should update book-
marks

23
24

– 304 Not Modified – Resource hasn’t changed; client can use cached version

• Client Error (4xx)

– 400 Bad Request – Malformed request syntax; server cannot understand

– 401 Unauthorized – Authentication required; client must authenticate
– 403 Forbidden – Authenticated but not authorized to access resource
– 404 Not Found – Resource doesn’t exist at the specified URL
– 429 Too Many Requests – Rate limit exceeded; client should slow down

• Server Error (5xx)

– 500 Internal Server Error – Generic server error; unexpected condition

– 502 Bad Gateway – Invalid response from upstream server
– 503 Service Unavailable – Server overloaded or down for maintenance

Request Components in API Calls

• URL/Endpoint: The address where the API resource is located
(e.g., [Link]

• HTTP Method: Indicates the desired operation: GET, POST, PUT, DELETE, etc.

• Headers: Metadata about the request, including:

– Content-Type: Format of data being sent (e.g., application/json, application/xml)

– Accept: Expected format of response
– Authorization: Authentication credentials
– User-Agent: Client identification

• Parameters: Can be included in different locations:

– Query parameters: Appended to URL after ? (e.g., ?page=1&limit=10)

– Path parameters: Part of URL path (e.g., /users/123 where 123 is a parameter)

• Request Body: Data sent with POST or PUT requests, usually in JSON or XML format

Response Components
• Status Code: Three-digit number indicating success or failure of the request

• Headers: Metadata about the response, such as Content-Type, Content-Length, Cache-Control

• Body: The actual data returned, usually in JSON format

Authentication Methods
• API Keys: Simple token passed in headers, URL, or body. Easy to implement but less
secure. Commonly used for public APIs with basic access control. Example:
Headers: {"X-API-Key": "your api key here"}

• Basic Authentication: Username and password sent in the Authorization header,

Base64 encoded. Requires HTTPS for security as encoding is not encryption. Example:
Headers: {"Authorization": "Basic dXNlcm5hbWU6cGFzc3dvcmQ="}

24
 25

• Bearer Tokens (OAuth 2.0): Token-based authentication where a token is issued after
authentication and included in subsequent requests. Tokens have limited lifetime and can
be scoped to specific permissions. Example:
Headers: {"Authorization": "Bearer eyJhbGciOiJIUzI1NiIs..."}

• JWT (JSON Web Tokens): Self-contained tokens that include claims (user infor-
mation, permissions) and a digital signature. The server can verify token authenticity
without storing session state.

SOAP APIs
Definition: SOAP (Simple Object Access Protocol) is a protocol specification for exchanging
structured information in web services. It uses XML for message format and typically relies on
HTTP or SMTP for transmission. SOAP has strict standards and formal contracts between
services.
Key Characteristics:

• XML-Based: All messages must be well-formed XML following strict schemas

• Protocol Agnostic: Can run over HTTP, SMTP, TCP, JMS, not limited to web proto-
cols

• Language and Platform Independent: Any language on any platform can implement
SOAP clients and servers

• Strict Standards: Formal contracts and validation through WSDL ensure compatibility

• Extensible: WS-* specifications add capabilities like security, transactions, and reliable
messaging

• Built-in Error Handling: SOAP Faults provide structured error information

SOAP Message Structure:

<?xml version="1.0"?>
<soap:Envelope
xmlns:soap="[Link]

<soap:Header>
<!-- Optional header elements for metadata -->
<auth:Authentication xmlns:auth="[Link]
<auth:Username>user</auth:Username>
<auth:Password>pass</auth:Password>
</auth:Authentication>
</soap:Header>

<soap:Body>
<!-- Required body with request or response data -->
<m:GetPrice xmlns:m="[Link]
<m:Symbol>IBM</m:Symbol>
</m:GetPrice>
</soap:Body>

<soap:Fault>
<!-- Optional fault element for errors (only in responses) -->

25
26

<faultcode>soap:Server</faultcode>
<faultstring>Processing error</faultstring>
<detail>Error details here</detail>
</soap:Fault>
</soap:Envelope>

SOAP Message Components

• Envelope (Required) Root element that wraps the entire message; identifies the XML
as a SOAP message

• Header (Optional) Contains metadata such as authentication, routing instructions, or

transaction context

• Body (Required) Contains the actual request or response data, including method name
and parameters

• Fault (Optional) Provides error information returned when a request fails; only appears
in responses

WSDL (Web Services Description Language)

WSDL is an XML document that describes a web service interface, providing a complete con-
tract for clients. Key components include:

• Types: Data type definitions using XML Schema

• Messages: Abstract definitions of the data being exchanged between client and service

• PortTypes: Operations that can be performed and the messages involved

• Bindings: Protocol and data format details for each port type

• Services: Collection of ports with network locations (endpoints)

SOAP vs. REST Comparison

• Protocol: SOAP – Protocol with strict rules
REST – Architectural style

• Message Format: SOAP – XML only

REST – JSON, XML, HTML, plain text

• Transport: SOAP – HTTP, SMTP, JMS, TCP, etc.

REST – HTTP/HTTPS only

• State: SOAP – Can be stateless or stateful

REST – Stateless

• Security: SOAP – WS-Security (built-in to protocol)

REST – HTTPS, OAuth (external)

• Caching: SOAP – Not supported natively

REST – HTTP caching mechanisms

• Performance: SOAP – Slower (XML parsing overhead)

REST – Faster (lightweight formats)

26
 27

• Error Handling: SOAP – Built-in Fault elements

REST – HTTP status codes

• Tooling: SOAP – Extensive enterprise tooling

REST – Simple HTTP tools

• Use Cases: SOAP – Enterprise, banking, telecom, healthcare

REST – Web, mobile, public APIs

API Best Practices

Rate Limiting: Rate limiting restricts how frequently a client can make requests within a
given time frame. This prevents abuse, ensures fair usage among clients, and protects server
resources.

• Common Rate Limit Headers:

– X-RateLimit-Limit: Maximum requests allowed per time window

– X-RateLimit-Remaining: Number of requests remaining in current window
– X-RateLimit-Reset: Time when the limit resets (Unix timestamp)
– Retry-After: Seconds to wait when rate limited (sent with 429 status)

Error Handling: Proper error handling distinguishes between different types of failures:

• HTTP Errors (4xx, 5xx): Server returned an error response

• Connection Errors: Network issues, DNS failures, refused connections

• Timeout Errors: Server took too long to respond

• SSL Errors: Certificate validation failures

Exception Patterns:

• Use Specific Exceptions: Different error types should raise different exceptions

• Include Context: Exceptions should include URL, status code, and error details

• Retry Logic: Transient errors (timeouts, 5xx) should be retried with backoff

• Fail Fast: Validation errors should fail immediately without making request

REST API PROGRAMS

Basic GET Requests
Installing and Importing Requests

1 # Install requests library

2 # pip install requests
3
4 import requests
5 import json
6 from pprint import pprint

27
 28

Simple GET Request

1 import requests
2
3 # Send a GET request to a public API
4 response = requests . get ( " https :// jsonplaceholder . typicode . com / posts /1 " )
5
6 # Check status code
7 print ( f " Status Code : { response . status_code } " )
8
9 # Check if request was successful
10 if response . status_code == 200:
11 print ( " Request successful ! " )
12
13 # Response content as string
14 print ( f " Raw response ( first 100 chars ) : { response . text [:100]}... " )
15
16 # Parse JSON response
17 data = response . json ()
18 print ( f " \ nParsed data type : { type ( data ) } " )
19 print ( f " Title : { data [ ’ title ’]} " )
20 print ( f " Body : { data [ ’ body ’][:50]}... " )
21 else :
22 print ( f " Error : { response . status_code } " )

Examining Response Headers

1 import requests
2
3 # Send GET request to API
4 response = requests . get ( " https :// jsonplaceholder . typicode . com / posts " )
5
6 # Basic response info
7 print ( f " Status Code : { response . status_code } " )
8 print ( f " Response URL : { response . url } " )
9
10 # View all response headers
11 print ( " \n - - - Response Headers ---" )
12 for key , value in response . headers . items () :
13 print ( f " { key }: { value } " )
14
15 # Check specific headers
16 content_type = response . headers . get ( ’ Content - Type ’)
17 print ( f " \ nContent Type : { content_type } " )
18 print ( f " Server : { response . headers . get ( ’ Server ’, ’ Not specified ’) } " )

Parsing Different Response Formats

1 import requests
2
3 # Send GET request to API
4 response = requests . get ( " https :// jsonplaceholder . typicode . com / posts /1 " )
5
6 # Method 1: Get as text ( raw string )
7 text_response = response . text
8 print ( f " Text response ( type : { type ( text_response ) }) " )
9 print ( f " First 100 chars : { text_response [:100]}... " )
10
11 # Method 2: Get as bytes ( for binary data )
12 bytes_response = response . content

28
 29

13 print ( f " \ nBytes response ( type : { type ( bytes_response ) }) " )

14 print ( f " First 50 bytes : { bytes_response [:50]} " )
15
16 # Method 3: Parse as JSON ( for JSON APIs )
17 json_response = response . json ()
18 print ( f " \ nJSON response ( type : { type ( json_response ) }) " )
19 print ( f " Title : { json_response [ ’ title ’]} " )

GET Request with Query Parameters

1 import requests
2
3 # Method 1: Pass parameters as dictionary
4 params = {
5 ’ userId ’: 1 ,
6 ’ _limit ’: 3
7 }
8
9 response = requests . get (
10 " https :// jsonplaceholder . typicode . com / posts " ,
11 params = params
12 )
13
14 print ( f " URL called : { response . url } " ) # Shows full URL with parameters
15 posts = response . json ()
16
17 print ( f " \ nRetrieved { len ( posts ) } posts for user 1: " )
18 for i , post in enumerate ( posts , 1) :
19 print ( f " { i }. { post [ ’ title ’]} " )
20
21 # Method 2: Different endpoint with parameters
22 response2 = requests . get (
23 " https :// jsonplaceholder . typicode . com / comments " ,
24 params ={ ’ postId ’: 1 , ’ _limit ’: 2}
25 )
26 comments = response2 . json ()
27 print ( f " \ nComments for post 1: { len ( comments ) } " )
28 for comment in comments :
29 print ( f " - { comment [ ’ name ’]} " )

Accessing Nested JSON Data

1 import requests
2
3 # Get a todo item
4 response = requests . get ( " https :// jsonplaceholder . typicode . com / todos /1 " )
5 todo = response . json ()
6
7 # Access nested data ( though this API is flat , pattern applies )
8 print ( " Todo Item : " )
9 print ( f " ID : { todo . get ( ’ id ’) } " )
10 print ( f " Title : { todo . get ( ’ title ’) } " )
11 print ( f " Completed : { todo . get ( ’ completed ’) } " )
12
13 # Get user details
14 user_response = requests . get ( f " https :// jsonplaceholder . typicode . com / users /{ todo
. get ( ’ userId ’) } " )
15 user = user_response . json ()
16
17 print ( f " \ nAssigned User : " )

29
 30

18 print ( f " Name : { user . get ( ’ name ’) } " )

19 print ( f " Email : { user . get ( ’ email ’) } " )
20 print ( f " City : { user . get ( ’ address ’, {}) . get ( ’ city ’) } " ) # Nested access

POST Requests
Sending JSON Data

1 import requests
2
3 # Data to send ( as Python dictionary )
4 new_post = {
5 ’ title ’: ’ My New Post ’ ,
6 ’ body ’: ’ This is the content of my new post created via API . ’ ,
7 ’ userId ’: 1
8 }
9
10 # Send POST request with JSON data
11 response = requests . post (
12 " https :// jsonplaceholder . typicode . com / posts " ,
13 json = new_post # ’ json ’ parameter automatically sets Content - Type header
14 )
15
16 print ( f " Status Code : { response . status_code } " )
17
18 if response . status_code == 201: # 201 = Created
19 created_post = response . json ()
20 print ( f " Created post with ID : { created_post . get ( ’ id ’) } " )
21 print ( f " Title : { created_post . get ( ’ title ’) } " )
22 print ( f " Body : { created_post . get ( ’ body ’) } " )

Sending Form Data

1 import requests
2
3 # Form data ( like HTML form submission )
4 form_data = {
5 ’ username ’: ’ john_doe ’ ,
6 ’ email ’: ’ john@example . com ’ ,
7 ’ age ’: 30
8 }
9
10 # Send as form - encoded data using ’ data ’ parameter
11 response = requests . post (
12 " https :// httpbin . org / post " , # Test endpoint that echoes request
13 data = form_data
14 )
15
16 result = response . json ()
17 print ( " Form data received by server : " )
18 for key , value in result [ ’ form ’ ]. items () :
19 print ( f " { key }: { value } " )
20
21 # Note the Content - Type header
22 print ( f " \ nContent - Type sent : { result [ ’ headers ’]. get ( ’ Content - Type ’) } " )
23 # Should be : application /x - www - form - urlencoded

Sending Custom Headers

30
 31

1 import requests
2
3 # Custom headers
4 headers = {
5 ’ User - Agent ’: ’ MyApplication /1.0 ’ ,
6 ’ Accept ’: ’ application / json ’ ,
7 ’X - Custom - Header ’: ’ custom - value -123 ’ ,
8 ’X - API - Version ’: ’2 ’
9 }
10
11 data = {
12 ’ name ’: ’ Test Product ’ ,
13 ’ price ’: 29.99 ,
14 ’ category ’: ’ electronics ’
15 }
16
17 response = requests . post (
18 " https :// httpbin . org / post " ,
19 json = data ,
20 headers = headers
21 )
22
23 result = response . json ()
24 print ( " Headers sent to server : " )
25 for key , value in result [ ’ headers ’ ]. items () :
26 print ( f " { key }: { value } " )
27
28 print ( " \ nData sent : " )
29 print ( f " { result [ ’ json ’]} " )

PUT and PATCH Requests

PUT Request (Full Update)

1 import requests
2
3 # PUT replaces the entire resource
4 updated_post = {
5 ’ id ’: 1 ,
6 ’ title ’: ’ Updated Title ’ ,
7 ’ body ’: ’ This post has been completely updated . ’ ,
8 ’ userId ’: 1
9 }
10
11 response = requests . put (
12 " https :// jsonplaceholder . typicode . com / posts /1 " ,
13 json = updated_post
14 )
15
16 print ( f " PUT Status : { response . status_code } " )
17 if response . status_code == 200:
18 result = response . json ()
19 print ( f " Updated title : { result [ ’ title ’]} " )

PATCH Request (Partial Update)

1 import requests
2
3 # PATCH updates only specified fields
4 partial_update = {

31
 32

5 ’ title ’: ’ Partially Updated Title ’

6 }
7
8 response = requests . patch (
9 " https :// jsonplaceholder . typicode . com / posts /1 " ,
10 json = partial_update
11 )
12
13 print ( f " PATCH Status : { response . status_code } " )
14 if response . status_code == 200:
15 result = response . json ()
16 print ( f " Updated title : { result [ ’ title ’]} " )

PATCH Request (Partial Update)

1 import requests
2
3 # PATCH only updates specified fields
4 partial_update = {
5 ’ title ’: ’ Partially Updated Title ’ # Only change the title
6 }
7
8 response = requests . patch (
9 " https :// jsonplaceholder . typicode . com / posts /1 " ,
10 json = partial_update
11 )
12
13 print ( f " PATCH Status : { response . status_code } " )
14 if response . status_code == 200:
15 result = response . json ()
16 print ( f " New title : { result [ ’ title ’]} " )
17 print ( f " Body unchanged : { result [ ’ body ’][:30]}... " )

DELETE Requests

1 import requests
2
3 # DELETE request
4 response = requests . delete ( " https :// jsonplaceholder . typicode . com / posts /1 " )
5
6 print ( f " DELETE Status : { response . status_code } " )
7
8 # 204 No Content indicates successful deletion with no response body
9 if response . status_code == 204:
10 print ( " Resource deleted successfully " )
11 elif response . status_code == 200:
12 print ( " Resource deleted , response body : " , response . json () )
13 elif response . status_code == 404:
14 print ( " Resource not found " )

Authentication Examples
API Key Authentication

1 import requests
2
3 # API key in headers ( common pattern )
4 headers = {
5 ’X - API - Key ’: ’ your_api_key_here ’ ,

32
 33

6 ’ Content - Type ’: ’ application / json ’

7 }
8
9 response = requests . get (
10 " https :// api . example . com / v1 / data " ,
11 headers = headers
12 )
13
14 if response . status_code == 200:
15 data = response . json ()
16 print ( " Authenticated successfully " )
17 else :
18 print ( f " Authentication failed : { response . status_code } " )

Basic Authentication

1 import requests
2 from requests . auth import HTTPBasicAuth
3
4 # Method 1: Using HTTPBasicAuth helper
5 response = requests . get (
6 " https :// api . example . com / protected " ,
7 auth = HTTPBasicAuth ( ’ username ’ , ’ password ’)
8 )
9
10 # Method 2: Shortcut syntax ( recommended )
11 response = requests . get (
12 " https :// api . example . com / protected " ,
13 auth =( ’ username ’ , ’ password ’)
14 )
15
16 print ( f " Status : { response . status_code } " )
17 if response . status_code == 200:
18 print ( " Authentication successful " )

Bearer Token Authentication

1 import requests
2
3 class APIClient :
4 " " " Simple API client with bearer token authentication " " "
5
6 def __init__ ( self , base_url , access_token ) :
7 self . base_url = base_url . rstrip ( ’/ ’)
8 self . access_token = access_token
9 self . session = requests . Session ()
10
11 # Set default headers for all requests
12 self . session . headers . update ({
13 ’ Authorization ’: f ’ Bearer { access_token } ’ ,
14 ’ Content - Type ’: ’ application / json ’ ,
15 ’ Accept ’: ’ application / json ’
16 })
17
18 def get ( self , endpoint , params = None ) :
19 " " " Make authenticated GET request " " "
20 url = f " { self . base_url }/{ endpoint . lstrip ( ’/ ’) } "
21 response = self . session . get ( url , params = params )
22 return response
23

33
 34

24 def post ( self , endpoint , data = None ) :

25 " " " Make authenticated POST request " " "
26 url = f " { self . base_url }/{ endpoint . lstrip ( ’/ ’) } "
27 response = self . session . post ( url , json = data )
28 return response
29
30 def close ( self ) :
31 " " " Close the session " " "
32 self . session . close ()
33
34 # Usage
35 client = APIClient (
36 base_url = " https :// api . example . com / v1 " ,
37 access_token = " e y J h b G c i O i J I U z I 1 N i I s I n R 5 c C I 6 I k p X V C J 9 ... "
38 )
39
40 try :
41 response = client . get ( " users / me " )
42 if response . status_code == 200:
43 user_data = response . json ()
44 print ( f " Authenticated as : { user_data . get ( ’ name ’, ’ Unknown ’) } " )
45 finally :
46 client . close ()

Session Management

1 import requests
2
3 # Using Session for multiple requests
4 # Sessions maintain cookies and connection pooling
5 session = requests . Session ()
6
7 # Set default headers for all requests in this session
8 session . headers . update ({
9 ’ User - Agent ’: ’ MyApp /1.0 ’ ,
10 ’ Accept ’: ’ application / json ’
11 })
12
13 # Login once ( if authentication sets cookies )
14 login_data = { ’ username ’: ’ user ’ , ’ password ’: ’ pass ’}
15 login_response = session . post (
16 " https :// api . example . com / login " ,
17 json = login_data
18 )
19
20 # Session automatically maintains cookies
21 if login_response . status_code == 200:
22 print ( " Logged in successfully " )
23
24 # Subsequent requests use the same session
25 # No need to send credentials again
26 profile_response = session . get ( " https :// api . example . com / profile " )
27 d ash board_response = session . get ( " https :// api . example . com / dashboard " )
28
29 print ( f " Profile status : { profile_response . status_code } " )
30 print ( f " Dashboard status : { dashboard_response . status_code } " )
31
32 # Access protected resources
33 data_response = session . get (
34 " https :// api . example . com / data " ,
35 params ={ ’ limit ’: 10}
36 )

34
 35

37
38 if data_response . status_code == 200:
39 data = data_response . json ()
40 print ( f " Retrieved { len ( data ) } items " )
41
42 # Close session when done
43 session . close ()

Error Handling

1 import requests
2 from requests . exceptions import RequestException , ConnectionError , HTTPError ,
Timeout
3
4 def safe_api_request ( url , timeout =10) :
5 " " " Make API request with comprehensive error handling " " "
6
7 try :
8 response = requests . get ( url , timeout = timeout )
9
10 # Raise exception for HTTP errors (4 xx , 5 xx )
11 response . raise_for_status ()
12
13 # Success - return parsed JSON
14 return response . json ()
15
16 except ConnectionError as e :
17 print ( f " Connection Error : Could not connect to server - { e } " )
18 print ( " Check your internet connection or if the server is available " )
19 return None
20
21 except Timeout as e :
22 print ( f " Timeout Error : Request timed out after { timeout } seconds " )
23 print ( " The server is taking too long to respond " )
24 return None
25
26 except HTTPError as e :
27 print ( f " HTTP Error : { response . status_code } " )
28
29 # Handle specific status codes
30 if response . status_code == 400:
31 print ( " Bad Request - Check your parameters " )
32 elif response . status_code == 401:
33 print ( " Unauthorized - Check your authentication " )
34 elif response . status_code == 403:
35 print ( " Forbidden - You don ’t have permission " )
36 elif response . status_code == 404:
37 print ( " Not Found - The resource doesn ’t exist " )
38 elif response . status_code == 429:
39 print ( " Rate Limited - Too many requests " )
40 retry_after = response . headers . get ( ’ Retry - After ’ , ’ unknown ’)
41 print ( f " Retry after { retry_after } seconds " )
42 elif response . status_code >= 500:
43 print ( " Server Error - The server had a problem " )
44 return None
45
46 except RequestException as e :
47 print ( f " General Request Error : { e } " )
48 return None
49
50 # Usage
51 data = safe_api_request ( " https :// jsonplaceholder . typicode . com / posts /1 " )

35
 36

52 if data :
53 print ( f " Success ! Retrieved : { data [ ’ title ’]} " )

Complete REST API Client Example

1 import requests
2 import time
3 import logging
4 from requests . exceptions import RequestException
5
6 # Set up logging
7 logging . basicConfig ( level = logging . INFO )
8 logger = logging . getLogger ( __name__ )
9
10 class RESTAPIClient :
11 """
12 Complete REST API Client with authentication , error handling ,
13 retry logic , and session management
14 """
15
16 def __init__ ( self , base_url , auth_token = None , timeout =30 , max_retries =3) :
17 self . base_url = base_url . rstrip ( ’/ ’)
18 self . timeout = timeout
19 self . max_retries = max_retries
20 self . session = requests . Session ()
21 self . session . headers . update ({
22 ’ User - Agent ’: ’ RESTAPIClient /1.0 ’ ,
23 ’ Accept ’: ’ application / json ’ ,
24 ’ Content - Type ’: ’ application / json ’
25 })
26 if auth_token :
27 self . session . headers . update ({ ’ Authorization ’: f ’ Bearer { auth_token }
’ })
28 logger . info ( f " API Client initialized for { base_url } " )
29
30 def _request ( self , method , endpoint , params = None , data = None , retry_count =0)
:
31 url = f " { self . base_url }/{ endpoint . lstrip ( ’/ ’) } "
32 try :
33 logger . debug ( f " Making { method } request to { url } " )
34 response = self . session . request (
35 method = method ,
36 url = url ,
37 params = params ,
38 json = data ,
39 timeout = self . timeout
40 )
41 if response . status_code == 429:
42 retry_after = int ( response . headers . get ( ’ Retry - After ’ , 5) )
43 logger . warning ( f " Rate limited . Waiting { retry_after } seconds " )
44 if retry_count < self . max_retries :
45 time . sleep ( retry_after )
46 return self . _request ( method , endpoint , params , data ,
retry_count + 1)
47 else :
48 logger . error ( " Max retries exceeded for rate limit " )
49 response . raise_for_status ()
50 response . raise_for_status ()
51 return response
52 except requests . exceptions . ConnectionError as e :
53 logger . error ( f " Connection error : { e } " )
54 if retry_count < self . max_retries :

36
 37

55 wait = 2 ** retry_count
56 logger . info ( f " Retrying in { wait } seconds ... " )
57 time . sleep ( wait )
58 return self . _request ( method , endpoint , params , data ,
retry_count + 1)
59 raise
60 except requests . exceptions . Timeout as e :
61 logger . error ( f " Timeout error : { e } " )
62 if retry_count < self . max_retries :
63 wait = 2 ** retry_count
64 logger . info ( f " Retrying in { wait } seconds ... " )
65 time . sleep ( wait )
66 return self . _request ( method , endpoint , params , data ,
retry_count + 1)
67 raise
68 except requests . exceptions . HTTPError as e :
69 logger . error ( f " HTTP error { response . status_code }: { e } " )
70 raise
71 except Exception as e :
72 logger . error ( f " Unexpected error : { e } " )
73 raise
74
75 def get ( self , endpoint , params = None ) :
76 response = self . _request ( ’ GET ’ , endpoint , params = params )
77 return response . json () if response . content else None
78
79 def post ( self , endpoint , data = None ) :
80 response = self . _request ( ’ POST ’ , endpoint , data = data )
81 return response . json () if response . content else None
82
83 def put ( self , endpoint , data = None ) :
84 response = self . _request ( ’ PUT ’ , endpoint , data = data )
85 return response . json () if response . content else None
86
87 def patch ( self , endpoint , data = None ) :
88 response = self . _request ( ’ PATCH ’ , endpoint , data = data )
89 return response . json () if response . content else None
90
91 def delete ( self , endpoint ) :
92 response = self . _request ( ’ DELETE ’ , endpoint )
93 return response . status_code == 204
94
95 def close ( self ) :
96 self . session . close ()
97 logger . info ( " API Client closed " )
98
99 # Usage example
100 if __name__ == " __main__ " :
101 client = RESTAPIClient (
102 base_url = " https :// jsonplaceholder . typicode . com " ,
103 timeout =10 ,
104 max_retries =2
105 )
106 try :
107 posts = client . get ( " posts " , params ={ " _limit " : 3})
108 print ( f " Retrieved { len ( posts ) } posts " )
109 for post in posts :
110 print ( f " - { post [ ’ title ’]} " )
111 new_post = client . post ( " posts " , data ={
112 " title " : " API Test Post " ,
113 " body " : " Created via REST client " ,
114 " userId " : 1
115 })

37
 38

116 print ( f " \ nCreated post with ID : { new_post . get ( ’ id ’) } " )

117 single = client . get ( f " posts /{ new_post . get ( ’ id ’) } " )
118 print ( f " Retrieved : { single [ ’ title ’]} " )
119 updated = client . put ( f " posts /{ new_post . get ( ’ id ’) } " , data ={
120 " id " : new_post . get ( ’ id ’) ,
121 " title " : " Updated Title " ,
122 " body " : " Updated content " ,
123 " userId " : 1
124 })
125 print ( f " Updated : { updated [ ’ title ’]} " )
126 success = client . delete ( f " posts /{ new_post . get ( ’ id ’) } " )
127 print ( f " Deleted : { success } " )
128 finally :
129 client . close ()

SOAP API PROGRAMS

Introduction to Zeep
Zeep is a Python library for SOAP web services. It inspects WSDL documents and generates
the corresponding code to interact with the SOAP web service.
1 # Install zeep
2 # pip install zeep
3
4 from zeep import Client
5 from zeep . exceptions import Fault , TransportError
6 import logging
7
8 # Enable logging for debugging
9 logging . basicConfig ( level = logging . INFO )
10 logger = logging . getLogger ( __name__ )

Basic SOAP Client

Creating a SOAP Client from WSDL

1 from zeep import Client

2
3 # Create a client from WSDL URL
4 wsdl_url = " http :// www . dneonline . com / calculator . asmx ? wsdl " # Public calculator
service
5 client = Client ( wsdl_url )
6
7 print ( " SOAP Client created successfully " )
8 print ( f " Service : { client . wsdl . services [0]. name } " )

Viewing Available Operations

1 from zeep import Client

2 from pprint import pprint
3
4 client = Client ( " http :// www . dneonline . com / calculator . asmx ? wsdl " )
5
6 # Get service
7 service = client . wsdl . services [0]
8 print ( f " Service : { service . name } " )
9
10 # Get ports

38
 39

11 for port in service . ports . values () :

12 print ( f " \ nPort : { port . name } " )
13 print ( " Operations : " )
14 for operation_name , operation in port . binding . _operations . items () :
15 print ( f " - { operation_name } " )
16 print ( f " Input : { operation . input . signature } " )

Calling SOAP Methods

1 from zeep import Client

2
3 # Calculator service
4 client = Client ( " http :// www . dneonline . com / calculator . asmx ? wsdl " )
5
6 # Call Add method
7 result = client . service . Add ( intA =10 , intB =20)
8 print ( f " 10 + 20 = { result } " )
9
10 # Call Subtract method
11 result = client . service . Subtract ( intA =50 , intB =15)
12 print ( f " 50 - 15 = { result } " )
13
14 # Call Multiply method
15 result = client . service . Multiply ( intA =6 , intB =7)
16 print ( f " 6 * 7 = { result } " )
17
18 # Call Divide method
19 result = client . service . Divide ( intA =100 , intB =4)
20 print ( f " 100 / 4 = { result } " )

Working with Complex Types

1 from zeep import Client

2 from zeep . helpers import serialize_object
3
4 # Example with a more complex SOAP service
5 # Note : This is a conceptual example using a public SOAP service
6 wsdl_url = " http :// www . dneonline . com / calculator . asmx ? wsdl "
7 client = Client ( wsdl_url )
8
9 # For complex types , you may need to create objects
10 # The exact structure depends on the WSDL
11
12 # Example of inspecting types
13 for name , type_obj in client . wsdl . types . types . items () :
14 print ( f " Type : { name } " )

SOAP with Authentication

1 from zeep import Client

2 from zeep . transports import Transport
3 from requests import Session
4 from requests . auth import HTTPBasicAuth
5
6 # Create session with authentication
7 session = Session ()
8 session . auth = HTTPBasicAuth ( ’ username ’ , ’ password ’)
9
10 # Create transport with the authenticated session

39
 40

11 transport = Transport ( session = session )

12
13 # Create client with transport
14 client = Client (
15 " https :// example . com / secure / service ? wsdl " ,
16 transport = transport
17 )
18
19 # Now all SOAP calls will include authentication
20 try :
21 result = client . service . SecureOperation ( param1 = " value " )
22 print ( f " Result : { result } " )
23 except Exception as e :
24 print ( f " SOAP call failed : { e } " )

Web Scraping using Requests and BeautifulSoup

Introduction to Web Scraping

Definition:
Web scraping is the practice of using a computer program to sift through a web page and
gather the data that you need in a format most useful for you.
Purpose and Applications:
Web scraping is used to collect data from websites for various purposes including data
analysis, research, and building datasets when APIs are not available.

1.2 Web Scraping Workflow

The process of web scraping can be broken down into two parts: actually fetching the web page
(downloading it) and then extracting information from it.
Step 1: Fetching the Web Page
This involves sending an HTTP request to the server and receiving the HTML content of
the page.
Step 2: Extracting Information
Once you have the HTML, you need to parse it and extract the specific pieces of data you’re
interested in.

1.3 The Requests Library

The requests library is the standard tool for making HTTP requests from Python. It handles
the fetching part of web scraping.
Key Capabilities:

• Sending GET and POST requests

• Handling headers and parameters

• Managing cookies and sessions

• Processing response status codes

• Following redirects

40
 41

1.4 The BeautifulSoup Library

BeautifulSoup is a library for parsing HTML documents and extracting data from them. It
creates a parse tree from the HTML that makes it easy to navigate and search.
Key Capabilities:

• Parsing HTML regardless of how messy it is

• Navigating the document tree

• Searching for elements by tags, classes, IDs, or attributes

• Extracting text and attribute values

1.5 The HTML Document Object Model

HTML documents have a hierarchical structure. A simple HTML document looks like:
1 < html >
2 < head >
3 < title > Page Title </ title >
4 </ head >
5 < body >
6 < h1 > Main Heading </ h1 >
7 <p > First paragraph </ p >
8 <p > Second paragraph </ p >
9 </ body >
10 </ html >

In this structure, the <html> tag contains all other tags, <head> contains metadata, and
<body> contains the visible content.

1.6 Ethical and Legal Considerations

[Link]
Before scraping a website, you should check its [Link] file (e.g., [Link]
which specifies which parts of the site can be accessed by automated tools.
Rate Limiting
You should implement delays between requests to avoid overloading the server. This is both
polite and helps avoid being blocked.
User-Agent Identification
You should identify your bot with a proper User-Agent string rather than pretending to be
something you’re not.
Terms of Service
Always review the website’s terms of service to ensure scraping is permitted.

INSTALLATION AND BASIC SETUP

2.1 Installing Required Libraries

1 # Install using pip

2 # pip install requests
3 # pip install beautifulsoup4
4
5 # For better performance , also install lxml parser
6 # pip install lxml

41
 42

2.2 Importing Libraries

1 import requests
2 from bs4 import BeautifulSoup

FETCHING WEB PAGES WITH REQUESTS

3.1 Basic GET Request

1 import requests
2
3 # Make a request to a web page
4 response = requests . get ( ’ https :// example . com ’)
5
6 # Check the status code
7 print ( f " Status code : { response . status_code } " )
8
9 # A status code of 200 means the request was successful
10 if response . status_code == 200:
11 # Get the HTML content
12 html_content = response . text
13 print ( f " Retrieved { len ( html_content ) } bytes " )
14 else :
15 print ( f " Failed to retrieve page : { response . status_code } " )

3.2 Adding Headers

1 import requests
2
3 # Some websites require a proper User - Agent header
4 headers = {
5 ’ User - Agent ’: ’ Mozilla /5.0 ( Windows NT 10.0; Win64 ; x64 ) AppleWebKit /537.36
’
6 }
7
8 response = requests . get ( ’ https :// example . com ’ , headers = headers )
9 print ( f " Status code : { response . status_code } " )

3.3 Handling Errors

1 import requests
2 from requests . exceptions import RequestException
3
4 def fetch_page ( url ) :
5 " " " Safely fetch a web page " " "
6 try :
7 response = requests . get ( url , timeout =10)
8 response . raise_for_status () # Raise exception for 4 xx /5 xx status codes
9 return response . text
10 except RequestException as e :
11 print ( f " Error fetching page : { e } " )
12 return None
13
14 html = fetch_page ( ’ https :// example . com ’)
15 if html :
16 print ( f " Retrieved { len ( html ) } bytes " )

42
 43

PARSING HTML WITH BEAUTIFULSOUP

4.1 Creating a BeautifulSoup Object

1 from bs4 import BeautifulSoup

2
3 # Sample HTML document
4 html_doc = " " "
5 < html >
6 < head >
7 < title > Test Page </ title >
8 </ head >
9 < body >
10 <h1 > Main Heading </ h1 >
11 <p class = " content " > First paragraph </ p >
12 <p class = " content " > Second paragraph </ p >
13 <a href = " https :// example . com / link " id = " link " > Click here </ a >
14 </ body >
15 </ html >
16 """
17
18 # Create a BeautifulSoup object
19 soup = BeautifulSoup ( html_doc , ’ html . parser ’)
20
21 # The prettify method formats the HTML for display
22 print ( soup . prettify () )

4.2 Accessing Elements by Tag Name

1 from bs4 import BeautifulSoup

2
3 html_doc = " < html > < body > < h1 > Title </ h1 > <p > Paragraph </ p > </ body > </ html > "
4 soup = BeautifulSoup ( html_doc , ’ html . parser ’)
5
6 # Access the title tag
7 title_tag = soup . title
8 print ( f " Title tag : { title_tag } " )
9
10 # Access the body tag
11 body_tag = soup . body
12 print ( f " Body tag : { body_tag . name } " )
13
14 # Access the first paragraph
15 first_p = soup . p
16 print ( f " First paragraph : { first_p . string } " )
17
18 # Access all paragraphs
19 all_p = soup . find_all ( ’p ’)
20 print ( f " Number of paragraphs : { len ( all_p ) } " )

NAVIGATING THE PARSE TREE

5.1Getting Tag Contents

1 from bs4 import BeautifulSoup

2
3 html_doc = " " "
4 < html >

43
 44

5 < body >

6 <p > This is a < strong > simple </ strong > paragraph . </p >
7 </ body >
8 </ html >
9 """
10
11 soup = BeautifulSoup ( html_doc , ’ html . parser ’)
12 p_tag = soup . p
13
14 # Getting the text content ( all text inside the tag )
15 print ( f " Text : { p_tag . get_text () } " )
16
17 # Getting the string ( only if tag has one child )
18 print ( f " String : { p_tag . string } " )
19
20 # Getting the raw HTML
21 print ( f " Raw HTML : { p_tag } " )

5.2 Navigating with Tag Names

1 from bs4 import BeautifulSoup

2
3 html_doc = " " "
4 < html >
5 < body >
6 <div >
7 <h1 > Title </ h1 >
8 <p > First paragraph </ p >
9 <p > Second paragraph </ p >
10 </ div >
11 </ body >
12 </ html >
13 """
14
15 soup = BeautifulSoup ( html_doc , ’ html . parser ’)
16
17 # Access children of body
18 body = soup . body
19 for child in body . children :
20 print ( f " Child : { child . name } " )
21
22 # Access all descendants
23 for descendant in body . descendants :
24 if descendant . name : # Only print tag names
25 print ( f " Descendant : { descendant . name } " )

5.3 Navigating with Parent and Sibling

1 from bs4 import BeautifulSoup

2
3 html_doc = " " "
4 < html >
5 < body >
6 <h1 > Title </ h1 >
7 <p > First paragraph </ p >
8 <p > Second paragraph </ p >
9 </ body >
10 </ html >
11 """
12

44
 45

13 soup = BeautifulSoup ( html_doc , ’ html . parser ’)

14 first_p = soup . p
15
16 # Parent
17 print ( f " Parent : { first_p . parent . name } " )
18
19 # Next sibling
20 next_sib = first_p . next_sibling
21 while next_sib and next_sib . name is None : # Skip NavigableString objects
22 next_sib = next_sib . next_sibling
23 print ( f " Next sibling : { next_sib . name if next_sib else None } " )
24
25 # Previous sibling
26 prev_sib = first_p . previous_sibling
27 while prev_sib and prev_sib . name is None :
28 prev_sib = prev_sib . previous_sibling
29 print ( f " Previous sibling : { prev_sib . name if prev_sib else None } " )

SEARCHING THE PARSE TREE

6.1 Searching with find and find all

1 from bs4 import BeautifulSoup

2
3 html_doc = " " "
4 < html >
5 < body >
6 < div class = " container " >
7 < h1 class = " title " > Main Title </ h1 >
8 <p class = " content " > First paragraph </ p >
9 <p class = " content " > Second paragraph </ p >
10 <p class = " footer " > Footer paragraph </ p >
11 </ div >
12 </ body >
13 </ html >
14 """
15
16 soup = BeautifulSoup ( html_doc , ’ html . parser ’)
17
18 # find_all returns a list of all matching tags
19 all_p = soup . find_all ( ’p ’)
20 print ( f " All p tags : { len ( all_p ) } " )
21
22 # find returns the first matching tag
23 first_p = soup . find ( ’p ’)
24 print ( f " First p tag : { first_p . text } " )
25
26 # Search by class
27 content_p = soup . find_all ( ’p ’ , class_ = ’ content ’)
28 print ( f " Content paragraphs : { len ( content_p ) } " )
29
30 # Search by multiple attributes
31 specific = soup . find_all ( ’p ’ , { ’ class ’: ’ footer ’ })
32 print ( f " Footer paragraph : { specific [0]. text if specific else ’ None ’} " )

6.2 Searching with CSS Selectors

1 from bs4 import BeautifulSoup

2

45
 46

3 html_doc = " " "

4 < html >
5 < body >
6 < div id = " main " >
7 <p class = " content " > First </ p >
8 <p class = " content special " > Second </ p >
9 < div class = " nested " >
10 <p class = " content " > Nested </ p >
11 </ div >
12 </ div >
13 </ body >
14 </ html >
15 """
16
17 soup = BeautifulSoup ( html_doc , ’ html . parser ’)
18
19 # Select by tag
20 content_p = soup . select ( ’p ’)
21 print ( f " All p tags : { len ( content_p ) } " )
22
23 # Select by class
24 content = soup . select ( ’. content ’)
25 print ( f " Class ’ content ’: { len ( content ) } " )
26
27 # Select by id
28 main = soup . select ( ’# main ’)
29 print ( f " Element with id ’ main ’: { main [0]. name if main else ’ None ’} " )
30
31 # Select nested elements
32 nested = soup . select ( ’# main . nested . content ’)
33 print ( f " Nested content : { nested [0]. text if nested else ’ None ’} " )

6.3 Getting Attribute Values

1 from bs4 import BeautifulSoup

2
3 html_doc = " " "
4 < html >
5 < body >
6 <a href = " https :// example . com " class = " link " id = " link1 " > Example Link </ a >
7 < img src = " image . jpg " alt = " Sample Image " width = " 300 " >
8 </ body >
9 </ html >
10 """
11
12 soup = BeautifulSoup ( html_doc , ’ html . parser ’)
13 link = soup . a
14 img = soup . img
15
16 # Getting attributes using dictionary syntax
17 print ( f " Link href : { link [ ’ href ’]} " )
18 print ( f " Link class : { link [ ’ class ’]} " )
19
20 # Using get method ( safe if attribute might be missing )
21 print ( f " Link id : { link . get ( ’ id ’, ’ No id ’) } " )
22 print ( f " Link target : { link . get ( ’ target ’, ’ No target ’) } " )
23
24 # Getting multiple attributes from an image
25 print ( f " Image src : { img [ ’ src ’]} " )
26 print ( f " Image alt : { img [ ’ alt ’]} " )
27 print ( f " Image width : { img [ ’ width ’]} " )

46
 47

COMPLETE WEB SCRAPING EXAMPLES

7.1 Basic Example: Scraping a Simple Page

1 import requests
2 from bs4 import BeautifulSoup
3
4 # Step 1: Fetch the page
5 url = " http :// quotes . toscrape . com "
6 response = requests . get ( url )
7
8 if response . status_code == 200:
9 # Step 2: Parse the HTML
10 soup = BeautifulSoup ( response . text , ’ html . parser ’)
11
12 # Step 3: Find all quotes
13 quotes = soup . find_all ( ’ div ’ , class_ = ’ quote ’)
14
15 # Step 4: Extract data from each quote
16 for quote in quotes :
17 text = quote . find ( ’ span ’ , class_ = ’ text ’) . text
18 author = quote . find ( ’ small ’ , class_ = ’ author ’) . text
19
20 tags = quote . find_all ( ’a ’ , class_ = ’ tag ’)
21 tag_list = [ tag . text for tag in tags ]
22
23 print ( f " Quote : { text } " )
24 print ( f " Author : { author } " )
25 print ( f " Tags : { ’ , ’. join ( tag_list ) } " )
26 print ( " -" * 50)
27 else :
28 print ( f " Failed to retrieve page : { response . status_code } " )

7.2 Saving Scraped Data to CSV

1 import requests
2 from bs4 import BeautifulSoup
3 import csv
4
5 # Fetch and parse the page
6 url = " http :// quotes . toscrape . com "
7 response = requests . get ( url )
8 soup = BeautifulSoup ( response . text , ’ html . parser ’)
9
10 # Find all quotes
11 quotes = soup . find_all ( ’ div ’ , class_ = ’ quote ’)
12
13 # Prepare data for CSV
14 data = []
15 for quote in quotes :
16 text = quote . find ( ’ span ’ , class_ = ’ text ’) . text
17 author = quote . find ( ’ small ’ , class_ = ’ author ’) . text
18
19 tags = quote . find_all ( ’a ’ , class_ = ’ tag ’)
20 tags_str = ’ , ’. join ([ tag . text for tag in tags ])
21
22 data . append ([ text , author , tags_str ])
23
24 # Write to CSV
25 with open ( ’ quotes . csv ’ , ’w ’ , newline = ’ ’ , encoding = ’utf -8 ’) as file :
26 writer = csv . writer ( file )

47
 48

27 writer . writerow ([ ’ Quote ’ , ’ Author ’ , ’ Tags ’ ]) # Header

28 writer . writerows ( data )
29
30 print ( f " Saved { len ( data ) } quotes to quotes . csv " )

7.3 Scraping Multiple Pages

1 import requests
2 from bs4 import BeautifulSoup
3 import time
4
5 base_url = " http :// quotes . toscrape . com / page /{}/ "
6 all_quotes = []
7 page = 1
8
9 while True :
10 print ( f " Scraping page { page }... " )
11 url = base_url . format ( page )
12 response = requests . get ( url )
13
14 if response . status_code != 200:
15 print ( f " Failed to retrieve page { page } " )
16 break
17
18 soup = BeautifulSoup ( response . text , ’ html . parser ’)
19 quotes = soup . find_all ( ’ div ’ , class_ = ’ quote ’)
20
21 if not quotes :
22 print ( f " No quotes found on page { page } , stopping . " )
23 break
24
25 for quote in quotes :
26 text = quote . find ( ’ span ’ , class_ = ’ text ’) . text
27 author = quote . find ( ’ small ’ , class_ = ’ author ’) . text
28 all_quotes . append ({ ’ text ’: text , ’ author ’: author })
29
30 print ( f " Found { len ( quotes ) } quotes on page { page } " )
31
32 # Be polite - don ’t hammer the server
33 time . sleep (2)
34 page += 1
35
36 print ( f " Total quotes scraped : { len ( all_quotes ) } " )

7.4 Scraping with Proper Headers

1 import requests
2 from bs4 import BeautifulSoup
3
4 # Some websites block requests without proper headers
5 headers = {
6 ’ User - Agent ’: ’ Mozilla /5.0 ( Windows NT 10.0; Win64 ; x64 ) AppleWebKit /537.36
( KHTML , like Gecko ) Chrome /91.0.4472.124 Safari /537.36 ’ ,
7 ’ Accept ’: ’ text / html , application / xhtml + xml , application / xml ; q =0.9 , image / webp
,*/*; q =0.8 ’ ,
8 ’ Accept - Language ’: ’en - US , en ; q =0.5 ’ ,
9 ’ Connection ’: ’ keep - alive ’ ,
10 }
11
12 url = " http :// quotes . toscrape . com "

48
 49

13 response = requests . get ( url , headers = headers )

14
15 if response . status_code == 200:
16 soup = BeautifulSoup ( response . text , ’ html . parser ’)
17 print ( f " Page title : { soup . title . text } " )
18 else :
19 print ( f " Failed : { response . status_code } " )

7.5 Handling Different Parsers

1 from bs4 import BeautifulSoup

2
3 # Sample HTML with some issues
4 messy_html = " " "
5 < html >
6 < body >
7 <p > This is a paragraph with <b > bold text
8 <p > This paragraph is missing a closing tag
9 """
10
11 # Using the built - in parser
12 soup1 = BeautifulSoup ( messy_html , ’ html . parser ’)
13 print ( " html . parser result : " )
14 print ( soup1 . prettify () [:200])
15
16 # Using lxml ( faster , better at handling broken HTML )
17 try :
18 soup2 = BeautifulSoup ( messy_html , ’ lxml ’)
19 print ( " \ nlxml result : " )
20 print ( soup2 . prettify () [:200])
21 except :
22 print ( " lxml not installed " )

SUMMARY
• [Link](): Fetch web pages Key Methods: get(), post(), status code, text

• BeautifulSoup(): Parse HTML Key Methods: find(), find all(), select()

• Navigating: Move through the parse tree Key Methods: parent, children, next sibling

• Extracting: Get data from tags Key Methods: get text(), get(), [’attribute’]

• Searching: Find elements in HTML Key Methods: By tag, class, id, CSS selector

Handling Large Datasets with Chunking and Lazy Evaluation

1.1 The Challenge of Large Datasets
Definition: Working with large datasets in Python often leads to a common problem: when you
attempt to load everything into memory simultaneously, your program slows down or crashes
entirely with out-of-memory (OOM) errors.
The Memory Constraint: Traditional tools like Pandas load the entire dataset into RAM
before processing. For datasets larger than available memory, this leads to:
• Slow performance due to excessive swapping

• System crashes with memory errors

49
50

• Limited interactivity and exploration capability

1.2 Chunking: Divide and Conquer

Definition: Chunking divides a large dataset into smaller, manageable pieces (chunks) that
are processed sequentially. Each chunk fits comfortably in memory, and results from all chunks
are combined at the end.
Theoretical Principle: Instead of loading all 10 million rows at once, you load 100,000
rows at a time. Your RAM only ever holds 100,000 rows, no matter how big the file is.
When to Use Chunking:
• Performing aggregations (sum, count, average)
• Applying filtering operations
• Processing data that cannot fit entirely in RAM

1.3 Lazy Evaluation

Definition: Lazy evaluation is a strategy where expressions are not evaluated until their values
are actually needed. This contrasts with eager evaluation, where all data is loaded and processed
immediately.
Core Characteristics:
• Delayed Computation: Operations are only performed when results are requested
• Memory Efficiency: Values are generated one at a time as needed
• Infinite Sequences: Can represent potentially infinite data structures
• Composition: Operations can be chained without intermediate storage

Lazy vs. Eager Evaluation

• Memory Usage:
– Lazy Evaluation: Low (current item only)
– Eager Evaluation: High (entire dataset)
• Startup Time:
– Lazy Evaluation: Fast (immediate)
– Eager Evaluation: Slow (loading time)
• Processing Model:
– Lazy Evaluation: Streaming
– Eager Evaluation: Batch
• Random Access:
– Lazy Evaluation: Not supported
– Eager Evaluation: Full support
• Use Case:
– Lazy Evaluation: Large datasets, pipelines
– Eager Evaluation: Small to medium datasets

50
 51

1.4 Generators: The Heart of Lazy Evaluation

• Definition:

– Generator functions use the yield statement to produce values.

– Execution can be paused and resumed, continuing from the last yield when the next
value is requested.

• Key Characteristics:

– When called, a generator function does not execute immediately.

– Returns a generator object.
– Values are generated one at a time as needed.
– The function’s state is preserved between calls.

1.5 Memory Complexity

• Traditional Approach: O(n) memory, where n is the dataset size.

• Lazy/Chunking Approach: O(1) memory, regardless of dataset size.

IMPORTANT PROGRAMS
2.1 Basic Generator Function

1 def read_large_file ( file_path ) :

2 " " " Read a large file line by line using generator " " "
3 with open ( file_path , ’r ’) as file :
4 for line in file :
5 yield line . strip ()
6
7 # Usage - process large log file line by line
8 for line in read_large_file ( " huge_log . txt " ) :
9 if " ERROR " in line :
10 print ( f " Found error : { line [:100]} " )

2.2 Generator Expression vs List Comprehension

1 import sys
2
3 # List comprehension ( eager ) - loads all into memory
4 list_squares = [ x * x for x in range (1000000) ]
5 print ( f " List memory : { sys . getsizeof ( list_squares ) } bytes " )
6
7 # Generator expression ( lazy ) - produces on demand
8 gen_squares = ( x * x for x in range (1000000) )
9 print ( f " Generator memory : { sys . getsizeof ( gen_squares ) } bytes " )
10
11 # Using the generator
12 for i , square in enumerate ( gen_squares ) :
13 if i < 5:
14 print ( square , end = " " )
15 else :
16 break

51
 52

2.3 Generator Pipeline

1 def read_lines ( filename ) :

2 " " " Generator : read lines from file " " "
3 with open ( filename ) as f :
4 for line in f :
5 yield line . strip ()
6
7 def filter_pattern ( lines , pattern ) :
8 " " " Generator : filter lines containing pattern " " "
9 for line in lines :
10 if pattern in line :
11 yield line
12
13 def extract_field ( lines , field_num , delimiter = ’ , ’) :
14 " " " Generator : extract specific field " " "
15 for line in lines :
16 fields = line . split ( delimiter )
17 if len ( fields ) > field_num :
18 yield fields [ field_num ]
19
20 # Build pipeline
21 lines = read_lines ( " large_data . txt " )
22 filtered = filter_pattern ( lines , " ERROR " )
23 timestamps = extract_field ( filtered , 0 , ’ ’)
24
25 # Execute pipeline lazily
26 for ts in timestamps :
27 print ( f " Timestamp : { ts } " )
28 break # Only process first one

2.4 Basic Chunking with Pandas

1 import pandas as pd
2
3 # Read CSV file in chunks
4 chunk_size = 100000 # Number of rows per chunk
5 total_rows = 0
6 total_sum = 0
7
8 print ( f " Processing file in chunks of { chunk_size } rows ... " )
9
10 for chunk in pd . read_csv ( ’ large_file . csv ’ , chunksize = chunk_size ) :
11 # Process each chunk
12 rows_in_chunk = len ( chunk )
13 total_rows += rows_in_chunk
14 total_sum += chunk [ ’ value_column ’ ]. sum ()
15
16 print ( f " Processed chunk with { rows_in_chunk } rows . Total so far : {
total_rows } " )
17
18 average = total_sum / total_rows if total_rows > 0 else 0
19 print ( f " \ nFinal result : Average = { average :.2 f } " )

2.5 Chunking with Multiple Operations

1 import pandas as pd
2
3 def pr oc ess_large_file ( filename , chunk_size =50000) :
4 """

52
 53

5 Process large file with multiple aggregations

6 """
7 results = {
8 ’ total_rows ’: 0 ,
9 ’ sum_amount ’: 0 ,
10 ’ max_amount ’: float ( ’ - inf ’) ,
11 ’ min_amount ’: float ( ’ inf ’) ,
12 ’ unique_categories ’: set ()
13 }
14
15 for chunk in pd . read_csv ( filename , chunksize = chunk_size ) :
16 # Update aggregations
17 results [ ’ total_rows ’] += len ( chunk )
18 results [ ’ sum_amount ’] += chunk [ ’ amount ’ ]. sum ()
19 results [ ’ max_amount ’] = max ( results [ ’ max_amount ’] , chunk [ ’ amount ’ ]. max
() )
20 results [ ’ min_amount ’] = min ( results [ ’ min_amount ’] , chunk [ ’ amount ’ ]. min
() )
21
22 # Add unique values
23 results [ ’ unique_categories ’ ]. update ( chunk [ ’ category ’ ]. unique () )
24
25 print ( f " Processed chunk . Total rows : { results [ ’ total_rows ’]} " )
26
27 # Calculate final average
28 results [ ’ avg_amount ’] = results [ ’ sum_amount ’] / results [ ’ total_rows ’]
29
30 return results
31
32 # Usage
33 stats = p rocess_large_file ( ’ sales_data . csv ’)
34 print ( f " Total rows : { stats [ ’ total_rows ’]} " )
35 print ( f " Total amount : $ { stats [ ’ sum_amount ’]: ,.2 f } " )
36 print ( f " Average amount : $ { stats [ ’ avg_amount ’]:.2 f } " )
37 print ( f " Unique categories : { len ( stats [ ’ unique_categories ’]) } " )

2.6 Filtering While Reading in Chunks

1 import pandas as pd
2
3 # Read and filter in chunks
4 chunk_size = 100000
5 filtered_chunks = []
6
7 for chunk in pd . read_csv ( ’ transactions . csv ’ , chunksize = chunk_size ) :
8 # Filter each chunk
9 filtered = chunk [ chunk [ ’ amount ’] > 1000]
10 if len ( filtered ) > 0:
11 filtered_chunks . append ( filtered )
12 print ( f " Found { len ( filtered ) } rows with amount > 1000 in this chunk " )
13
14 # Combine only the filtered results
15 if filtered_chunks :
16 result = pd . concat ( filtered_chunks , ignore_index = True )
17 print ( f " \ nTotal filtered rows : { len ( result ) } " )

2.7 Column Selection with Chunking

1 import pandas as pd
2

53
 54

3 # Only load specific columns to save memory

4 columns_needed = [ ’ id ’ , ’ name ’ , ’ amount ’ , ’ date ’]
5
6 chunk_size = 100000
7 total_amount = 0
8
9 for chunk in pd . read_csv ( ’ transactions . csv ’ ,
10 chunksize = chunk_size ,
11 usecols = columns_needed ) :
12 # Work with only the columns we need
13 total_amount += chunk [ ’ amount ’ ]. sum ()
14 print ( f " Running total : $ { total_amount : ,.2 f } " )
15
16 print ( f " \ nFinal total amount : $ { total_amount : ,.2 f } " )

2.8 Custom Iterator Class

1 class FileChunkIterator :
2 """
3 Custom iterator that reads files in chunks
4 """
5 def __init__ ( self , filename , chunk_size =1024) :
6 self . filename = filename
7 self . chunk_size = chunk_size
8 self . file = None
9 self . eof = False
10
11 def __iter__ ( self ) :
12 return self
13
14 def __enter__ ( self ) :
15 self . file = open ( self . filename , ’ rb ’)
16 return self
17
18 def __exit__ ( self , * args ) :
19 if self . file :
20 self . file . close ()
21
22 def __next__ ( self ) :
23 if self . eof :
24 raise StopIteration
25
26 chunk = self . file . read ( self . chunk_size )
27 if not chunk :
28 self . eof = True
29 raise StopIteration
30
31 return chunk
32
33 # Usage
34 with FileChunkIterator ( ’ large_file . bin ’ , chunk_size =4096) as it :
35 for i , chunk in enumerate ( it ) :
36 print ( f " Chunk { i +1}: { len ( chunk ) } bytes " )
37 if i >= 4: # Stop after 5 chunks
38 break

2.9 CSV Chunk Reader with Iterator Protocol

1 import csv
2

54
 55

3 class CSVChunkReader :
4 """
5 Read CSV file in chunks using iterator protocol
6 """
7 def __init__ ( self , filename , chunk_size =1000) :
8 self . filename = filename
9 self . chunk_size = chunk_size
10 self . file = None
11 self . reader = None
12 self . header = None
13
14 def __enter__ ( self ) :
15 self . file = open ( self . filename , ’r ’)
16 self . reader = csv . reader ( self . file )
17 self . header = next ( self . reader ) # Read header
18 return self
19
20 def __exit__ ( self , * args ) :
21 if self . file :
22 self . file . close ()
23
24 def __iter__ ( self ) :
25 return self
26
27 def __next__ ( self ) :
28 " " " Return next chunk of rows " " "
29 chunk = []
30 for i in range ( self . chunk_size ) :
31 try :
32 row = next ( self . reader )
33 chunk . append ( row )
34 except StopIteration :
35 break
36 if not chunk :
37 raise StopIteration
38 return chunk
39
40 # Usage
41 with CSVChunkReader ( ’ large_data . csv ’ , chunk_size =1000) as reader :
42 for chunk_num , chunk in enumerate ( reader , 1) :
43 print ( f " Chunk { chunk_num }: { len ( chunk ) } rows " )
44 if chunk_num >= 5:
45 break

2.10 Infinite Sequence Generator

1 def f i b on acci_generator () :
2 " " " Generate Fibonacci numbers infinitely " " "
3 a, b = 0, 1
4 while True :
5 yield a
6 a, b = b, a + b
7
8 # Get first 10 Fibonacci numbers
9 fib = f i b onacci_generator ()
10 first_ten = [ next ( fib ) for _ in range (10) ]
11 print ( f " First 10 Fibonacci : { first_ten } " )
12
13 # Get next 5
14 next_five = [ next ( fib ) for _ in range (5) ]
15 print ( f " Next 5: { next_five } " )

55
 56

2.11 Using itertools for Lazy Processing

1 from itertools import islice , count

2
3 # count : infinite counter
4 counter = count ( start =1 , step =1)
5 first_10 = list ( islice ( counter , 10) )
6 print ( f " First 10 counts : { first_10 } " )
7
8 # islice : slice any iterator
9 numbers = range (1000000)
10 first_5 = list ( islice ( numbers , 5) )
11 last_5 = list ( islice ( numbers , 999995 , 1000000) )
12 print ( f " First 5: { first_5 } " )

2.12 Complete Large File Processor Class

1 import pandas as pd
2
3 class LargeCSVProcessor :
4 """
5 Complete solution for processing large CSV files
6 """
7 def __init__ ( self , filename , chunk_size =50000) :
8 self . filename = filename
9 self . chunk_size = chunk_size
10 self . total_rows = 0
11 self . chunk_count = 0
12
13 def process ( self , operations ) :
14 """
15 Apply operations to each chunk and combine results
16 """
17 results = {}
18 for chunk in pd . read_csv ( self . filename , chunksize = self . chunk_size ) :
19 self . chunk_count += 1
20 self . total_rows += len ( chunk )
21 print ( f " Processing chunk { self . chunk_count } ({ len ( chunk ) } rows ) " )
22
23 # Apply each operation to this chunk
24 for op_name , op_func in operations . items () :
25 chunk_result = op_func ( chunk )
26
27 # Combine results
28 if op_name not in results :
29 results [ op_name ] = chunk_result
30 else :
31 if isinstance ( chunk_result , ( int , float ) ) :
32 results [ op_name ] += chunk_result
33 elif isinstance ( chunk_result , set ) :
34 results [ op_name ]. update ( chunk_result )
35
36 results [ ’ total_rows ’] = self . total_rows
37 results [ ’ chunks ’] = self . chunk_count
38 return results
39
40 # Define operations
41 def sum_amount ( chunk ) :
42 return chunk [ ’ amount ’ ]. sum ()
43
44 def avg_amount ( chunk ) :
45 return chunk [ ’ amount ’ ]. mean ()

56
 57

46
47 def uniqu e_categories ( chunk ) :
48 return set ( chunk [ ’ category ’ ]. unique () )
49
50 # Usage
51 processor = LargeCSVProcessor ( ’ sales_data . csv ’ , chunk_size =100000)
52 operations = {
53 ’ total_amount ’: sum_amount ,
54 ’ unique_categories ’: unique_categories
55 }
56 results = processor . process ( operations )
57
58 print ( f " \ nProcessed { results [ ’ total_rows ’]: ,} rows in { results [ ’ chunks ’]}
chunks " )
59 print ( f " Total amount : $ { results [ ’ total_amount ’]: ,.2 f } " )
60 print ( f " Unique categories : { len ( results [ ’ unique_categories ’]) } " )

2.13 Memory-Efficient Log File Analysis

1 def analyze_log_file ( filename ) :

2 """
3 Analyze large log file line by line
4 """
5 error_count = 0
6 warning_count = 0
7 unique_ips = set ()
8
9 with open ( filename , ’r ’) as f :
10 for line_num , line in enumerate (f , 1) :
11 if " ERROR " in line :
12 error_count += 1
13 elif " WARNING " in line :
14 warning_count += 1
15
16 # Extract IP if present
17 parts = line . split ()
18 if len ( parts ) > 0 and ’. ’ in parts [0]:
19 unique_ips . add ( parts [0])
20
21 # Progress indicator
22 if line_num % 100000 == 0:
23 print ( f " Processed { line_num : ,} lines ... " )
24
25 print ( f " \ nAnalysis complete : " )
26 print ( f " Total lines : { line_num : ,} " )
27 print ( f " Errors : { error_count : ,} " )
28 print ( f " Warnings : { warning_count : ,} " )
29 print ( f " Unique IPs : { len ( unique_ips ) : ,} " )
30
31 # analyze_log_file (" webserver . log ")

2.14 JSON Streaming

1 import json
2
3 def stream_json ( filename ) :
4 """
5 Stream JSON objects from a file containing
6 one JSON object per line
7 """

57
 58

8 with open ( filename , ’r ’) as f :

9 for line in f :
10 yield json . loads ( line )
11
12 # Usage
13 for record in stream_json ( ’ large_data . jsonl ’) :
14 if record . get ( ’ status ’) == ’ active ’:
15 print ( f " Active record : { record [ ’ id ’]} " )
16 break # Process only first one

Summary of Key Programs

• Basic Generator — Line-by-line file reading; Use for log files and text processing.

• Pandas Chunking — CSV aggregation; Use for large CSV files.

• Filtering Chunks — Selective data extraction; Use when filtering large datasets.

• Custom Iterator — Binary file chunking; Use for non-text files.

• Generator Pipeline — Multi-step processing; Use for complex data transformations.

• CSVChunkReader — Custom CSV handling; Use when pandas is not available.

• LargeCSVProcessor — Complete solution; Use in production applications.

58
Chapter-3
Advanced Data Wrangling and Trans-
formation

Data Wrangling and Transformation

This section covers essential theory of data wrangling and transformation for exam preparation
and practical data analysis.

1. Definition and Importance of Data Wrangling

Data wrangling is the process of cleaning, transforming, and mapping raw data from one form
into another format suitable for downstream purposes such as analytics, machine learning, or
visualization.
The majority of real-world data is not ready for immediate analysis. Before applying algo-
rithms or statistical methods, data must be cleaned and transformed. This preparatory work
often consumes more time and effort than analysis itself—sometimes up to 80% of project time.
Data cleaning and transformation are integral parts of exploratory data analysis (EDA).
Understanding a dataset inevitably reveals inconsistencies, missing values, and structural issues
that must be addressed before meaningful analysis.
Data analysis is rarely linear; it involves an iterative cycle of cleaning, transforming, visu-
alizing, and modeling. Proficiency in data wrangling tools enables smooth navigation through
this cycle.
Robust data processing requires attention to data formats, encoding, and error handling.
Production pipelines must handle malformed records gracefully.

2. Tidy Data Concept

Tidy data is a standardized way of structuring datasets:

• Each variable forms a column

• Each observation forms a row
• Each type of observational unit forms a table

This framework ensures consistent organization, making data easier to manipulate, visualize,
and model. Most Pandas operations are designed to work seamlessly with tidy data.

3. Data Types and Structures

3.1 Basic Data Types
• Numeric: Integers and floating-point numbers

1
2

• Text/String: Character data

• Boolean: True/False values

• DateTime: Temporal data with special operations

• Categorical: Limited number of distinct values

3.2 Common Data Structures in Python

• Series: One-dimensional labeled array holding any data type

• DataFrame: Two-dimensional labeled data structure with columns of potentially different

types

• Panel: Three-dimensional data structure (less commonly used)

4. Data Cleaning
Data cleaning involves identifying and correcting errors, inconsistencies, and inaccuracies.

4.1 Common Data Quality Issues

• Missing data

• Duplicate data

• Inconsistent formatting

• Outliers

• Invalid values

• Inconsistent categorizations

4.2 Approaches to Missing Data

• Removal: Drop rows/columns with missing values

• Imputation: Fill missing values with mean, median, mode, or forward/backward fill

• Interpolation: Estimate missing values from surrounding data

• Flagging: Create indicator columns for missingness

4.3 Data Type Conversion

• String to numeric (e.g., “1,234” → 1234)

• String to datetime

• Categorical encoding

2
 3

4.4 String Cleaning

• Stripping whitespace

• Changing case

• Removing/replacing characters

• Extracting patterns with regular expressions

• Splitting and combining columns

5. Data Transformation
5.1 Filtering and Selection
• Select subsets of rows based on conditions

• Select subsets of columns

• Random sampling

5.2 Creating New Columns

• Derived from existing columns (ratios, differences)

• Applying functions

• Conditional assignments

5.3 Handling Duplicates

• Identify duplicate rows

• Remove duplicates

• Keep first or last occurrence

5.4 Discretization and Binning

• Equal-width binning

• Equal-frequency binning

• Custom bin edges

5.5 Encoding Categorical Variables

• Label encoding: integers for categories

• One-hot encoding: binary columns per category

• Dummy encoding: k-1 columns to avoid multicollinearity

5.6 Normalization and Standardization

• Normalization: scale data to [0,1] or [-1,1]

• Standardization: center data (mean=0, variance=1)

3
4

5.7 Applying Functions

• Element-wise operations

• Row-wise or column-wise operations

• Window/rolling operations

6. Combining Datasets
6.1 Concatenation
Stack datasets vertically (add rows) or horizontally (add columns). Attention to indices is
important.

6.2 Merging and Joining

Combine datasets based on keys or indices (like SQL joins):

• Inner join: Keep matching keys only

• Outer join: Keep all keys, fill missing with NaN

• Left join: Keep all keys from left

• Right join: Keep all keys from right

6.3 Key Considerations for Joins

• Key columns may differ in name

• One-to-one, one-to-many, many-to-many relationships

• Overlapping column names handled with suffixes

• Index-based vs. column-based joining

7. Reshaping Data
7.1 Wide vs. Long Format
• Wide: One row per subject, multiple columns

• Long: Multiple rows per subject, key column indicates measurement type

7.2 Pivoting (Long to Wide)

Convert long format to wide format. Useful for summary tables or modeling.

7.3 Melting (Wide to Long)

Convert wide format to long format (key-value pairs), often needed for visualization.

7.4 Stacking and Unstacking

• Stack: Columns → rows (creates MultiIndex)

• Unstack: Rows → columns

4
 5

8. Group Operations
8.1 Split-Apply-Combine Strategy
• Split: Break data into groups

• Apply: Perform operation per group

• Combine: Assemble results into new structure

8.2 Common Group Operations

• Aggregation: summary statistics per group

• Transformation: group-specific computations aligned with original data

• Filtration: discard groups based on properties

• Apply arbitrary functions

9. Handling Time Series Data

9.1 Date and Time Fundamentals
• Parsing from strings

• Time zone handling

• Date ranges and frequencies

9.2 Time Series Operations

• Resampling (changing frequency)

• Time shifts and lags

• Rolling windows

• Date-based indexing and selection

10. Common Challenges in Data Wrangling

10.1 Scale and Performance
• Large datasets that don’t fit in memory

• Computational efficiency

• Vectorization vs. iteration

10.2 Data Inconsistencies

• Naming convention differences

• Unit mismatches (metric vs. imperial)

• Different levels of aggregation

5
6

10.3 Reproducibility
• Document transformation steps

• Reusable pipelines

• Version control for data and transformations

10.4 Data Provenance

• Track data origin

• Record transformations applied

• Maintain audit trails

Summary for Quick Revision

• Data wrangling: Cleaning, transforming, mapping raw data to usable formats; consumes
majority of analysis time

• Tidy data framework: Variables as columns, observations as rows, each unit as a separate
table

• Data cleaning: Handle missing values, duplicates, inconsistent formatting, outliers, invalid
values

• Data transformation: Filtering, creating columns, binning, encoding, normalization

• Combining datasets: Concatenation and merging/joining based on keys

• Reshaping: Pivoting, melting, stacking/unstacking

• Group operations: Split-apply-combine paradigm

• Time series: Resampling, shifting, rolling windows

• Challenges: Scale, inconsistency, reproducibility, provenance

Merging and Joining DataFrames

1.1 Theory
Merging combines DataFrames based on common columns or indices using [Link](). It is
similar to SQL joins.
Types of joins:
• inner: keep only rows with matching keys in both DataFrames

• outer: keep all rows, fill missing with NaN

• left: keep all rows from left DataFrame, match from right

• right: keep all rows from right DataFrame, match from left
Key parameters:
• on: column(s) to join on (must exist in both)

• left on, right on: specify different key columns

6
 7

• left index, right index: use index as join key

• suffixes: tuple of strings to append to overlapping column names

• how: type of join (’inner’, ’outer’, ’left’, ’right’)

[Link]() joins on index by default and can also join on a column using on.

1.2 Example Programs

a) Inner join
1 import pandas as pd
2
3 df1 = pd . DataFrame ({ ’ key ’: [ ’A ’ , ’B ’ , ’C ’ , ’D ’] ,
4 ’ value1 ’: [1 , 2 , 3 , 4]})
5 df2 = pd . DataFrame ({ ’ key ’: [ ’B ’ , ’D ’ , ’E ’ , ’F ’] ,
6 ’ value2 ’: [5 , 6 , 7 , 8]})
7
8 merged_inner = pd . merge ( df1 , df2 , on = ’ key ’ , how = ’ inner ’)
9 print ( merged_inner )

Output:

key value1 value2

B 2 5
D 4 6

b) Left, right, and outer joins

1 # Left join
2 merged_left = pd . merge ( df1 , df2 , on = ’ key ’ , how = ’ left ’)
3 print ( merged_left )
4
5 # Right join
6 merged_right = pd . merge ( df1 , df2 , on = ’ key ’ , how = ’ right ’)
7 print ( merged_right )
8
9 # Outer join
10 merged_outer = pd . merge ( df1 , df2 , on = ’ key ’ , how = ’ outer ’)
11 print ( merged_outer )

Left join output:

key value1 value2

A 1 NaN
B 2 5
C 3 NaN
D 4 6

Right join output:

key value1 value2

B 2 5
D 4 6
E NaN 7
F NaN 8

Outer join output:

7
 8

key value1 value2

A 1 NaN
B 2 5
C 3 NaN
D 4 6
E NaN 7
F NaN 8

c) Merging on different column names

1 df1 = pd . DataFrame ({ ’ key1 ’: [ ’A ’ , ’B ’ , ’C ’] ,
2 ’ value ’: [1 , 2 , 3]})
3 df2 = pd . DataFrame ({ ’ key2 ’: [ ’B ’ , ’C ’ , ’D ’] ,
4 ’ score ’: [4 , 5 , 6]})
5
6 merged_diff = pd . merge ( df1 , df2 , left_on = ’ key1 ’ , right_on = ’ key2 ’)
7 print ( merged_diff )

Output:

key1 value key2 score

B 2 B 4
C 3 C 5

d) Using join method (index-based)

1 df1 = pd . DataFrame ({ ’A ’: [1 , 2 , 3]} , index =[ ’x ’ , ’y ’ , ’z ’ ])
2 df2 = pd . DataFrame ({ ’B ’: [4 , 5 , 6]} , index =[ ’x ’ , ’y ’ , ’w ’ ])
3
4 joined_left = df1 . join ( df2 , how = ’ left ’)
5 print ( joined_left )

Output:

Index A B
x 1 4
y 2 5
z 3 NaN

Reshaping DataFrames
Reshaping changes the layout of a DataFrame without altering the data itself.

• stack(): Compresses a level from the column index to the row index, creating a MultiIn-
dex Series.

• unstack(): Inverse of stack; pivots a level of the row index to the column index.

• melt(): Unpivots a DataFrame from wide format to long format; identifier columns re-
main.

• pivot(): Creates a reshaped DataFrame from long to wide format; requires unique in-
dex/column combinations. For duplicates, use pivot table().

8
 9

2.2 Example Programs

a) stack() and unstack()
1 import pandas as pd
2
3 df = pd . DataFrame ({ ’A ’: [1 , 2] , ’B ’: [3 , 4]} , index =[ ’X ’ , ’Y ’ ])
4 print ( df )
5
6 stacked = df . stack ()
7 print ( stacked )
8
9 unstacked = stacked . unstack ()
10 print ( unstacked )

Original DataFrame:

Index A B
X 1 3
Y 2 4

Stacked output:

Index Column Value

X A 1
X B 3
Y A 2
Y B 4

Unstacked output (back to original):

Index A B
X 1 3
Y 2 4

b) melt() – wide to long

1 df_wide = pd . DataFrame ({
2 ’ id ’: [1 , 2 , 3] ,
3 ’ name ’: [ ’ Alice ’ , ’ Bob ’ , ’ Charlie ’] ,
4 ’ math ’: [85 , 90 , 78] ,
5 ’ science ’: [92 , 88 , 81]
6 })
7
8 melted = pd . melt ( df_wide , id_vars =[ ’ id ’ , ’ name ’] ,
9 value_vars =[ ’ math ’ , ’ science ’] ,
10 var_name = ’ subject ’ , value_name = ’ score ’)
11 print ( melted )

Melted output:

id name subject score

1 Alice math 85
2 Bob math 90
3 Charlie math 78
1 Alice science 92
2 Bob science 88
3 Charlie science 81

c) pivot() – long to wide (unique combinations)

9
 10

1 pivoted = melted . pivot ( index = ’ id ’ , columns = ’ subject ’ , values = ’ score ’)

2 pivoted . reset_index ( inplace = True )
3 pivoted . columns . name = None
4 print ( pivoted )

Pivoted output:

id math science
1 85 92
2 90 88
3 78 81

d) pivot table() – aggregation when duplicates exist

1 df_sales2 = pd . DataFrame ({
2 ’ date ’: [ ’ 2023 -01 -01 ’ , ’ 2023 -01 -01 ’ , ’ 2023 -01 -01 ’ , ’ 2023 -01 -02 ’] ,
3 ’ product ’: [ ’A ’ , ’A ’ , ’B ’ , ’B ’] ,
4 ’ sales ’: [100 ,50 ,150 ,175]
5 })
6
7 pivot_sum = df_sales2 . pivot_table ( index = ’ date ’ ,
8 columns = ’ product ’ ,
9 values = ’ sales ’ ,
10 aggfunc = ’ sum ’)
11 print ( pivot_sum )

Pivot table with sum aggregation:

date A B
2023-01-01 150 150
2023-01-02 NaN 175

Advanced: MultiIndex and Reshaping

Theory
MultiIndex (hierarchical index) allows working with higher-dimensional data in a 2D structure.
Common operations:

• set index() / reset index(): create or flatten MultiIndex

• swaplevel(): swap levels of a MultiIndex

• sort index(): sort by index levels

• Stacking/Unstacking with MultiIndex

3.2 Example: Creating and Unstacking MultiIndex

1 import pandas as pd
2
3 # Create MultiIndex DataFrame
4 arrays = [[ ’A ’ , ’A ’ , ’B ’ , ’B ’] , [1 , 2 , 1 , 2]]
5 index = pd . MultiIndex . from_arrays ( arrays , names =[ ’ letter ’ , ’ number ’ ])
6 df = pd . DataFrame ({ ’ value ’: [10 , 20 , 30 , 40]} , index = index )
7 print ( df )
8
9 # Unstack the ’ number ’ level
10 unstacked = df . unstack ( level = ’ number ’)
11 print ( unstacked )

10
 11

Original MultiIndex DataFrame:

letter number value

A 1 10
A 2 20
B 1 30
B 2 40

Unstacked DataFrame (number level → columns):

letter 1 2
A 10 20
B 30 40

4. Summary for Quick Revision

Merging ([Link]):

• Combine DataFrames on columns.

• how: inner, outer, left, right.

• left on/right on: for different key names.

• suffixes: handle overlapping column names.

Joining ([Link]):

• Convenient index-based join.

Reshaping:

• stack(): columns → rows.

• unstack(): rows → columns.

• melt(): wide → long (unpivot).

• pivot(): long → wide (when no duplicates).

• pivot table(): long → wide with aggregation (handles duplicates).

Common Pitfalls:

• Forgetting to specify how in merge may lead to default inner join (data loss).

• Using pivot() with duplicates raises error; use pivot table().

• Not resetting index after pivoting leads to MultiIndex columns.

• Confusing id vars and value vars in melt().

11
12

Handling Missing Data

Missing data is a common issue in real-world datasets. Proper handling is crucial because most
analytical methods cannot process missing values directly.
Types of Missing Data:

• Missing Completely at Random (MCAR): The probability of missing is the same

for all observations.

• Missing at Random (MAR): The probability of missing depends on observed data.

• Missing Not at Random (MNAR): The probability of missing depends on unobserved

data.

Common Representations of Missing Data:

• NaN (Not a Number) in pandas

• None in Python

• Placeholder values like -999, 0, or empty strings

Detection Methods:

• isnull() / isna(): Detect missing values

• notnull() / notna(): Detect non-missing values

• info(): Summary of non-null counts per column

• isnull().sum(): Count missing values per column

Handling Strategies:

• Removal / Dropping: Use dropna() to remove rows or columns with missing values.
Suitable when missing data is minimal and random.

• Imputation / Filling: Use fillna() to replace missing values.

– Constant imputation: Fill with a specific value (e.g., 0, “Unknown”).

– Statistical imputation: Use mean, median, or mode.
– Forward/backward fill: Propagate previous or next value.
– Interpolation: Estimate missing values based on surrounding data.

• Flagging: Create indicator columns to mark missing values. This preserves information
about the missingness pattern.

• Advanced Methods:

– Model-based imputation using regression or machine learning.

– Multiple imputation for statistical inference.

12
 13

1.2 Example Programs

a) Detecting Missing Values
import pandas as pd
import numpy as np

# Create DataFrame with missing values

df = [Link]({
’A’: [1, 2, [Link], 4, 5],
’B’: [[Link], 2, 3, [Link], 5],
’C’: [’a’, ’b’, [Link], ’d’, ’e’],
’D’: [10, 20, 30, 40, 50]
})

print("Original DataFrame:")
print(df)

# Check for missing values

print([Link]())

# Count missing per column

print([Link]().sum())

# Total missing values

print(f"Total missing values: {[Link]().sum().sum()}")

# Rows with any missing values

print(df[[Link]().any(axis=1)])

# Rows with no missing values

print(df[[Link]().all(axis=1)])

Output Table: Original DataFrame

A B C D
1 NaN a 10
2 2 b 20
NaN 3 NaN 30
4 NaN d 40
5 5 e 50

b) Dropping Missing Values

# Drop rows with any missing value
df_drop_rows = [Link]()
print(df_drop_rows)

# Drop rows where all values are missing

df_all_missing = [Link]({
’X’: [[Link], 2, [Link]],
’Y’: [[Link], 3, [Link]],
’Z’: [[Link], 4, [Link]]
})

13
14

print(df_all_missing.dropna(how=’all’))

# Drop columns with any missing

df_drop_cols = [Link](axis=1)
print(df_drop_cols)

# Drop columns with fewer than a threshold of non-null values

df_thresh = [Link](axis=1, thresh=3)
print(df_thresh)

Output Table: Drop Rows with Any Missing Values

A B C
2 2 b
5 5 e
Output Table: Drop Rows Where All Values are Missing
X Y Z
2 3 4
Output Table: Drop Columns with Any Missing Values
C
a
b
NaN
d
e
Output Table: Keep Columns with At Least 3 Non-Null Values
A C
1 a
2 b
NaN NaN
4 d
5 e

Handling Missing Data

1. Detecting Missing Values
import pandas as pd
import numpy as np

# Create DataFrame with missing values

df = [Link]({
’A’: [1, 2, [Link], 4, 5],
’B’: [[Link], 2, 3, [Link], 5],
’C’: [’a’, ’b’, [Link], ’d’, ’e’],
’D’: [10, 20, 30, 40, 50]
})

print("Original DataFrame:")
print(df)

# Check for missing values

print("\nIs null?")

14
 15

print([Link]())

# Count missing per column

print("\nMissing count per column:")
print([Link]().sum())

# Total missing values

print(f"\nTotal missing values: {[Link]().sum().sum()}")

# Rows with any missing

print("\nRows with any missing:")
print(df[[Link]().any(axis=1)])

# Rows with all values present

print("\nRows with no missing:")
print(df[[Link]().all(axis=1)])

Output:

Original DataFrame:
A B C D
0 1.0 NaN a 10
1 2.0 2.0 b 20
2 NaN 3.0 NaN 30
3 4.0 NaN d 40
4 5.0 5.0 e 50

Is null?
A B C D
0 False True False False
1 False False False False
2 True False True False
3 False True False False
4 False False False False

Missing count per column:

A 1
B 2
C 1
D 0
dtype: int64

Total missing values: 4

Rows with any missing:

A B C D
0 1.0 NaN a 10
2 NaN 3.0 NaN 30
3 4.0 NaN d 40

Rows with no missing:

A B C D

15
16

1 2.0 2.0 b 20
4 5.0 5.0 e 50

2. Dropping Missing Values

# Drop rows with any missing value
df_drop_rows = [Link]()
print("\nDrop rows with any missing:")
print(df_drop_rows)

# Drop rows where all values are missing

df_all_missing = [Link]({
’X’: [[Link], 2, [Link]],
’Y’: [[Link], 3, [Link]],
’Z’: [[Link], 4, [Link]]
})
print("\nDrop rows where all values are missing:")
print(df_all_missing.dropna(how=’all’))

# Drop columns with any missing

df_drop_cols = [Link](axis=1)
print("\nDrop columns with any missing:")
print(df_drop_cols)

# Drop columns with fewer than a threshold of non-null values

df_thresh = [Link](axis=1, thresh=3)
print("\nKeep columns with at least 3 non-null:")
print(df_thresh)

Output:
Drop rows with any missing:
A B C D
1 2.0 2.0 b 20
4 5.0 5.0 e 50

Drop rows where all values are missing:

X Y Z
0 NaN NaN NaN
1 2.0 3.0 4.0
2 NaN NaN NaN

Drop columns with any missing:

D
0 10
1 20
2 30
3 40
4 50

Keep columns with at least 3 non-null:

A B D
0 1.0 NaN 10

16
 17

1 2.0 2.0 20
2 NaN 3.0 30
3 4.0 NaN 40
4 5.0 5.0 50

3. Filling Missing Values

df = [Link]({
’A’: [1, 2, [Link], 4, 5, [Link]],
’B’: [10, [Link], 30, [Link], 50, 60],
’C’: [100, 200, 300, 400, 500, 600]
})

# Fill with constant

df_fill_constant = [Link](0)
print("\nFill with 0:")
print(df_fill_constant)

# Fill with column-specific values

df_fill_dict = [Link]({’A’:999, ’B’:-1})
print("\nFill with column-specific values:")
print(df_fill_dict)

# Forward fill
df_ffill = [Link](method=’ffill’)
print("\nForward fill:")
print(df_ffill)

# Backward fill
df_bfill = [Link](method=’bfill’)
print("\nBackward fill:")
print(df_bfill)

# Fill with mean

df_fill_mean = [Link]()
df_fill_mean[’A’] = df[’A’].fillna(df[’A’].mean())
df_fill_mean[’B’] = df[’B’].fillna(df[’B’].mean())
print("\nFill with column means:")
print(df_fill_mean)

# Limit forward fill

df_limit = [Link](method=’ffill’, limit=1)
print("\nForward fill with limit=1:")
print(df_limit)

Output:
Fill with 0:
A B C
0 1.0 10.0 100
1 2.0 0.0 200
2 0.0 30.0 300
3 4.0 0.0 400

17
18

4 5.0 50.0 500

5 0.0 60.0 600

Fill with column-specific values:

A B C
0 1.0 10 100
1 2.0 -1 200
2 999.0 30 300
3 4.0 -1 400
4 5.0 50 500
5 999.0 60 600

Forward fill:
A B C
0 1.0 10.0 100
1 2.0 10.0 200
2 2.0 30.0 300
3 4.0 30.0 400
4 5.0 50.0 500
5 5.0 60.0 600

Backward fill:
A B C
0 1.0 10.0 100
1 2.0 30.0 200
2 4.0 30.0 300
3 4.0 50.0 400
4 5.0 50.0 500
5 NaN 60.0 600

Fill with column means:

A B C
0 1.000000 10.0 100
1 2.000000 37.5 200
2 3.000000 30.0 300
3 4.000000 37.5 400
4 5.000000 50.0 500
5 3.000000 60.0 600

Forward fill with limit=1:

A B C
0 1.0 10.0 100
1 2.0 10.0 200
2 2.0 30.0 300
3 4.0 30.0 400
4 5.0 50.0 500
5 5.0 60.0 600

4. Interpolation
df = [Link]({’value’:[1,2,[Link],[Link],5,6,[Link],8,9,10]})

18
 19

# Linear
df_linear = [Link](method=’linear’)
print("\nLinear interpolation:")
print(df_linear)

# Polynomial order 2
df_poly = [Link](method=’polynomial’, order=2)
print("\nPolynomial interpolation:")
print(df_poly)

# Time-based
dates = pd.date_range(’2023-01-01’, periods=10)
df_time = [Link]({’value’:[1,2,[Link],[Link],5,6,[Link],8,9,10]}, index=dates)
df_time_interp = df_time.interpolate(method=’time’)
print("\nTime-based interpolation:")
print(df_time_interp)

Output:

Linear interpolation:
value
0 1.0
1 2.0
2 3.0
3 4.0
4 5.0
5 6.0
6 7.0
7 8.0
8 9.0
9 10.0

Polynomial interpolation:
value
0 1.0
1 2.0
2 3.0
3 4.0
4 5.0
5 6.0
6 7.0
7 8.0
8 9.0
9 10.0

Time-based interpolation:
value
2023-01-01 1.0
2023-01-02 2.0
2023-01-03 3.0
2023-01-04 4.0
2023-01-05 5.0

19
20

2023-01-06 6.0
2023-01-07 7.0
2023-01-08 8.0
2023-01-09 9.0
2023-01-10 10.0

5. Flagging Missing Values

df = [Link]({
’age’:[25,30,[Link],35,40,[Link]],
’income’:[50000,60000,75000,[Link],82000,90000]
})

# Create flags
df[’age_missing’] = df[’age’].isnull()
df[’income_missing’] = df[’income’].isnull()

# Fill missing while preserving flags

df[’age_filled’] = df[’age’].fillna(df[’age’].mean())
df[’income_filled’] = df[’income’].fillna(df[’income’].median())
print(df)

Output:

age income age_missing income_missing age_filled income_filled

0 25.0 50000.0 False False 25.000000 50000.0
1 30.0 60000.0 False False 30.000000 60000.0
2 NaN 75000.0 True False 32.5 75000.0
3 35.0 NaN False True 35.000000 71000.0
4 40.0 82000.0 False False 40.000000 82000.0
5 NaN 90000.0 True False 32.5 90000.0

6. Complete Missing Data Handling Pipeline

df_raw = [Link]({
’id’:[1,2,3,4,5,6],
’name’:[’Alice’,’Bob’,[Link],’David’,’Eve’,’Frank’],
’age’:[25,[Link],35,[Link],45,50],
’salary’:[50000,60000,[Link],80000,[Link],100000],
’department’:[’HR’,’IT’,’IT’,[Link],’HR’,’Finance’]
})

# Analyze
missing_summary = df_raw.isnull().sum()
missing_percent = (df_raw.isnull().sum()/len(df_raw))*100

# Clean
df_clean = df_raw.dropna(subset=[’name’])
df_clean[’age’] = df_clean[’age’].fillna(df_clean[’age’].median())
df_clean[’salary_missing’] = df_clean[’salary’].isnull()
df_clean[’salary’] = df_clean[’salary’].fillna(df_clean[’salary’].mean())
df_clean[’department’] = df_clean[’department’].fillna(df_clean[’department’].mode()[0])

20
 21

print(df_clean)
print(df_clean.isnull().sum())

Output:

id name age salary department salary_missing

0 1 Alice 25.0 50000.0 HR False
1 2 Bob 37.5 60000.0 IT False
3 4 David 37.5 80000.0 HR False
4 5 Eve 45.0 71250.0 HR True
5 6 Frank 50.0 100000.0 Finance False

id 0
name 0
age 0
salary 0
department 0
salary_missing 0
dtype: int64

2. Handling Categorical Data

Theory
Categorical data represents variables with a limited, fixed number of possible values. Proper
handling is essential for statistical analysis and machine learning.

Types of Categorical Data

• Nominal: Categories with no inherent order. Examples: Colors, countries, gender, de-
partment names.

• Ordinal: Categories with a natural order. Examples: Education level (High School ¡
Bachelor’s ¡ Master’s ¡ PhD), ratings (poor, average, good).

• Binary: Special case with only two categories. Examples: Yes/No, True/False, Male/Fe-
male.

Representation in pandas
• object dtype: String values

• category dtype: Special categorical type (memory efficient)

• bool dtype: Boolean values

Common Operations
Frequency Analysis:

• value counts(): Count occurrences of each category.

• crosstab(): Create contingency tables for two categorical variables.

21
 22

Encoding Methods:

• Label Encoding: Convert categories to integers (preserves order for ordinal variables).

• One-Hot Encoding: Create binary columns for each category.

• Dummy Encoding: One-hot with k − 1 columns to avoid multicollinearity.

• Ordinal Encoding: Map categories to ordered integers.

Creating Categorical Variables:

• cut(): Bin continuous data into categories.

• qcut(): Bin continuous data based on quantiles.

• astype(’category’): Convert a column to categorical type.

Handling New Categories:

• Plan for unseen categories in test data.

• Use a special “other” category or implement appropriate handling.

Factorizing:

• factorize(): Encode categorical data as integer categories.

2.2 Example Programs

a) Creating and Exploring Categorical Data

1 import pandas as pd
2 import numpy as np
3
4 # Create DataFrame with categorical columns
5 df = pd . DataFrame ({
6 ’ name ’: [ ’ Alice ’ , ’ Bob ’ , ’ Charlie ’ , ’ David ’ , ’ Eve ’] ,
7 ’ department ’: [ ’ HR ’ , ’ IT ’ , ’ IT ’ , ’ Finance ’ , ’ HR ’] ,
8 ’ rating ’: [ ’ Good ’ , ’ Excellent ’ , ’ Average ’ , ’ Good ’ , ’ Poor ’] ,
9 ’ gender ’: [ ’F ’ , ’M ’ , ’M ’ , ’M ’ , ’F ’] ,
10 ’ salary ’: [50000 , 80000 , 75000 , 90000 , 55000]
11 })
12
13 print ( df )

Output:

name department rating gender salary

Alice HR Good F 50000
Bob IT Excellent M 80000
Charlie IT Average M 75000
David Finance Good M 90000
Eve HR Poor F 55000

Table 1: Original DataFrame

22
 23

1 # Convert to categorical type

2 df [ ’ department ’] = df [ ’ department ’ ]. astype ( ’ category ’)
3 df [ ’ rating ’] = df [ ’ rating ’ ]. astype ( ’ category ’)
4 df [ ’ gender ’] = df [ ’ gender ’ ]. astype ( ’ category ’)
5
6 print ( df . dtypes )

Output:

Column Data Type

name object
department category
rating category
gender category
salary int64

Table 2: Data types after converting to categorical

1 print ( " Department categories : " , df [ ’ department ’ ]. cat . categories )

2 print ( " Rating categories : " , df [ ’ rating ’ ]. cat . categories )
3 print ( " Gender categories : " , df [ ’ gender ’ ]. cat . categories )

Output:

Column Categories
department Finance, HR, IT
rating Average, Excellent, Good, Poor
gender F, M

Table 3: Categorical column categories

b) Value Counts and Frequency Analysis

1 df = pd . DataFrame ({
2 ’ department ’: [ ’ HR ’ , ’ IT ’ , ’ IT ’ , ’ Finance ’ , ’ HR ’ , ’ IT ’ , ’ HR ’ , ’ Finance ’] ,
3 ’ rating ’: [ ’ Good ’ , ’ Excellent ’ , ’ Average ’ , ’ Good ’ , ’ Poor ’ , ’ Excellent ’ , ’
Good ’ , ’ Average ’]
4 })
5
6 # Department counts
7 print ( df [ ’ department ’ ]. value_counts () )
8
9 # Rating counts
10 print ( df [ ’ rating ’ ]. value_counts () )
11
12 # Rating counts normalized
13 print ( df [ ’ rating ’ ]. value_counts ( normalize = True ) )
14
15 # Cross - tabulation
16 print ( pd . crosstab ( df [ ’ department ’] , df [ ’ rating ’ ]) )
17
18 # Cross - tab with margins
19 print ( pd . crosstab ( df [ ’ department ’] , df [ ’ rating ’] , margins = True ) )

Output:

23
 24

Rating Count
Department Count
Good 3
HR 3
Excellent 2
IT 3
Average 2
Finance 2
Poor 1

Department Average Excellent Good Poor

Finance 1 0 1 0
HR 0 0 2 1
IT 1 2 0 0

Table 4: Cross-tabulation Department vs Rating

Department Average Excellent Good Poor All

Finance 1 0 1 0 2
HR 0 0 2 1 3
IT 1 2 0 0 3
All 2 2 3 1 8

Table 5: Cross-tab with totals

c) Label Encoding (Ordinal Encoding)

1 df = pd . DataFrame ({
2 ’ name ’: [ ’ Alice ’ , ’ Bob ’ , ’ Charlie ’ , ’ David ’ , ’ Eve ’] ,
3 ’ rating ’: [ ’ Good ’ , ’ Excellent ’ , ’ Average ’ , ’ Good ’ , ’ Poor ’]
4 })
5
6 # Manual mapping
7 rating_map = { ’ Poor ’:1 , ’ Average ’:2 , ’ Good ’:3 , ’ Excellent ’ :4}
8 df [ ’ rating_encoded ’] = df [ ’ rating ’ ]. map ( rating_map )
9
10 # Factorize
11 df [ ’ ratin g_factorized ’] , unique = pd . factorize ( df [ ’ rating ’ ])
12 print ( df )
13 print ( " Unique values : " , unique . tolist () )

Output:

name rating rating encoded rating factorized

Alice Good 3 0
Bob Excellent 4 1
Charlie Average 2 2
David Good 3 0
Eve Poor 1 3

Table 6: Label encoding outputs

d) One-Hot Encoding

1 df = pd . DataFrame ({
2 ’ id ’: [1 ,2 ,3 ,4 ,5] ,
3 ’ department ’: [ ’ HR ’ , ’ IT ’ , ’ IT ’ , ’ Finance ’ , ’ HR ’] ,
4 ’ gender ’: [ ’F ’ , ’M ’ , ’M ’ , ’M ’ , ’F ’]

24
 25

5 })
6
7 df_encoded = pd . get_dummies ( df , columns =[ ’ department ’ , ’ gender ’ ])
8 df_dummy = pd . get_dummies ( df , columns =[ ’ department ’ , ’ gender ’] , drop_first = True )
9 df_prefix = pd . get_dummies ( df , columns =[ ’ department ’] , prefix = ’ dept ’)

Output (one-hot encoded):

id department Finance department HR department IT gender F gender M

1 0 1 0 1 0
2 0 0 1 0 1
3 0 0 1 0 1
4 1 0 0 0 1
5 0 1 0 1 0

Table 7: One-hot encoding output

3. Handling Time-Series Data

3.1 Theory
Time-series data consists of observations indexed by time. Proper handling requires specialized
techniques and understanding of temporal structures.
Key Concepts:

• Timestamp: Specific point in time, e.g., 2023-03-11 09:00:00.

• Time Period: Interval between two timestamps.

• Time Delta: Duration or difference between times.

• Frequency: Regularity of observations (daily, monthly, hourly, etc.).

Common Operations:

1. Date/Time Parsing
• [Link] datetime(): Convert strings to datetime objects.

• [Link] range(): Create sequences of dates with specified frequency.

2. Indexing and Selection

• DatetimeIndex as row index for efficient selection.

• Partial string indexing: e.g., df[’2023-01’] selects all January 2023 data.

• Slicing with dates: df[’2023-01-01’:’2023-01-31’].

3. Resampling
• Change frequency of time series: upsampling or downsampling.

• Aggregation functions (mean, sum, etc.) for downsampling.

• Interpolation methods for upsampling.

25
 26

4. Shifting and Lags

• shift(): Move data forward or backward in time.

• Create lagged features for predictive modeling.

5. Rolling Windows
• rolling(): Apply functions over a sliding time window.

• Use for moving averages, rolling statistics, or feature engineering.

6. Time Zone Handling

• tz localize(): Assign timezone to naive datetime.

• tz convert(): Convert datetime between timezones.

7. Date Components Extraction

• Extract year, month, day, weekday, hour, minute, etc., from datetime for analysis or
modeling.

3.2 Example Programs

a) Creating Datetime Objects and Indexes

1 import pandas as pd
2 import numpy as np
3 from datetime import datetime
4
5 # Create Timestamp from string
6 ts1 = pd . Timestamp ( ’ 2023 -01 -15 ’)
7 ts2 = pd . Timestamp ( ’ 2023 -12 -31 ’)
8
9 # Create from datetime object
10 dt = datetime (2023 , 6 , 15 , 14 , 30)
11 ts3 = pd . Timestamp ( dt )
12
13 # Create date range
14 dates = pd . date_range ( start = ’ 2023 -01 -01 ’ , end = ’ 2023 -01 -10 ’ , freq = ’D ’)
15
16 # Create with specific frequency
17 month_starts = pd . date_range ( start = ’ 2023 -01 -01 ’ , periods =5 , freq = ’ MS ’) # Month
start
18 business_days = pd . date_range ( start = ’ 2023 -01 -01 ’ , periods =10 , freq = ’B ’) #
Business days
19
20 # Create DataFrame with datetime index
21 df = pd . DataFrame ({ ’ value ’: np . random . randn ( len ( dates ) ) } , index = dates )

b) Parsing and Converting to Datetime

1 # Various date string formats

2 df = pd . DataFrame ({
3 ’ date_str1 ’: [ ’ 2023 -01 -15 ’ , ’ 2023 -02 -20 ’ , ’ 2023 -03 -25 ’] ,
4 ’ date_str2 ’: [ ’ 15/01/2023 ’ , ’ 20/02/2023 ’ , ’ 25/03/2023 ’] ,
5 ’ date_str3 ’: [ ’ Jan 15 , 2023 ’ , ’ Feb 20 , 2023 ’ , ’ Mar 25 , 2023 ’] ,
6 ’ value ’: [100 , 200 , 300]

26
 27

7 })
8
9 # Parse different formats
10 df [ ’ date1 ’] = pd . to_datetime ( df [ ’ date_str1 ’ ])
11 df [ ’ date2 ’] = pd . to_datetime ( df [ ’ date_str2 ’] , format = ’% d /% m /% Y ’)
12 df [ ’ date3 ’] = pd . to_datetime ( df [ ’ date_str3 ’ ])
13
14 # Handle errors
15 df_bad = pd . DataFrame ({
16 ’ date ’: [ ’ 2023 -01 -15 ’ , ’ invalid_date ’ , ’ 2023 -03 -25 ’] ,
17 ’ value ’: [100 , 200 , 300]
18 })
19 df_bad [ ’ date_coerce ’] = pd . to_datetime ( df_bad [ ’ date ’] , errors = ’ coerce ’)

c) Indexing and Selecting Time Series Data

1 # Create time series data

2 dates = pd . date_range ( ’ 2023 -01 -01 ’ , periods =100 , freq = ’D ’)
3 df = pd . DataFrame ({
4 ’ value ’: np . random . randn (100) . cumsum () ,
5 ’ volume ’: np . random . randint (100 , 1000 , 100)
6 } , index = dates )
7
8 # Select by year
9 df [ ’ 2023 ’]
10
11 # Select by year - month
12 df [ ’ 2023 -01 ’]
13
14 # Select by date range
15 df [ ’ 2023 -01 -15 ’: ’ 2023 -01 -25 ’]
16
17 # Select with loc on specific date
18 df . loc [ ’ 2023 -01 -15 ’]
19
20 # Extract date components
21 df [ ’ year ’] = df . index . year
22 df [ ’ month ’] = df . index . month
23 df [ ’ day ’] = df . index . day
24 df [ ’ dayofweek ’] = df . index . dayofweek # Monday =0 , Sunday =6
25 df [ ’ quarter ’] = df . index . quarter
26 print ( " \ nWith date components : " )
27 print ( df [[ ’ value ’ , ’ year ’ , ’ month ’ , ’ day ’ , ’ dayofweek ’ , ’ quarter ’ ]]. head () )

d) Resampling Time Series Data

1 import pandas as pd
2 import numpy as np
3
4 # Create high - frequency data ( hourly )
5 dates = pd . date_range ( ’ 2023 -01 -01 00:00 ’ , periods =24*7 , freq = ’H ’) # One week of
hourly data
6 df = pd . DataFrame ({ ’ value ’: np . random . randn ( len ( dates ) ) . cumsum () } , index = dates )
7
8 # Downsampling : Hourly to daily
9 daily_mean = df . resample ( ’D ’) . mean ()
10 daily_sum = df . resample ( ’D ’) . sum ()
11 daily_max = df . resample ( ’D ’) . max ()
12
13 # Downsampling with multiple aggregations
14 daily_stats = df . resample ( ’D ’) . agg ([ ’ mean ’ , ’ std ’ , ’ min ’ , ’ max ’ ])

27
 28

15
16 # Create lower - frequency data ( daily )
17 dates_daily = pd . date_range ( ’ 2023 -01 -01 ’ , periods =30 , freq = ’D ’)
18 df_daily = pd . DataFrame ({ ’ value ’: np . random . randn (30) . cumsum () } , index =
dates_daily )
19
20 # Upsampling : Daily to hourly ( interpolation )
21 h o u rl y _ in t erpolated = df_daily . resample ( ’H ’) . interpolate ( method = ’ linear ’)
22
23 # Upsampling with forward fill
24 hourly_ffill = df_daily . resample ( ’H ’) . ffill ()
25
26 # Custom resampling functions
27 def custom_range ( x ) :
28 return x . max () - x . min ()
29
30 daily_range = df . resample ( ’D ’) . apply ( custom_range )

e) Shifting and Lagging

1 import pandas as pd
2 import numpy as np
3
4 # Create simple time series
5 dates = pd . date_range ( ’ 2023 -01 -01 ’ , periods =10 , freq = ’D ’)
6 df = pd . DataFrame ({ ’ value ’: [10 , 12 , 15 , 14 , 18 , 20 , 22 , 21 , 25 , 28]} , index =
dates )
7
8 # Shift forward ( lag ) and backward ( lead )
9 df [ ’ lag_1 ’] = df [ ’ value ’ ]. shift (1)
10 df [ ’ lag_2 ’] = df [ ’ value ’ ]. shift (2)
11 df [ ’ lag_ -1 ’] = df [ ’ value ’ ]. shift ( -1)
12
13 # Create features for time series prediction
14 df [ ’ value_lag1 ’] = df [ ’ value ’ ]. shift (1)
15 df [ ’ value_lag2 ’] = df [ ’ value ’ ]. shift (2)
16 df [ ’ value_lag3 ’] = df [ ’ value ’ ]. shift (3)
17 df [ ’ value_lead1 ’] = df [ ’ value ’ ]. shift ( -1)
18
19 # Calculate differences and percentage change
20 df [ ’ diff_1 ’] = df [ ’ value ’ ]. diff (1)
21 df [ ’ diff_2 ’] = df [ ’ value ’ ]. diff (2)
22 df [ ’ pct_change ’] = df [ ’ value ’ ]. pct_change ()

f ) Rolling Window Operations

1 import pandas as pd
2 import numpy as np
3
4 # Create time series data
5 dates = pd . date_range ( ’ 2023 -01 -01 ’ , periods =30 , freq = ’D ’)
6 df = pd . DataFrame ({ ’ value ’: np . random . randn (30) . cumsum () } , index = dates )
7
8 print ( " Original data : " )
9 print ( df . head (10) )
10
11 # Rolling mean ( moving average )
12 df [ ’ rolling_mean_3 ’] = df [ ’ value ’ ]. rolling ( window =3) . mean ()
13 df [ ’ rolling_mean_7 ’] = df [ ’ value ’ ]. rolling ( window =7) . mean ()
14
15 print ( " \ nWith rolling means : " )

28
 29

16 print ( df . head (10) )

17
18 # Other rolling statistics
19 df [ ’ rolling_std_3 ’] = df [ ’ value ’ ]. rolling ( window =3) . std ()
20 df [ ’ rolling_max_3 ’] = df [ ’ value ’ ]. rolling ( window =3) . max ()
21 df [ ’ rolling_min_3 ’] = df [ ’ value ’ ]. rolling ( window =3) . min ()
22 df [ ’ rolling_sum_3 ’] = df [ ’ value ’ ]. rolling ( window =3) . sum ()
23
24 print ( " \ nWith various rolling statistics : " )
25 print ( df . head (10) )
26
27 # Rolling with custom function
28 df [ ’ rolling_range_3 ’] = df [ ’ value ’ ]. rolling ( window =3) . apply ( lambda x : x . max () -
x . min () )
29 print ( " \ nRolling range ( max - min ) with window 3: " )
30 print ( df [[ ’ value ’ , ’ rolling_range_3 ’ ]]. head (10) )
31
32 # Centered rolling window
33 df [ ’ centered_ma_3 ’] = df [ ’ value ’ ]. rolling ( window =3 , center = True ) . mean ()
34 print ( " \ nCentered moving average ( window 3) : " )
35 print ( df [[ ’ value ’ , ’ centered_ma_3 ’ ]]. head (10) )
36
37 # Rolling with min_periods
38 df [ ’ rolling_min5_3 ’] = df [ ’ value ’ ]. rolling ( window =3 , min_periods =2) . mean ()
39 print ( " \ nRolling mean with min_periods =2: " )
40 print ( df [[ ’ value ’ , ’ rolling_min5_3 ’ ]]. head (10) )

g) Time Zone Handling

1 import pandas as pd
2 import numpy as np
3
4 # Create time series without timezone
5 dates = pd . date_range ( ’ 2023 -01 -01 09:00 ’ , periods =5 , freq = ’H ’)
6 df = pd . DataFrame ({ ’ value ’: [100 , 110 , 120 , 130 , 140]} , index = dates )
7
8 print ( " Original ( naive ) : " )
9 print ( df )
10
11 # Localize to a timezone ( assign timezone )
12 df_tz = df . copy ()
13 df_tz . index = df_tz . index . tz_localize ( ’ UTC ’)
14 print ( " \ nLocalized to UTC : " )
15 print ( df_tz )
16
17 # Convert to another timezone
18 df_ny = df_tz . tz_convert ( ’ America / New_York ’)
19 print ( " \ nConverted to New York time : " )
20 print ( df_ny )
21
22 df_london = df_tz . tz_convert ( ’ Europe / London ’)
23 print ( " \ nConverted to London time : " )
24 print ( df_london )
25
26 # Create with timezone directly
27 dates_tz = pd . date_range ( ’ 2023 -01 -01 09:00 ’ , periods =5 , freq = ’H ’ , tz = ’ Asia / Tokyo
’)
28 df_tokyo = pd . DataFrame ({ ’ value ’: [200 , 210 , 220 , 230 , 240]} , index = dates_tz )
29 print ( " \ nCreated with Tokyo timezone : " )
30 print ( df_tokyo )

29
 30

h) Extracting Date Components

1 import pandas as pd
2 import numpy as np
3
4 # Create time series
5 dates = pd . date_range ( ’ 2023 -01 -01 ’ , periods =365 , freq = ’D ’)
6 df = pd . DataFrame ({ ’ value ’: np . random . randn (365) . cumsum () } , index = dates )
7
8 # Extract various components
9 df [ ’ year ’] = df . index . year
10 df [ ’ month ’] = df . index . month
11 df [ ’ day ’] = df . index . day
12 df [ ’ dayofweek ’] = df . index . dayofweek # Monday =0 , Sunday =6
13 df [ ’ day_name ’] = df . index . day_name ()
14 df [ ’ month_name ’] = df . index . month_name ()
15 df [ ’ quarter ’] = df . index . quarter
16 df [ ’ week ’] = df . index . isocalendar () . week
17 df [ ’ dayofyear ’] = df . index . dayofyear
18 df [ ’ is_month_end ’] = df . index . is_month_end
19 df [ ’ is_month_start ’] = df . index . is_month_start
20 df [ ’ is_quarter_end ’] = df . index . is_quarter_end
21
22 print ( " Date components ( first 10 rows ) : " )
23 print ( df . head (10) )
24
25 # Group by date components
26 monthly_avg = df . groupby ( ’ month ’) [ ’ value ’ ]. mean ()
27 print ( " \ nMonthly averages : " )
28 print ( monthly_avg )
29
30 # Count by day of week
31 dow_counts = df . groupby ( ’ day_name ’) . size ()
32 print ( " \ nCount by day of week : " )
33 print ( dow_counts )

i) Complete Time Series Pipeline

1 import pandas as pd
2 import numpy as np
3 from datetime import datetime , timedelta
4
5 # Generate sample sales data
6 np . random . seed (42)
7 dates = pd . date_range ( ’ 2023 -01 -01 ’ , ’ 2023 -12 -31 ’ , freq = ’D ’)
8 df = pd . DataFrame ({
9 ’ date ’: dates ,
10 ’ sales ’: np . random . poisson ( lam =100 , size = len ( dates ) ) +
11 np . sin ( np . arange ( len ( dates ) ) * 2 * np . pi / 365) * 20 + # Seasonal
12 np . random . randn ( len ( dates ) ) * 10 # Noise
13 })
14
15 print ( " Raw sales data : " )
16 print ( df . head () )
17
18 # Step 1: Set datetime as index
19 df [ ’ date ’] = pd . to_datetime ( df [ ’ date ’ ])
20 df . set_index ( ’ date ’ , inplace = True )
21
22 # Step 2: Check for missing dates
23 date_range = pd . date_range ( start = df . index . min () , end = df . index . max () , freq = ’D ’)
24 if len ( df ) < len ( date_range ) :

30
 31

25 print ( f " Missing { len ( date_range ) - len ( df ) } dates " )

26 df = df . reindex ( date_range )
27
28 # Step 3: Handle any missing values
29 if df [ ’ sales ’ ]. isnull () . any () :
30 print ( f " Filling { df [ ’ sales ’]. isnull () . sum () } missing values " )
31 df [ ’ sales ’] = df [ ’ sales ’ ]. interpolate ( method = ’ time ’)
32
33 # Step 4: Extract time features
34 df [ ’ year ’] = df . index . year
35 df [ ’ month ’] = df . index . month
36 df [ ’ day ’] = df . index . day
37 df [ ’ dayofweek ’] = df . index . dayofweek
38 df [ ’ day_name ’] = df . index . day_name ()
39 df [ ’ quarter ’] = df . index . quarter
40 df [ ’ weekend ’] = ( df . index . dayofweek >= 5) . astype ( int )
41
42 # Step 5: Create lag features for prediction
43 for lag in [1 , 2 , 3 , 7 , 14 , 30]:
44 df [ f ’ sales_lag_ { lag } ’] = df [ ’ sales ’ ]. shift ( lag )
45
46 # Step 6: Create rolling statistics
47 df [ ’ rolling_mean_7 ’] = df [ ’ sales ’ ]. rolling ( window =7) . mean ()
48 df [ ’ rolling_std_7 ’] = df [ ’ sales ’ ]. rolling ( window =7) . std ()
49 df [ ’ rolling_mean_30 ’] = df [ ’ sales ’ ]. rolling ( window =30) . mean ()
50
51 # Step 7: Create differences
52 df [ ’ sales_diff_1 ’] = df [ ’ sales ’ ]. diff (1)
53 df [ ’ sales_diff_7 ’] = df [ ’ sales ’ ]. diff (7)
54 df [ ’ sales_pct_change ’] = df [ ’ sales ’ ]. pct_change ()
55
56 # Step 8: Resample to different frequencies
57 weekly_sales = df [ ’ sales ’ ]. resample ( ’W ’) . sum ()
58 monthly_sales = df [ ’ sales ’ ]. resample ( ’M ’) . mean ()
59 quarterly_sales = df [ ’ sales ’ ]. resample ( ’Q ’) . sum ()
60
61 print ( " \ nProcessed data ( first 10 rows ) : " )
62 print ( df . head (10) )
63 print ( f " \ nColumns in processed data : { df . columns . tolist () } " )
64 print ( f " Shape : { df . shape } " )
65
66 print ( " \ nWeekly sales ( first 5 weeks ) : " )
67 print ( weekly_sales . head () )
68
69 print ( " \ nMonthly average sales : " )
70 print ( monthly_sales . head () )

Quick Revision Summary

1. Missing Data
• Detection: isnull(), notnull(), info(), isnull().sum()

• Removal: dropna() with parameters how, thresh, subset

• Imputation: fillna() with constant, method, or statistics

• Interpolation: interpolate() with various methods

• Flagging: Create indicator columns for missingness

31
32

2. Categorical Data
• Types:

– Nominal (no order)

– Ordinal (has order)
– Binary

• Analysis: value counts(), crosstab(), groupby()

• Encoding:

– Label encoding (map)

– One-hot (get dummies)
– Dummy (drop first)

• Binning: cut() (equal-width), qcut() (equal-frequency)

• Categorical dtype: astype(’category’) for memory efficiency

3. Time-Series Data
• Creation: to datetime(), date range(), Timestamp

• Indexing: DatetimeIndex, partial string indexing, slicing

• Resampling: resample() with aggregation (downsampling) or interpolation (upsam-

pling)

• Shifting: shift() for lags/leads, diff() for differences, pct change() for percentage
change

• Rolling: rolling() for window statistics (mean, std, min, max, sum, custom)

• Components: Extract year, month, day, dayofweek, quarter, etc.

• Time zones: tz localize(), tz convert()

32
Feature Transformation, Scaling, and
Encoding

1. Feature Transformation
1.1 Theory
Feature transformation involves applying mathematical functions to modify the distribution or
relationships of features. It is often used to:

• Make data more normally distributed (many statistical models assume normality)

• Stabilize variance

• Linearize relationships

• Reduce skewness

1.2 Common Transformations

• Log Transformation: log(x) — Useful for right-skewed positive data.
√
• Square Root Transformation: x — Useful for count data or moderately skewed data.

• Reciprocal Transformation: 1/x — Useful when values represent rates or ratios.

• Box-Cox Transformation: (xλ − 1)/λ (if λ ̸= 0) — General power transformation for

positive data.

• Yeo-Johnson Transformation: Similar to Box-Cox but can handle negative values —

Useful for data with zeros or negative values.

• Power Transformation: xλ — General power transformation for adjusting skewness or

variance.

1.3 Key Considerations

• Log transformation requires positive values (add a constant if zeros exist)

• Box-Cox automatically finds the optimal λ

• Transformations should be applied to training data, then the same parameters used on
test/validation data

33
34

Common Transformations with Examples

1. Log Transformation
Formula:
y = log(x) or y = log(1 + x) (x ≥ 0)
Purpose: Compresses the right tail of a distribution; suitable for right-skewed positive data.
Use log(1 + x) if zeros are present.
Example: x = [1, 2, 5, 10, 100]

log(x) = [0.0000, 0.6931, 1.6094, 2.3026, 4.6052]

2. Square Root Transformation

Formula: √
y= x
Purpose: Reduces moderate right skewness; often used for count data.
Example: x = [1, 4, 9, 16, 25]
√
x = [1.0000, 2.0000, 3.0000, 4.0000, 5.0000]

3. Reciprocal Transformation
Formula:
1
y=
x
Purpose: Useful for rates or ratios; strongly reduces right skewness, sensitive to small
values.
Example: x = [1, 2, 5, 10, 20]
1
= [1.0000, 0.5000, 0.2000, 0.1000, 0.0500]
x

4. Box–Cox Transformation
Formula: 
λ
 x − 1 , λ ̸= 0

y(λ) = λ

log(x), λ = 0

Purpose: Power transformation to make data more normal; requires x > 0.

Example: x = [1, 2, 3, 4, 5], λ = 2

x2 − 1
y= = [0.0, 1.5, 4.0, 7.5, 12.0]
2

5. Yeo–Johnson Transformation
Formula: 
λ
 (x + 1) − 1 ,

 x ≥ 0, λ ̸= 0


 λ
log(x + 1), x ≥ 0, λ = 0

y(λ) =
 (−x + 1)2−λ − 1

− , x < 0, λ ̸= 2
2−λ




− log(−x + 1), x < 0, λ = 2

34
 35

Purpose: Extension of Box–Cox; handles zero and negative values.

Example: x = [−2, −1, 0, 1, 2], λ = 0.5

y ≈ [−2.7975, −1.2189, 0, 0.8284, 1.4641]

6. Power Transformation
Formula:
y = xλ

Purpose: General transformation to reduce skewness or stabilize variance.

Example: x = [1, 2, 3, 4, 5], λ = 2

x2 = [1, 4, 9, 16, 25]

Summary of Transformations

Transformation Formula Example Input Example Output

Log log(x)
√ 10 2.3026
Square Root x 16 4
Reciprocal 1/x 5 0.2
Box–Cox (xλ − 1)/λ (λ ̸= 0) 4, λ = 2 7.5
Yeo–Johnson Piecewise -1, λ = 0.5 -1.2189
Power xλ 3, λ = 2 9

Table 8: Common feature transformations with formulas and examples

a) Log Transformation

1 import pandas as pd
2 import numpy as np
3 import matplotlib . pyplot as plt
4
5 # Create skewed data
6 np . random . seed (42)
7 data = np . random . exponential ( scale =2 , size =1000)
8 df = pd . DataFrame ({ ’ original ’: data })
9
10 # Apply log transformation
11 df [ ’ log ’] = np . log ( df [ ’ original ’ ])
12 df [ ’ log1p ’] = np . log1p ( df [ ’ original ’ ]) # handles zeros
13
14 # Compare skewness
15 print ( " Original - Skewness : " , df [ ’ original ’ ]. skew () )
16 print ( " Log - Skewness : " , df [ ’ log ’ ]. skew () )
17 print ( " Log1p - Skewness : " , df [ ’ log1p ’ ]. skew () )
18
19 # Optional visualization
20 # df . hist ( bins =50 , figsize =(10 ,4) )
21 # plt . show ()
Listing 1: Log Transformation on Skewed Data

35
 36

b) Square Root Transformation

1 import pandas as pd
2 import numpy as np
3
4 # Poisson count data
5 np . random . seed (42)
6 counts = np . random . poisson ( lam =5 , size =1000)
7 df = pd . DataFrame ({ ’ counts ’: counts })
8
9 # Square root transformation
10 df [ ’ sqrt ’] = np . sqrt ( df [ ’ counts ’ ])
11
12 print ( " Original - Skewness : " , df [ ’ counts ’ ]. skew () )
13 print ( " Sqrt - Skewness : " , df [ ’ sqrt ’ ]. skew () )
Listing 2: Square Root Transformation for Count Data

c) Box-Cox Transformation

1 import pandas as pd
2 import numpy as np
3 from scipy import stats
4
5 # Generate right - skewed data
6 np . random . seed (42)
7 data = np . random . gamma ( shape =2 , scale =2 , size =1000)
8 df = pd . DataFrame ({ ’ original ’: data })
9
10 # Box - Cox transformation ( add 1 if zeros present )
11 df [ ’ boxcox ’] , fitted_lambda = stats . boxcox ( df [ ’ original ’] + 1)
12
13 print ( f " Optimal lambda : { fitted_lambda :.4 f } " )
14 print ( " Original - Skewness : " , df [ ’ original ’ ]. skew () )
15 print ( " Box - Cox - Skewness : " , df [ ’ boxcox ’ ]. skew () )
Listing 3: Box-Cox Transformation

d) Yeo-Johnson Transformation (handles negative values)

1 import pandas as pd
2 import numpy as np
3 from sklearn . preprocessing import PowerTransformer
4
5 # Data with negative values
6 np . random . seed (42)
7 data = np . random . randn (1000) * 2 + 1
8 df = pd . DataFrame ({ ’ original ’: data })
9
10 # Yeo - Johnson via PowerTransformer
11 pt = PowerTransformer ( method = ’yeo - johnson ’)
12 df [ ’ yeojohnson ’] = pt . fit_transform ( df [[ ’ original ’ ]])
13
14 print ( " Original - Skewness : " , df [ ’ original ’ ]. skew () )
15 print ( " Yeo - Johnson - Skewness : " , df [ ’ yeojohnson ’ ]. skew () )
16 print ( " Lambda used : " , pt . lambdas_ [0])
Listing 4: Yeo-Johnson Transformation using scikit-learn

36
 37

e) Applying Transformation in a Pipeline

1 import pandas as pd
2 import numpy as np
3 from sklearn . model_selection import train_test_split
4 from sklearn . preprocessing import PowerTransformer
5 from sklearn . linear_model import LinearRegression
6 from sklearn . pipeline import make_pipeline
7 from sklearn . metrics import r2_score
8
9 # Generate synthetic data
10 np . random . seed (42)
11 X = np . random . exponential ( size =(1000 , 1) )
12 y = 2 * np . log ( X ) . ravel () + np . random . randn (1000) * 0.5
13
14 X_train , X_test , y_train , y_test = train_test_split (X , y , test_size =0.2)
15
16 # Pipeline : transform then model
17 pipeline = make_pipeline (
18 PowerTransformer ( method = ’box - cox ’) , # automatically finds lambda
19 LinearRegression ()
20 )
21 pipeline . fit ( X_train , y_train )
22 y_pred = pipeline . predict ( X_test )
23 print ( f " R score : { r2_score ( y_test , y_pred ) :.4 f } " )
Listing 5: Feature Transformation in ML Pipeline

Feature Scaling
Theory
Feature scaling adjusts the range or distribution of numerical features. It is essential for algo-
rithms that rely on distance measures, such as k-NN, SVM, or PCA, and for methods that use
gradient descent, like neural networks and linear models with regularization.

Common Scaling Techniques

• Standardization (Z-score): Transform data by subtracting the mean and dividing by
the standard deviation:
x−µ
z=
σ
Resulting distribution has mean 0 and variance 1. Useful when data is roughly Gaussian
and for many machine learning algorithms.

• Min-Max Scaling: Scale data to a fixed range, typically [0,1], using:

x − xmin
x′ =
xmax − xmin
Useful when a bounded range is needed, e.g., for neural networks.

• Robust Scaling: Center data by the median and scale according to the interquartile
range (IQR):
x − median
x′ =
IQR
Robust to outliers and effective when data contains extreme values.

37
 38

• MaxAbs Scaling: Scale data by the maximum absolute value:

x
x′ =
max(|x|)

Produces a range [-1,1] and is suitable for sparse data.

Important Considerations

• Always fit the scaler on the training data only, then apply the transformation to validation
or test sets.

• Standardization does not guarantee a fixed range, whereas Min-Max scaling produces an
exact bounded range.

Example Programs
a) Standardization (Z-score)

1 import pandas as pd
2 import numpy as np
3 from sklearn . preprocessing import StandardScaler
4
5 # Sample data
6 df = pd . DataFrame ({
7 ’ age ’: [25 , 30 , 35 , 40 , 45] ,
8 ’ income ’: [50000 , 60000 , 75000 , 90000 , 120000]
9 })
10
11 scaler = StandardScaler ()
12 scaled = scaler . fit_transform ( df )
13 df_scaled = pd . DataFrame ( scaled , columns = df . columns )
14
15 print ( " Original mean :\ n " , df . mean () )
16 print ( " Original std :\ n " , df . std () )
17 print ( " \ nAfter scaling mean :\ n " , df_scaled . mean () )
18 print ( " After scaling std :\ n " , df_scaled . std () )

b) Min-Max Scaling

1 import pandas as pd
2 from sklearn . preprocessing import MinMaxScaler
3
4 df = pd . DataFrame ({
5 ’ age ’: [25 , 30 , 35 , 40 , 45] ,
6 ’ income ’: [50000 , 60000 , 75000 , 90000 , 120000]
7 })
8
9 scaler = MinMaxScaler ()
10 scaled = scaler . fit_transform ( df )
11 df_scaled = pd . DataFrame ( scaled , columns = df . columns )
12
13 print ( " Original min :\ n " , df . min () )
14 print ( " Original max :\ n " , df . max () )
15 print ( " \ nAfter scaling min :\ n " , df_scaled . min () )
16 print ( " After scaling max :\ n " , df_scaled . max () )

38
 39

c) Robust Scaling (Median and IQR)

1 import pandas as pd
2 from sklearn . preprocessing import RobustScaler
3
4 # Include an outlier
5 df = pd . DataFrame ({
6 ’ value ’: [10 , 12 , 11 , 100 , 13 , 14 , 12] # 100 is outlier
7 })
8
9 scaler = RobustScaler ()
10 scaled = scaler . fit_transform ( df )
11 df [ ’ scaled ’] = scaled
12
13 print ( " Original :\ n " , df )
14 print ( " \ nMedian : " , df [ ’ value ’ ]. median () )
15 print ( " IQR : " , df [ ’ value ’ ]. quantile (0.75) - df [ ’ value ’ ]. quantile (0.25) )
16 print ( " Scaled median : " , df [ ’ scaled ’ ]. median () )

d) Applying Scaling in a Pipeline

1 import pandas as pd
2 import numpy as np
3 from sklearn . model_selection import train_test_split
4 from sklearn . preprocessing import StandardScaler
5 from sklearn . svm import SVR
6 from sklearn . pipeline import make_pipeline
7 from sklearn . metrics import mean_squared_error
8
9 # Generate data with different scales
10 np . random . seed (42)
11 X = np . random . rand (100 , 2) * [100 , 0.1] # one feature large scale , one small
12 y = X [: , 0] * 2 + X [: , 1] * 100 + np . random . randn (100) * 5
13
14 X_train , X_test , y_train , y_test = train_test_split (X , y , test_size =0.2)
15
16 # SVM without scaling
17 svm_raw = SVR ()
18 svm_raw . fit ( X_train , y_train )
19 y_pred_raw = svm_raw . predict ( X_test )
20 mse_raw = mean_squared_error ( y_test , y_pred_raw )
21
22 # SVM with scaling
23 pipeline = make_pipeline ( StandardScaler () , SVR () )
24 pipeline . fit ( X_train , y_train )
25 y_pred_scaled = pipeline . predict ( X_test )
26 mse_scaled = mean_squared_error ( y_test , y_pred_scaled )
27
28 print ( f " MSE without scaling : { mse_raw :.4 f } " )
29 print ( f " MSE with scaling : { mse_scaled :.4 f } " )

e) Comparison of Scaling Methods

1 import pandas as pd
2 import numpy as np
3 from sklearn . preprocessing import StandardScaler , MinMaxScaler , RobustScaler ,
MaxAbsScaler
4
5 # Create data with outliers
6 np . random . seed (42)

39
 40

7 data = np . concatenate ([ np . random . normal (0 , 1 , 50) , np . random . normal (10 , 1 , 5) ])

# outliers
8 df = pd . DataFrame ({ ’ original ’: data })
9
10 scalers = {
11 ’ Standard ’: StandardScaler () ,
12 ’ MinMax ’: MinMaxScaler () ,
13 ’ Robust ’: RobustScaler () ,
14 ’ MaxAbs ’: MaxAbsScaler ()
15 }
16
17 for name , scaler in scalers . items () :
18 scaled = scaler . fit_transform ( df [[ ’ original ’ ]])
19 df [ name ] = scaled
20
21 print ( df . describe () )

Example Programs
a) Standardization (Z-score)

1 import pandas as pd
2 import numpy as np
3 from sklearn . preprocessing import StandardScaler
4
5 # Sample data
6 df = pd . DataFrame ({
7 ’ age ’: [25 , 30 , 35 , 40 , 45] ,
8 ’ income ’: [50000 , 60000 , 75000 , 90000 , 120000]
9 })
10
11 scaler = StandardScaler ()
12 scaled = scaler . fit_transform ( df )
13 df_scaled = pd . DataFrame ( scaled , columns = df . columns )
14
15 print ( " Original mean :\ n " , df . mean () )
16 print ( " Original std :\ n " , df . std () )
17 print ( " \ nAfter scaling mean :\ n " , df_scaled . mean () )
18 print ( " After scaling std :\ n " , df_scaled . std () )

b) Min-Max Scaling

1 import pandas as pd
2 from sklearn . preprocessing import MinMaxScaler
3
4 df = pd . DataFrame ({
5 ’ age ’: [25 , 30 , 35 , 40 , 45] ,
6 ’ income ’: [50000 , 60000 , 75000 , 90000 , 120000]
7 })
8
9 scaler = MinMaxScaler ()
10 scaled = scaler . fit_transform ( df )
11 df_scaled = pd . DataFrame ( scaled , columns = df . columns )
12
13 print ( " Original min :\ n " , df . min () )
14 print ( " Original max :\ n " , df . max () )
15 print ( " \ nAfter scaling min :\ n " , df_scaled . min () )
16 print ( " After scaling max :\ n " , df_scaled . max () )

40
 41

c) Robust Scaling (Median and IQR)

1 import pandas as pd
2 from sklearn . preprocessing import RobustScaler
3
4 # Include an outlier
5 df = pd . DataFrame ({
6 ’ value ’: [10 , 12 , 11 , 100 , 13 , 14 , 12] # 100 is outlier
7 })
8
9 scaler = RobustScaler ()
10 scaled = scaler . fit_transform ( df )
11 df [ ’ scaled ’] = scaled
12
13 print ( " Original :\ n " , df )
14 print ( " \ nMedian : " , df [ ’ value ’ ]. median () )
15 print ( " IQR : " , df [ ’ value ’ ]. quantile (0.75) - df [ ’ value ’ ]. quantile (0.25) )
16 print ( " Scaled median : " , df [ ’ scaled ’ ]. median () )

d) Applying Scaling in a Pipeline

1 import pandas as pd
2 import numpy as np
3 from sklearn . model_selection import train_test_split
4 from sklearn . preprocessing import StandardScaler
5 from sklearn . svm import SVR
6 from sklearn . pipeline import make_pipeline
7 from sklearn . metrics import mean_squared_error
8
9 # Generate data with different scales
10 np . random . seed (42)
11 X = np . random . rand (100 , 2) * [100 , 0.1] # one feature large scale , one small
12 y = X [: , 0] * 2 + X [: , 1] * 100 + np . random . randn (100) * 5
13
14 X_train , X_test , y_train , y_test = train_test_split (X , y , test_size =0.2)
15
16 # SVM without scaling
17 svm_raw = SVR ()
18 svm_raw . fit ( X_train , y_train )
19 y_pred_raw = svm_raw . predict ( X_test )
20 mse_raw = mean_squared_error ( y_test , y_pred_raw )
21
22 # SVM with scaling
23 pipeline = make_pipeline ( StandardScaler () , SVR () )
24 pipeline . fit ( X_train , y_train )
25 y_pred_scaled = pipeline . predict ( X_test )
26 mse_scaled = mean_squared_error ( y_test , y_pred_scaled )
27
28 print ( f " MSE without scaling : { mse_raw :.4 f } " )
29 print ( f " MSE with scaling : { mse_scaled :.4 f } " )

e) Comparison of Scaling Methods

1 import pandas as pd
2 import numpy as np
3 from sklearn . preprocessing import StandardScaler , MinMaxScaler , RobustScaler ,
MaxAbsScaler
4
5 # Create data with outliers
6 np . random . seed (42)

41
 42

7 data = np . concatenate ([ np . random . normal (0 , 1 , 50) , np . random . normal (10 , 1 , 5) ])

# outliers
8 df = pd . DataFrame ({ ’ original ’: data })
9
10 scalers = {
11 ’ Standard ’: StandardScaler () ,
12 ’ MinMax ’: MinMaxScaler () ,
13 ’ Robust ’: RobustScaler () ,
14 ’ MaxAbs ’: MaxAbsScaler ()
15 }
16
17 for name , scaler in scalers . items () :
18 scaled = scaler . fit_transform ( df [[ ’ original ’ ]])
19 df [ name ] = scaled
20
21 print ( df . describe () )

Feature Encoding
Theory

Feature encoding converts categorical data into numerical form suitable for machine learning
algorithms. Proper encoding ensures models can interpret categorical variables correctly.
Types of Categorical Encoding:

• Label Encoding: Assigns a unique integer to each category. Useful for ordinal data or
tree-based models.

• One-Hot Encoding: Creates binary columns for each category. Suitable for nominal
data and linear models.

• Dummy Encoding: Similar to one-hot but uses k − 1 columns to avoid multicollinearity.

Used when intercept is present.

• Ordinal Encoding: Maps categories to ordered integers. Useful for ordinal data with a
known order.

• Frequency Encoding: Replaces each category with its frequency. Effective for high
cardinality features and tree models.

• Target Encoding: Replaces each category with the mean of the target variable. Suitable
for classification/regression tasks with high cardinality features. Must use cross-validation
to avoid overfitting.

• Binary Encoding: Converts categories to binary digits and splits them into separate
columns. Memory-efficient for high-cardinality features.

Key Considerations:

• One-hot encoding can create many columns for features with high cardinality.

• Target encoding can lead to overfitting; always use cross-validation.

• For tree-based models, simple label encoding may be sufficient.

42
 43

Feature Encoding Examples with Input and Output

a) Label Encoding

1 import pandas as pd
2 from sklearn . preprocessing import LabelEncoder
3
4 df = pd . DataFrame ({ ’ color ’: [ ’ red ’ , ’ blue ’ , ’ green ’ , ’ blue ’ , ’ red ’ ]})
5
6 # Using sklearn
7 le = LabelEncoder ()
8 df [ ’ color_encoded ’] = le . fit_transform ( df [ ’ color ’ ])
9
10 # Using pandas factorize
11 df [ ’ color_factor ’] = pd . factorize ( df [ ’ color ’ ]) [0]
12
13 print ( df )
14 print ( " Classes : " , le . classes_ )
15 # Output :
16 # color color_encoded color_factor
17 # 0 red 2 2
18 # 1 blue 0 0
19 # 2 green 1 1
20 # 3 blue 0 0
21 # 4 red 2 2
22 # Classes : [ ’ blue ’ ’ green ’ ’ red ’]

b) One-Hot Encoding

1 df = pd . DataFrame ({
2 ’ id ’: [1 ,2 ,3 ,4] ,
3 ’ color ’: [ ’ red ’ , ’ blue ’ , ’ green ’ , ’ blue ’] ,
4 ’ size ’: [ ’S ’ , ’M ’ , ’L ’ , ’M ’]
5 })
6
7 df_encoded = pd . get_dummies ( df , columns =[ ’ color ’ , ’ size ’ ])
8 df_dummy = pd . get_dummies ( df , columns =[ ’ color ’ , ’ size ’] , drop_first = True )
9
10 print ( df_encoded )
11 print ( df_dummy )
12 # Output :
13 # df_encoded :
14 # id color_blue color_green color_red size_L size_M size_S
15 # 0 1 0 0 1 0 0 1
16 # 1 2 1 0 0 0 1 0
17 # 2 3 0 1 0 1 0 0
18 # 3 4 1 0 0 0 1 0
19 # df_dummy :
20 # id color_green color_red color_blue size_M size_S
21 # 0 1 0 1 0 0 1
22 # 1 2 0 0 1 1 0
23 # 2 3 1 0 0 0 0
24 # 3 4 0 0 1 1 0

c) Ordinal Encoding

1 df = pd . DataFrame ({ ’ education ’: [ ’ Bachelor ’ , ’ Master ’ , ’ PhD ’ , ’ Bachelor ’ , ’ Master ’ , ’

PhD ’ ]})
2 education_order = { ’ Bachelor ’:1 , ’ Master ’:2 , ’ PhD ’ :3}
3 df [ ’ educa tion_ordinal ’] = df [ ’ education ’ ]. map ( education_order )

43
 44

4 print ( df )
5 # Output :
6 # education education_ordinal
7 # 0 Bachelor 1
8 # 1 Master 2
9 # 2 PhD 3
10 # 3 Bachelor 1
11 # 4 Master 2
12 # 5 PhD 3

d) Frequency Encoding

1 df = pd . DataFrame ({ ’ city ’: [ ’ NYC ’ , ’ LA ’ , ’ NYC ’ , ’ Chicago ’ , ’ LA ’ , ’ NYC ’ , ’ Chicago ’ , ’

Boston ’ ]})
2 freq = df [ ’ city ’ ]. value_counts ( normalize = True )
3 df [ ’ city_freq ’] = df [ ’ city ’ ]. map ( freq )
4 print ( df )
5 # Output :
6 # city city_freq
7 # 0 NYC 0.375
8 # 1 LA 0.250
9 # 2 NYC 0.375
10 # 3 Chicago 0.250
11 # 4 LA 0.250
12 # 5 NYC 0.375
13 # 6 Chicago 0.250
14 # 7 Boston 0.125

e) Target Encoding (Mean Encoding)

1 df = pd . DataFrame ({
2 ’ category ’: [ ’A ’ , ’B ’ , ’A ’ , ’C ’ , ’B ’ , ’A ’ , ’C ’ , ’B ’ , ’A ’ , ’C ’] ,
3 ’ target ’: [1 ,0 ,1 ,0 ,1 ,0 ,1 ,0 ,1 ,1]
4 })
5
6 # Simple target encoding
7 target_map = df . groupby ( ’ category ’) [ ’ target ’ ]. mean ()
8 df [ ’ t a r g e t _e ncode d_si mple ’] = df [ ’ category ’ ]. map ( target_map )
9
10 # Cross - validated target encoding
11 from sklearn . model_selection import KFold
12 import numpy as np
13 df [ ’ targe t_encoded_cv ’] = np . nan
14 kf = KFold ( n_splits =3 , shuffle = True , random_state =42)
15 for train_idx , val_idx in kf . split ( df ) :
16 train , val = df . iloc [ train_idx ] , df . iloc [ val_idx ]
17 means = train . groupby ( ’ category ’) [ ’ target ’ ]. mean ()
18 df . loc [ val_idx , ’ target_encoded_cv ’] = val [ ’ category ’ ]. map ( means )
19 df [ ’ targe t_encoded_cv ’] = df [ ’ target_encoded_cv ’ ]. fillna ( df [ ’ target ’ ]. mean () )
20
21 print ( df )
22 # Output : df with columns category , target , target_encoded_simple ,
targ et_encoded_cv

f ) Binary Encoding (Manual)

1 df = pd . DataFrame ({ ’ color ’: [ ’ red ’ , ’ blue ’ , ’ green ’ , ’ yellow ’ ]})

2 unique = df [ ’ color ’ ]. unique ()
3 mapping = { cat : i for i , cat in enumerate ( unique ) }

44
 45

4 df [ ’ code ’] = df [ ’ color ’ ]. map ( mapping )

5 max_bits = len ( bin ( len ( unique ) -1) ) -2
6 for i in range ( max_bits ) :
7 df [ f ’ bit_ { i } ’] = ( df [ ’ code ’] >> i ) & 1
8 print ( df )
9 # Output :
10 # color code bit_0 bit_1
11 # 0 red 0 0 0
12 # 1 blue 1 1 0
13 # 2 green 2 0 1
14 # 3 yellow 3 1 1

g) Combining Encodings in a Pipeline

1 from sklearn . compose import ColumnTransformer

2 from sklearn . preprocessing import OneHotEncoder , StandardScaler
3 from sklearn . pipeline import Pipeline
4 from sklearn . ensemble import Ra nd omF or es tC las si fi er
5
6 # Assume X_train , X_test , y_train , y_test exist
7 preprocessor = ColumnTransformer (
8 transformers =[
9 ( ’ num ’ , StandardScaler () , [ ’ height ’ ]) ,
10 ( ’ cat ’ , OneHotEncoder ( drop = ’ first ’) , [ ’ color ’ , ’ size ’ ])
11 ])
12
13 pipeline = Pipeline ( steps =[( ’ preprocessor ’ , preprocessor ) ,
14 ( ’ classifier ’ , Ra nd omF or es tCl as si fi er ( random_state
=42) ) ])
15 pipeline . fit ( X_train , y_train )
16 score = pipeline . score ( X_test , y_test )
17 print ( f " Accuracy : { score :.4 f } " )
18 # Output : Accuracy ~0.80 -0.95 depending on random seed

Complete Preprocessing Pipeline Example

1 import pandas as pd
2 import numpy as np
3 from sklearn . model_selection import train_test_split
4 from sklearn . preprocessing import StandardScaler , OneHotEncoder
5 from sklearn . compose import ColumnTransformer
6 from sklearn . pipeline import Pipeline
7 from sklearn . ensemble import Rand omFor estR egre ssor
8 from sklearn . metrics import mean_squared_error
9
10 # Create a mixed dataset
11 np . random . seed (42)
12 n_samples = 1000
13 df = pd . DataFrame ({
14 ’ age ’: np . random . randint (18 , 70 , n_samples ) ,
15 ’ income ’: np . random . exponential (50 , n_samples ) * 1000 ,
16 ’ education ’: np . random . choice ([ ’ High School ’ , ’ Bachelor ’ , ’ Master ’ , ’ PhD ’] ,
n_samples ) ,
17 ’ city ’: np . random . choice ([ ’ NYC ’ , ’ LA ’ , ’ Chicago ’ , ’ Houston ’ , ’ Phoenix ’] ,
n_samples ) ,
18 ’ purchases ’: np . random . poisson (5 , n_samples ) * 10
19 })
20
21 # Target : function of features + noise
22 df [ ’ spending ’] = ( df [ ’ age ’ ]*2 +
23 df [ ’ income ’ ]*0.01 +

45
 46

24 ( df [ ’ education ’ ]. map ({ ’ High School ’:1 , ’ Bachelor ’:2 , ’ Master ’:3 ,

’ PhD ’ :4}) *50) +
25 np . random . randn ( n_samples ) *100)
26
27 # Separate features and target
28 X = df . drop ( ’ spending ’ , axis =1)
29 y = df [ ’ spending ’]
30
31 X_train , X_test , y_train , y_test = train_test_split (X , y , test_size =0.2 ,
random_state =42)
32
33 # Preprocessing
34 numeric_features = [ ’ age ’ , ’ income ’ , ’ purchases ’]
35 c a t e g o r i c a l_features = [ ’ education ’ , ’ city ’]
36
37 n u m er i c _t r ansformer = Pipeline ( steps =[( ’ scaler ’ , StandardScaler () ) ])
38
39 c a t e g o r i c a l _ tr a ns fo r me r = Pipeline ( steps =[( ’ onehot ’ , OneHotEncoder ( drop = ’ first ’ ,
sparse_output = False ) ) ])
40
41 preprocessor = ColumnTransformer (
42 transformers =[
43 ( ’ num ’ , numeric_transformer , numeric_features ) ,
44 ( ’ cat ’ , categorical_transformer , categorical_features )
45 ])
46
47 pipeline = Pipeline ( steps =[
48 ( ’ preprocessor ’ , preprocessor ) ,
49 ( ’ model ’ , Rand omFo restR egre ssor ( n_estimators =100 , random_state =42) )
50 ])
51
52 # Train
53 pipeline . fit ( X_train , y_train )
54
55 # Predict and evaluate
56 y_pred = pipeline . predict ( X_test )
57 rmse = np . sqrt ( mean_squared_error ( y_test , y_pred ) )
58 print ( f " RMSE : { rmse :.2 f } " )
59 # Sample Output : RMSE : 267.43 ( may vary slightly )
60
61 # Inspect transformed feature names
62 preprocessor . fit ( X_train )
63 cat_f eature_names = preprocessor . named_transformers_ [ ’ cat ’ ]. named_steps [ ’ onehot ’
]. g e t _ fe ature _nam es_ou t ( categorical_features )
64 all_f eature_names = numeric_features + list ( cat_feature_names )
65 print ( f " Number of features : { len ( all_feature_names ) } " )
66 # Sample Output : Number of features : 17 ( depends on categories )

Memory Optimization and Efficient Data Processing

1. Core Theory and Concepts
1.1 Python Memory Management Fundamentals
Python manages memory through:

• Reference counting: Each object tracks references; deallocated when count reaches zero.

• Cyclic garbage collector: Handles reference cycles.

Memory Profiling Tools: tracemalloc, memory profiler, psutil.

46
 47

1.2 Views vs. Copies

1 # View ( pointer , minimal memory )

2 test_list = [1 , 2 , 3]
3 view_list = test_list
4 view_list [1] = 4
5 print ( test_list ) # Output : [1 , 4 , 3]
6
7 # Copy ( independent , full memory )
8 test_list = [1 , 2 , 3]
9 copy_list = test_list . copy ()
10 copy_list [1] = 4
11 print ( test_list ) # Output : [1 , 2 , 3]

1.3 Vectorization and Broadcasting

NumPy and Pandas achieve high performance through vectorization—applying operations to
entire arrays without explicit Python loops. This leverages:

• C implementation: Core functions are written in compiled C, avoiding Python inter-

preter overhead
• SIMD instructions: Modern processors apply one instruction across multiple variables
simultaneously
• Broadcasting: Operations are applied element-wise without looping

Performance comparison (sum of 1 million elements):

1 # Python loop ( slow )
2 total = 0
3 for i in range (1 _000_000 ) :
4 total += i
5
6 # NumPy vectorized ( fast )
7 import numpy as np
8 total = np . sum ( np . arange (1 _000_000 ) )

1.4 Chunking and Out-of-Core Processing

When datasets exceed available RAM, chunking processes data in manageable pieces:

• Pandas chunking: Read CSV files in chunks using chunksize parameter

• Out-of-core computing: Streams data from disk without loading entirely into memory

2.1 Use Generators and Iterators for Streaming

Process one item at a time rather than loading entire datasets:
1 # Anti - pattern : loads all into memory
2 def fetch_all_pages ( urls ) :
3 results = []
4 for url in urls :
5 results . append ( fetch_page ( url ) )
6 return results
7
8 # Optimized : generator yields one at a time
9 def fe tc h_ pages_stream ( urls ) :
10 for url in urls :
11 yield fetch_page ( url ) # yields without storing all

47
 48

2.2 Choose Data Structures Wisely

• For numeric data:

– Use NumPy arrays instead of Python lists for large numeric collections.
– Arrays store data contiguously in memory, enabling vectorized operations.

• For objects with many instances:

– Using slots can reduce memory overhead by 30-50% because it prevents each
object from creating a dict .

1 # Without __slots__ ( each object has __dict__ overhead )

2 class Product :
3 def __init__ ( self , id , name , price ) :
4 self . id = id
5 self . name = name
6 self . price = price
7
8 # With __slots__ ( reduces memory 30 -50%)
9 class ProductOptimized :
10 __slots__ = ( ’ id ’ , ’ name ’ , ’ price ’)
11 def __init__ ( self , id , name , price ) :
12 self . id = id
13 self . name = name
14 self . price = price

3.1 Optimize Data Types

Pandas often uses unnecessarily large data types. Downcasting numeric columns can dramati-
cally reduce memory usage.
1 import pandas as pd
2 import numpy as np
3
4 def reduce_mem_usage ( df ) :
5 " " " Iterate through columns and downcast numeric types " " "
6 start_mem = df . memory_usage () . sum () / 1024**2
7 print ( f " Initial memory : { start_mem :.2 f } MB " )
8
9 for col in df . columns :
10 col_type = df [ col ]. dtype
11
12 if col_type != object :
13 c_min = df [ col ]. min ()
14 c_max = df [ col ]. max ()
15
16 # Test if column can be integer
17 if str ( col_type ) [:3] == ’ int ’:
18 if c_min >= 0:
19 if c_max < 255:
20 df [ col ] = df [ col ]. astype ( np . uint8 )
21 elif c_max < 65535:
22 df [ col ] = df [ col ]. astype ( np . uint16 )
23 elif c_max < 4294967295:
24 df [ col ] = df [ col ]. astype ( np . uint32 )
25 else :
26 df [ col ] = df [ col ]. astype ( np . uint64 )
27 else :
28 if c_min > np . iinfo ( np . int8 ) . min and c_max < np . iinfo ( np .
int8 ) . max :

48
 49

29 df [ col ] = df [ col ]. astype ( np . int8 )

30 elif c_min > np . iinfo ( np . int16 ) . min and c_max < np . iinfo ( np .
int16 ) . max :
31 df [ col ] = df [ col ]. astype ( np . int16 )
32 elif c_min > np . iinfo ( np . int32 ) . min and c_max < np . iinfo ( np .
int32 ) . max :
33 df [ col ] = df [ col ]. astype ( np . int32 )
34 else :
35 df [ col ] = df [ col ]. astype ( np . float32 )
36
37 end_mem = df . memory_usage () . sum () / 1024**2
38 print ( f " Final memory : { end_mem :.2 f } MB " )
39 print ( f " Reduced by : {(1 - end_mem / start_mem ) *100:.1 f }% " )
40 return df

3.2 Use Categorical Data for Low-Cardinality Columns

Converting string columns with few unique values to category type saves significant memory.
1 # Before : object dtype ( expensive )
2 df [ ’ country ’] = [ ’ USA ’ , ’ Canada ’ , ’ USA ’ , ’ Mexico ’ , ...] # ~50 MB
3
4 # After : categorical ( much smaller )
5 df [ ’ country ’] = df [ ’ country ’ ]. astype ( ’ category ’) # ~5 MB

3.3 Read Data in Chunks

For large CSV files, never load the entire file at once. Process in manageable chunks.
1 # Bad : loads entire file into memory
2 df = pd . read_csv ( ’ huge_file . csv ’)
3
4 # Good : read and process in chunks
5 chunk_size = 100000
6 for chunk in pd . read_csv ( ’ huge_file . csv ’ , chunksize = chunk_size ) :
7 # Process each chunk
8 process_chunk ( chunk )
9 # Memory is freed after each iteration

3.4 Drop Unnecessary Columns Early

Drop unneeded columns as early as possible to save memory.
1 # Load only the needed columns
2 df = pd . read_csv ( ’ data . csv ’ , usecols =[ ’ id ’ , ’ value ’ , ’ timestamp ’ ])
3
4 # Or drop columns after loading
5 df . drop ([ ’ temporary_column ’ , ’ debug_info ’] , axis =1 , inplace = True )

3.5 Use Method Chaining with inplace=True

Avoid creating multiple intermediate DataFrames; use method chaining and inplace operations.
1 # Anti - pattern : creates multiple intermediate DataFrames
2 df_temp = df [ df [ ’ value ’] > 0]
3 df_temp = df_temp . dropna ()
4 df_temp = df_temp . rename ( columns ={ ’ old ’: ’ new ’ })
5
6 # Optimized : method chaining with inplace operations
7 df_filtered = df [ df [ ’ value ’] > 0]. copy ()
8 df_filtered . dropna ( inplace = True )
9 df_filtered . rename ( columns ={ ’ old ’: ’ new ’} , inplace = True )

49
 50

4.1 Vectorization Over Loops

NumPy vectorization avoids Python loops and is significantly faster.
1 import numpy as np
2 import time
3
4 # Create large array
5 arr = np . random . randn (10 _000_000 )
6
7 # Slow : Python loop
8 start = time . time ()
9 result = [ np . tanh ( x ) for x in arr ]
10 print ( f " Loop time : { time . time () - start :.2 f } s " )
11
12 # Fast : vectorized
13 start = time . time ()
14 result = np . tanh ( arr )
15 print ( f " Vectorized time : { time . time () - start :.2 f } s " )
16 # Typically 50 -100 x faster

4.2 Use Views Instead of Copies

Using views avoids unnecessary memory usage compared to creating copies.
1 # Creates a copy ( memory intensive )
2 arr_copy = arr [ arr > 0]
3
4 # Creates a view ( memory efficient )
5 mask = arr > 0
6 arr_view = arr [ mask ] # Still a copy in this case , but ...
7
8 # Better : use boolean indexing without creating intermediate copy
9 result = arr [ arr > 0]. mean () # No persistent copy

4.3 In-Place Operations

In-place operations modify existing arrays, avoiding new memory allocation.
1 # Creates new array ( memory intensive )
2 arr = arr + 1
3
4 # Modifies in place ( memory efficient )
5 arr += 1

5.1 Parallel Pandas with Multiprocessing

For CPU-bound operations, split work across multiple cores.
1 import pandas as pd
2 import numpy as np
3 import multiprocessing as mp
4
5 def parallel_apply ( df , func , cores = None ) :
6 " " " Apply function to DataFrame in parallel " " "
7 if cores is None :
8 cores = mp . cpu_count ()
9
10 # Split DataFrame
11 df_split = np . array_split ( df , cores )
12
13 # Create pool

50
 51

14 pool = mp . Pool ( cores )

15
16 # Process in parallel
17 df = pd . concat ( pool . map ( func , df_split ) )
18
19 pool . close ()
20 pool . join ()
21 return df
22
23 # Example usage
24 def process_chunk ( chunk ) :
25 chunk [ ’ new_col ’] = chunk [ ’ value ’] * 2
26 return chunk
27
28 result = parallel_apply ( large_df , process_chunk )

5.2 Dask for Out-of-Core Computing

For datasets too large for memory, Dask provides parallel, out-of-core DataFrames.
1 import dask . dataframe as dd
2
3 # Read large CSV with Dask ( doesn ’t load into memory )
4 ddf = dd . read_csv ( ’ huge_file . csv ’)
5
6 # Operations are lazy
7 result = ddf . groupby ( ’ category ’) . value . mean ()
8
9 # Compute only when needed
10 final_result = result . compute ()

6.1 Using memory profiler

1 # Install : pip install memory_profiler

2 from memory_profiler import profile
3 import numpy as np
4
5 @profile
6 def m e m o r y _ i n t e n s i v e _ f u n c t i on () :
7 large_list = [ i for i in range (1 _000_000 ) ]
8 large_array = np . arange (1 _000_000 )
9 del large_list
10 return large_array . mean ()
11
12 # Run the script :
13 # python -m memory_profiler script . py

6.2 Using tracemalloc for Detailed Tracking

1 import tracemalloc
2 import pandas as pd
3
4 tracemalloc . start ()
5
6 # Your code here
7 df = pd . read_csv ( ’ large_file . csv ’)
8 result = df . groupby ( ’ category ’) . sum ()
9
10 current , peak = tracemalloc . get_traced_memory ()
11 print ( f " Current memory : { current / 1024**2:.1 f } MB " )

51
 52

12 print ( f " Peak memory : { peak / 1024**2:.1 f } MB " )

13
14 tracemalloc . stop ()

6.3 Using psutil for Process Monitoring

1 import psutil
2 import os
3
4 def memory_usage () :
5 process = psutil . Process ( os . getpid () )
6 mem = process . memory_info () . rss / 1024**2 # MB
7 return mem
8
9 print ( f " Memory usage : { memory_usage () :.2 f } MB " )

7. Complete Optimization Pipeline Example

1 import pandas as pd
2 import numpy as np
3 import psutil
4 import tracemalloc
5
6 def op ti mi zed_pipeline ( filepath ) :
7 """
8 Complete optimized data processing pipeline
9 """
10 tracemalloc . start ()
11 start_mem = psutil . Process () . memory_info () . rss / 1024**2
12
13 # 1. Read only necessary columns in chunks
14 chunk_size = 50000
15 chunks = []
16
17 for chunk in pd . read_csv ( filepath , chunksize = chunk_size ,
18 usecols =[ ’ id ’ , ’ category ’ , ’ value ’ , ’ timestamp ’ ]) :
19
20 # 2. Optimize data types per chunk
21 chunk [ ’ category ’] = chunk [ ’ category ’ ]. astype ( ’ category ’)
22
23 # 3. Downcast numeric columns
24 if chunk [ ’ value ’ ]. dtype == ’ float64 ’:
25 chunk [ ’ value ’] = pd . to_numeric ( chunk [ ’ value ’] , downcast = ’ float ’)
26
27 # 4. Process chunk ( vectorized operations )
28 chunk [ ’ value_squared ’] = chunk [ ’ value ’] ** 2
29 chunk [ ’ value_log ’] = np . log1p ( chunk [ ’ value ’ ])
30
31 # 5. Filter early
32 chunk = chunk [ chunk [ ’ value ’] > 0]
33
34 chunks . append ( chunk )
35
36 # 6. Clear any temporary variables
37 del chunk
38
39 # 7. Combine results
40 final_df = pd . concat ( chunks , ignore_index = True )
41
42 # 8. Final aggregations
43 result = final_df . groupby ( ’ category ’) . agg ({

52
 53

44 ’ value ’: [ ’ mean ’ , ’ std ’ , ’ count ’] ,

45 ’ value_squared ’: ’ mean ’
46 })
47
48 # Memory reporting
49 current , peak = tracemalloc . get_traced_memory ()
50 end_mem = psutil . Process () . memory_info () . rss / 1024**2
51
52 print ( f " Start memory : { start_mem :.1 f } MB " )
53 print ( f " End memory : { end_mem :.1 f } MB " )
54 print ( f " Peak memory ( tracemalloc ) : { peak / 1024**2:.1 f } MB " )
55 print ( f " Memory saved : {(1 - end_mem / start_mem ) *100:.1 f }% " )
56
57 tracemalloc . stop ()
58 return result

Summary for Quick Revision

• Memory Optimization Principles

– Measure first: Use memory profiler, tracemalloc, psutil

– Understand views vs. copies: Views share memory, copies duplicate
– Choose right data structures: Arrays vs. lists, slots for classes

• Pandas Optimizations

– Downcast numeric types: [Link] numeric(..., downcast=’integer/float’)

– Use categorical: astype(’category’) for low-cardinality strings
– Read in chunks: chunksize parameter
– Drop columns early: usecols or drop()
– Avoid apply: Use vectorized operations

• NumPy Optimizations

– Vectorize everything: Replace loops with array operations

– Use in-place operations: +=, *= to avoid copies
– Fancy indexing carefully: Creates copies, not views

• Parallel Processing

– Multiprocessing: Split DataFrames across cores

– Dask: Out-of-core processing for datasets larger than RAM
– GPU computing: For deep learning, use mixed precision (FP16)

• Key Anti-Patterns to Avoid

– Loading entire datasets into memory at once

– Using Python loops instead of vectorized operations
– Creating unnecessary copies with slicing or methods
– Keeping large intermediate DataFrames alive
– Using object dtype for categorical data

53
54

Building a Reusable Data-Cleaning Pipeline

This guide covers the theory and practical implementation of reusable data-cleaning pipelines
based on authoritative Python data analysis resources. A well-designed pipeline automates
repetitive cleaning tasks, ensures consistency across projects, and makes data preparation more
maintainable.

1.1 What is a Data Pipeline?

A data pipeline is a series of data processing steps where the output of one step becomes the input
for the next. Think of it like an assembly line in manufacturing—each step performs a specific
function, transforming raw materials (messy data) into finished products (clean, analysis-ready
data).
The ETL framework provides the foundation:

• Extract: Pull messy or scattered data from various sources (CSV files, databases, APIs,
JSON)

• Transform: Clean, reformat, validate, and enrich the data

• Load: Save the polished result to a destination (new file, database, data warehouse)

1.2 Why Build Reusable Pipelines?

Without a pipeline, data cleaning becomes a maze of ever-growing Jupyter notebooks and man-
ual steps. Each new dataset means re-running cells in the right order, copying formulas, and
hoping nothing breaks. This repetition is draining, and the chance of subtle mistakes is ever
present.
A reusable pipeline offers several advantages:

• Consistency: Apply the same cleaning rules across multiple datasets

• Reproducibility: Run the exact same transformations on refreshed data

• Maintainability: Update cleaning logic in one place rather than scattered scripts

• Testing: Validate each component independently

• Documentation: The pipeline structure itself documents the cleaning process

1.3 Key Principles of Pipeline Design

• Modularity: Each cleaning operation should be a separate, focused function. This makes
the pipeline easier to understand, test, and modify.

• Configuration over hard-coding: Define cleaning rules, thresholds, and parameters in

configuration files rather than embedding them directly in code.

• Idempotency: Running the same pipeline multiple times on the same input should pro-
duce identical output. This is crucial for reproducibility.

• Error handling: The pipeline should handle problematic records gracefully—capturing

errors without crashing, and continuing to process valid data.

• Reporting: Track what changes were made during processing (duplicates removed, miss-
ing values filled, validation errors) to provide transparency.

54
 55

2. Core Components of a Data-Cleaning Pipeline

2.1 Data Ingestion
The first step is reading data from various sources. A reusable pipeline should handle multiple
formats:

• CSV, TSV, and other delimited files

• Excel spreadsheets

• JSON (including nested structures)

• Databases via SQL

• APIs

Best practices:

• Always check file formats and encodings before processing

• Create a consistent organization system (e.g., dictionary of DataFrames keyed by source

name)

• Verify data after reading—examine a few rows to ensure proper loading

2.2 Data Profiling

Before cleaning, understand what you’re working with. Profiling examines:

• Data types and their appropriateness

• Missing value counts and patterns

• Unique values and cardinality

• Statistical summaries (mean, median, min, max, quartiles)

• Potential outliers

This information guides cleaning decisions.

2.3 Data Cleaning Operations

Common cleaning tasks that belong in a pipeline include:
Missing Data Handling

• Removal of rows or columns with excessive missing values

• Imputation using mean, median, mode, or forward/backward fill

• Creating indicator columns to flag missing values

Duplicate Removal

• Removing exact duplicates

• Identifying near-duplicates using similarity thresholds

55
56

Data Type Correction

• Converting strings to numeric values

• Parsing dates and timestamps

• Converting low-cardinality columns to categorical types

Standardization

• Converting text to consistent case (upper, lower, title)

• Removing extra whitespace

• Standardizing categorical labels (e.g., “NYC” → “New York City”)

Outlier Handling

• Capping or flooring extreme values

• Removing values that are clearly erroneous

2.4 Data Validation

Validation ensures data meets business rules and quality standards. Each record should be
checked against a schema that defines:

• Required fields

• Expected data types

• Value ranges or allowed categories

• Format patterns (e.g., email address or phone number)

When validation fails, the pipeline should:

• Capture detailed error information

• Separate valid rows from invalid ones

• Continue processing valid data

2.5 Feature Engineering

For machine learning pipelines, feature engineering transforms raw data into model-ready fea-
tures:

• Creating derived columns (e.g., age from birth date)

• Binning continuous variables

• Encoding categorical variables (one-hot, ordinal, frequency encoding)

• Scaling numerical features

56
 57

2.6 Output and Reporting

The final pipeline should produce:

• Cleaned and validated datasets

• Error reports containing invalid records

• Processing statistics such as counts of duplicates removed or values imputed

• Logs for auditing, debugging, and monitoring pipeline behavior

0.0.1 3. Implementation Patterns

3.1 Function-Based Pipeline
The simplest approach chains together functions, each performing one cleaning operation:
1 def remov e_duplicates ( df ) :
2 initial_count = len ( df )
3 df = df . drop_duplicates ()
4 stats = { ’ duplicates_removed ’: initial_count - len ( df ) }
5 return df , stats
6
7 def f i l l_ m issing_values ( df ) :
8 stats = {}
9
10 # Handle numeric columns
11 num_cols = df . select_dtypes ( include =[ ’ number ’ ]) . columns
12 for col in num_cols :
13 missing = df [ col ]. isnull () . sum ()
14 if missing > 0:
15 df [ col ] = df [ col ]. fillna ( df [ col ]. median () )
16 stats [ f ’{ col } _filled ’] = missing
17
18 # Handle categorical columns
19 cat_cols = df . select_dtypes ( include =[ ’ object ’ , ’ category ’ ]) . columns
20 for col in cat_cols :
21 missing = df [ col ]. isnull () . sum ()
22 if missing > 0:
23 df [ col ] = df [ col ]. fillna ( ’ Unknown ’)
24 stats [ f ’{ col } _filled ’] = missing
25
26 return df , stats
27
28 def standardize_text ( df , columns ) :
29 for col in columns :
30 if col in df . columns :
31 df [ col ] = df [ col ]. str . strip () . str . lower ()
32 return df , { ’ text_standardized ’: len ( columns ) }
33
34 # Pipeline orchestrator
35 def run_pipeline ( df , steps ) :
36 " " " Execute a sequence of cleaning steps " " "
37 all_stats = {}
38
39 for step_name , step_func , * args in steps :
40 if args :
41 df , stats = step_func ( df , * args [0])
42 else :
43 df , stats = step_func ( df )
44 all_stats [ step_name ] = stats
45

57
 58

46 return df , all_stats
47
48
49 # Usage
50 steps = [
51 ( ’ remove_duplicates ’ , remove_duplicates ) ,
52 ( ’ fill_missing ’ , fill_missing_values ) ,
53 ( ’ standardize ’ , standardize_text , [ ’ name ’ , ’ address ’ ])
54 ]
55
56 clean_data , stats = run_pipeline ( raw_data , steps )

3. Implementation Patterns
3.1 Function-Based Pipeline
The simplest approach chains together functions, each performing one cleaning operation:
1 def remov e_duplicates ( df ) :
2 initial_count = len ( df )
3 df = df . drop_duplicates ()
4 stats = { ’ duplicates_removed ’: initial_count - len ( df ) }
5 return df , stats
6
7 def f i l l_ m issing_values ( df ) :
8 stats = {}
9
10 # Handle numeric columns
11 num_cols = df . select_dtypes ( include =[ ’ number ’ ]) . columns
12 for col in num_cols :
13 missing = df [ col ]. isnull () . sum ()
14 if missing > 0:
15 df [ col ] = df [ col ]. fillna ( df [ col ]. median () )
16 stats [ f ’{ col } _filled ’] = missing
17
18 # Handle categorical columns
19 cat_cols = df . select_dtypes ( include =[ ’ object ’ , ’ category ’ ]) . columns
20 for col in cat_cols :
21 missing = df [ col ]. isnull () . sum ()
22 if missing > 0:
23 df [ col ] = df [ col ]. fillna ( ’ Unknown ’)
24 stats [ f ’{ col } _filled ’] = missing
25
26 return df , stats
27
28 def standardize_text ( df , columns ) :
29 for col in columns :
30 if col in df . columns :
31 df [ col ] = df [ col ]. str . strip () . str . lower ()
32 return df , { ’ text_standardized ’: len ( columns ) }
33
34 # Pipeline orchestrator
35 def run_pipeline ( df , steps ) :
36 " " " Execute a sequence of cleaning steps " " "
37 all_stats = {}
38
39 for step_name , step_func , * args in steps :
40 if args :
41 df , stats = step_func ( df , * args [0])
42 else :
43 df , stats = step_func ( df )
44 all_stats [ step_name ] = stats
45

58
 59

46 return df , all_stats
47
48
49 # Usage
50 steps = [
51 ( ’ remove_duplicates ’ , remove_duplicates ) ,
52 ( ’ fill_missing ’ , fill_missing_values ) ,
53 ( ’ standardize ’ , standardize_text , [ ’ name ’ , ’ address ’ ])
54 ]
55
56 clean_data , stats = run_pipeline ( raw_data , steps )

3.2 Class-Based Pipeline

A class encapsulates state (configuration, statistics) and methods:
1 class D a t aCleaningPipeline :
2 def __init__ ( self , config = None ) :
3 self . config = config or {}
4 self . stats = {
5 ’ duplicates_removed ’: 0 ,
6 ’ missing_filled ’: 0 ,
7 ’ validation_errors ’: 0 ,
8 ’ outliers_capped ’: 0
9 }
10 self . errors = []
11
12 def clean_data ( self , df ) :
13 " " " Main cleaning workflow " " "
14 df = self . _remove_duplicates ( df )
15 df = self . _h an dl e_m is si ng_ va lu es ( df )
16 df = self . _standardize_columns ( df )
17 df = self . _handle_outliers ( df )
18 return df
19
20 def _remove_duplicates ( self , df ) :
21 initial = len ( df )
22 df = df . drop_duplicates (
23 subset = self . config . get ( ’ dedup_columns ’) ,
24 keep = self . config . get ( ’ dedup_keep ’ , ’ first ’)
25 )
26 self . stats [ ’ duplicates_removed ’] = initial - len ( df )
27 return df
28
29 def _ h an dl e_m is si ng_ va lu es ( self , df ) :
30 for col in df . columns :
31 missing = df [ col ]. isnull () . sum ()
32 if missing == 0:
33 continue
34
35 if col in self . config . get ( ’ numeric_columns ’ , []) :
36 strategy = self . config . get ( ’ numeric_impute ’ , ’ median ’)
37 if strategy == ’ median ’:
38 df [ col ] = df [ col ]. fillna ( df [ col ]. median () )
39 elif strategy == ’ mean ’:
40 df [ col ] = df [ col ]. fillna ( df [ col ]. mean () )
41 elif col in self . config . get ( ’ categorical_columns ’ , []) :
42 df [ col ] = df [ col ]. fillna (
43 self . config . get ( ’ missing_category ’ , ’ Unknown ’)
44 )
45
46 self . stats [ ’ missing_filled ’] += missing
47 return df

59
 60

48
49 def _ standardize_columns ( self , df ) :
50 text_cols = self . config . get ( ’ text_columns ’ , [])
51 for col in text_cols :
52 if col in df . columns :
53 df [ col ] = df [ col ]. str . strip () . str . lower ()
54 return df
55
56 def _handle_outliers ( self , df ) :
57 for col in self . config . get ( ’ numeric_columns ’ , []) :
58 if col not in df . columns :
59 continue
60
61 q1 = df [ col ]. quantile (0.25)
62 q3 = df [ col ]. quantile (0.75)
63 iqr = q3 - q1
64 lower_bound = q1 - 1.5 * iqr
65 upper_bound = q3 + 1.5 * iqr
66
67 outliers = (
68 ( df [ col ] < lower_bound ) | ( df [ col ] > upper_bound )
69 ) . sum ()
70
71 if outliers > 0 and self . config . get ( ’ cap_outliers ’ , True ) :
72 df [ col ] = df [ col ]. clip ( lower_bound , upper_bound )
73 self . stats [ ’ outliers_capped ’] += outliers
74
75 return df
76
77 def validate_data ( self , df , schema ) :
78 " " " Validate against a schema " " "
79 valid_rows = []
80
81 for idx , row in df . iterrows () :
82 try :
83 # Apply validation rules from schema
84 validated = self . _validate_row ( row , schema )
85 valid_rows . append ( validated )
86 except Exception as e :
87 self . errors . append ({ ’ row ’: idx , ’ error ’: str ( e ) })
88 self . stats [ ’ validation_errors ’] += 1
89
90 return pd . DataFrame ( valid_rows )
91
92 def run ( self , df ) :
93 " " " Execute complete pipeline " " "
94 cleaned = self . clean_data ( df )
95
96 if hasattr ( self , ’ validation_schema ’) :
97 cleaned = self . validate_data ( cleaned , self . validation_schema )
98
99 return {
100 ’ data ’: cleaned ,
101 ’ stats ’: self . stats ,
102 ’ errors ’: self . errors
103 }
104
105
106 # Usage
107 config = {
108 ’ numeric_columns ’: [ ’ age ’ , ’ salary ’ , ’ experience ’] ,
109 ’ c a t egorical_columns ’: [ ’ department ’ , ’ gender ’] ,
110 ’ text_columns ’: [ ’ name ’ , ’ address ’] ,

60
 61

111 ’ numeric_impute ’: ’ median ’ ,

112 ’ cap_outliers ’: True
113 }
114
115 pipeline = DataCleaningPipeline ( config )
116 result = pipeline . run ( raw_data )
117
118 clean_df = result [ ’ data ’]
119 print ( result [ ’ stats ’ ])

3.3 Pipeline with Configuration Files

For maximum reusability, separate configuration from code.
Configuration file (YAML)
1 # config . yaml
2 data_sources :
3 main :
4 path : " data / raw / sales . csv "
5 format : csv
6 encoding : utf -8
7 dtype :
8 customer_id : str
9 amount : float64
10
11 cleaning :
12 duplicates :
13 subset : [ customer_id , date ]
14 keep : first
15
16 missing_values :
17 numeric_strategy : median
18 categorical_fill : " Unknown "
19 max_missing_pct : 0.3 # drop columns exceeding this
20
21 outliers :
22 method : iqr
23 multiplier : 1.5
24 cap : true
25
26 text_columns :
27 - customer_name
28 - p r o duct_description
29 operations :
30 - strip
31 - lower
32
33 validation :
34 schema :
35 customer_id :
36 type : str
37 required : true
38 pattern : " ^ CUST \\ d {6} $ "
39 amount :
40 type : float
41 required : true
42 min : 0
43 max : 1000000
44 date :
45 type : datetime
46 required : true

Loading and using the configuration

61
 62

1 import yaml
2 import pandas as pd
3
4 class C o n figurablePipeline :
5 def __init__ ( self , config_path ) :
6 with open ( config_path , ’r ’) as f :
7 self . config = yaml . safe_load ( f )
8 self . stats = {}
9
10 def run ( self ) :
11 # Extract
12 df = self . _extract ()
13
14 # Transform
15 df = self . _clean_data ( df )
16
17 # Validate
18 df , errors = self . _validate ( df )
19
20 # Load
21 self . _load ( df )
22
23 return {
24 ’ data ’: df ,
25 ’ stats ’: self . stats ,
26 ’ errors ’: errors
27 }
28
29 def _extract ( self ) :
30 sources = self . config [ ’ data_sources ’]
31 # Implementation for reading from various sources
32 # ...
33
34 def _clean_data ( self , df ) :
35 # Apply cleaning steps from config
36 # ...
37 return df

3.4 Pipeline with Error Handling and Reporting

A robust pipeline captures detailed information about what was fixed:
1 class ReportingPipeline :
2 def __init__ ( self ) :
3 self . report = {
4 ’ input_rows ’: 0 ,
5 ’ output_rows ’: 0 ,
6 ’ steps ’: [] ,
7 ’ errors ’: [] ,
8 ’ warnings ’: []
9 }
10
11 def add_step_result ( self , step_name , before_count , after_count , details = None
):
12 self . report [ ’ steps ’ ]. append ({
13 ’ step ’: step_name ,
14 ’ rows_removed ’: before_count - after_count ,
15 ’ details ’: details or {}
16 })
17
18 def generate_report ( self ) :
19 " " " Create human - readable report " " "
20 lines = []

62
 63

21 lines . append ( " = " * 50)

22 lines . append ( " DATA CLEANING PIPELINE REPORT " )
23 lines . append ( " = " * 50)
24 lines . append ( f " Input rows : { self . report [ ’ input_rows ’]} " )
25 lines . append ( f " Output rows : { self . report [ ’ output_rows ’]} " )
26 lines . append (
27 f " Rows removed : { self . report [ ’ input_rows ’] - self . report [ ’
output_rows ’]} "
28 )
29 lines . append ( " \ nStep Details : " )
30
31 for step in self . report [ ’ steps ’ ]:
32 lines . append (
33 f " - { step [ ’ step ’]}: removed { step [ ’ rows_removed ’]} rows "
34 )
35 for key , value in step [ ’ details ’ ]. items () :
36 lines . append ( f " { key }: { value } " )
37
38 if self . report [ ’ errors ’ ]:
39 lines . append ( " \ nErrors : " )
40 for err in self . report [ ’ errors ’ ]:
41 lines . append ( f " - { err } " )
42
43 return " \ n " . join ( lines )
44
45
46 # Pipeline with reporting
47 def ru n_ wi th_reporting ( df , pipeline_funcs ) :
48 reporter = ReportingPipeline ()
49 reporter . report [ ’ input_rows ’] = len ( df )
50
51 current_df = df . copy ()
52
53 for name , func in pipeline_funcs :
54 before = len ( current_df )
55 current_df , details = func ( current_df )
56 after = len ( current_df )
57 reporter . add_step_result ( name , before , after , details )
58
59 reporter . report [ ’ output_rows ’] = len ( current_df )
60
61 return current_df , reporter

4. Testing and Validation

4.1 Unit Testing Pipeline Components
Each cleaning function should have tests:
1 import pytest
2 import pandas as pd
3 import numpy as np
4
5 def t e s t _ r emo ve _d upl ic at es () :
6 # Test data with duplicates
7 df = pd . DataFrame ({
8 ’ id ’: [1 , 1 , 2 , 3 , 3] ,
9 ’ value ’: [ ’a ’ , ’a ’ , ’b ’ , ’c ’ , ’c ’]
10 })
11
12 cleaned , stats = remove_duplicates ( df )
13
14 assert len ( cleaned ) == 3 # IDs 1 , 2 , 3

63
 64

15 assert stats [ ’ duplicates_removed ’] == 2

16 assert cleaned [ ’ id ’ ]. tolist () == [1 , 2 , 3]
17
18
19 def t e s t _ f i l l _m i s s in g _ va l u e s () :
20 df = pd . DataFrame ({
21 ’ numeric ’: [1 , np . nan , 3 , np . nan , 5] ,
22 ’ category ’: [ ’a ’ , ’b ’ , np . nan , ’d ’ , ’e ’]
23 })
24
25 cleaned , stats = fill_missing_values ( df )
26
27 assert cleaned [ ’ numeric ’ ]. isnull () . sum () == 0
28 assert cleaned [ ’ numeric ’ ]. iloc [1] == 3 # median of [1 ,3 ,5]
29 assert cleaned [ ’ category ’ ]. isnull () . sum () == 0
30 assert cleaned [ ’ category ’ ]. iloc [2] == ’ Unknown ’

4.2 Pipeline Integration Testing

Test the entire pipeline on a small sample:
1 def te st _f ull_pipeline () :
2 # Create small messy dataset
3 test_data = pd . DataFrame ({
4 ’ id ’: [1 , 1 , 2 , 3 , 4] ,
5 ’ name ’: [ ’ Alice ’ , ’ Alice ’ , ’ Bob ’ , None , ’ David ’] ,
6 ’ age ’: [25 , 25 , -5 , 35 , 150] ,
7 ’ score ’: [85 , 85 , 92 , None , 78]
8 })
9
10 pipeline = DataCleaningPipeline ( config )
11 result = pipeline . run ( test_data )
12
13 assert result [ ’ stats ’ ][ ’ duplicates_removed ’] == 1
14 assert result [ ’ stats ’ ][ ’ missing_filled ’] >= 1
15 assert result [ ’ stats ’ ][ ’ validation_errors ’] >= 1
16 assert len ( result [ ’ data ’ ]) < len ( test_data ) # Some rows removed

5. Complete Production-Ready Pipeline Example

Here is a comprehensive example combining all the concepts discussed earlier. This pipeline in-
cludes configuration loading, logging, error handling, validation, transformation, and reporting.
1 import pandas as pd
2 import numpy as np
3 import logging
4 from datetime import datetime
5 from typing import Dict , List , Tuple
6 import yaml
7 import json
8 import re
9
10 class P r o d uc tio nD at aPi pe li ne :
11 """
12 Production - ready data cleaning pipeline with logging ,
13 error handling , and comprehensive reporting .
14 """
15
16 def __init__ ( self , config_path : str , log_path : str = None ) :
17 with open ( config_path , ’r ’) as f :
18 self . config = yaml . safe_load ( f )
19

64
 65

20 # Logging setup
21 self . logger = logging . getLogger ( __name__ )
22 if log_path :
23 handler = logging . FileHandler ( log_path )
24 formatter = logging . Formatter (
25 ’ %( asctime ) s - %( name ) s - %( levelname ) s - %( message ) s ’
26 )
27 handler . setFormatter ( formatter )
28 self . logger . addHandler ( handler )
29 self . logger . setLevel ( logging . INFO )
30
31 self . stats = {
32 ’ start_time ’: datetime . now () ,
33 ’ end_time ’: None ,
34 ’ input_rows ’: 0 ,
35 ’ output_rows ’: 0 ,
36 ’ steps ’: [] ,
37 ’ errors ’: []
38 }
39
40 def extract ( self ) -> pd . DataFrame :
41 " " " Extract data from configured source " " "
42 source = self . config [ ’ data_source ’]
43
44 if source [ ’ format ’] == ’ csv ’:
45 df = pd . read_csv ( source [ ’ path ’ ])
46 elif source [ ’ format ’] == ’ excel ’:
47 df = pd . read_excel ( source [ ’ path ’ ])
48 elif source [ ’ format ’] == ’ json ’:
49 df = pd . read_json ( source [ ’ path ’ ])
50 else :
51 raise ValueError ( " Unsupported format " )
52
53 self . stats [ ’ input_rows ’] = len ( df )
54 return df
55
56 def clean ( self , df : pd . DataFrame ) -> pd . DataFrame :
57 " " " Apply cleaning operations " " "
58
59 # Remove duplicates
60 before = len ( df )
61 df = df . drop_duplicates ()
62 removed = before - len ( df )
63
64 self . stats [ ’ steps ’ ]. append ({
65 ’ step ’: ’ remove_duplicates ’ ,
66 ’ rows_removed ’: removed
67 })
68
69 # Handle missing values
70 missing_filled = 0
71 for col in df . columns :
72 missing = df [ col ]. isnull () . sum ()
73
74 if missing == 0:
75 continue
76
77 if pd . api . types . is_numeric_dtype ( df [ col ]) :
78 df [ col ] = df [ col ]. fillna ( df [ col ]. median () )
79 else :
80 df [ col ] = df [ col ]. fillna ( " Unknown " )
81
82 missing_filled += missing

65
 66

83
84 self . stats [ ’ steps ’ ]. append ({
85 ’ step ’: ’ handle_missing ’ ,
86 ’ values_filled ’: missing_filled
87 })
88
89 return df
90
91 def validate ( self , df : pd . DataFrame ) -> Tuple [ pd . DataFrame , List [ Dict ]]:
92 " " " Validate dataset using schema rules " " "
93
94 schema = self . config . get ( ’ validation ’ , {}) . get ( ’ schema ’ , {})
95 valid_rows = []
96 errors = []
97
98 for idx , row in df . iterrows () :
99 row_errors = []
100
101 for field , rules in schema . items () :
102 value = row . get ( field )
103
104 if rules . get ( ’ required ’) and pd . isna ( value ) :
105 row_errors . append ( f " { field } missing " )
106
107 if ’ min ’ in rules and value < rules [ ’ min ’ ]:
108 row_errors . append ( f " { field } below minimum " )
109
110 if ’ max ’ in rules and value > rules [ ’ max ’ ]:
111 row_errors . append ( f " { field } above maximum " )
112
113 if row_errors :
114 errors . append ({
115 " row_index " : idx ,
116 " errors " : row_errors
117 })
118 else :
119 valid_rows . append ( row )
120
121 return pd . DataFrame ( valid_rows ) , errors
122
123 def transform ( self , df : pd . DataFrame ) -> pd . DataFrame :
124 " " " Apply feature transformations " " "
125
126 transforms = self . config . get ( " transformations " , {})
127
128 for encode in transforms . get ( " encoding " , []) :
129 col = encode [ " column " ]
130
131 if encode [ " method " ] == " one_hot " :
132 dummies = pd . get_dummies ( df [ col ] , prefix = col )
133 df = pd . concat ([ df , dummies ] , axis =1)
134 df . drop ( columns =[ col ] , inplace = True )
135
136 return df
137
138 def load ( self , df : pd . DataFrame , errors : List [ Dict ]) :
139 " " " Save cleaned data and reports " " "
140
141 output = self . config [ " output " ]
142
143 df . to_csv ( output [ " clean_path " ] , index = False )
144
145 if errors :

66
 67

146 pd . DataFrame ( errors ) . to_csv (

147 output [ " error_path " ] , index = False
148 )
149
150 self . stats [ " end_time " ] = datetime . now ()
151 self . stats [ " output_rows " ] = len ( df )
152
153 with open ( output [ " stats_path " ] , " w " ) as f :
154 json . dump ( self . stats , f , indent =2 , default = str )
155
156 def run ( self ) -> Dict :
157 " " " Run full pipeline " " "
158
159 try :
160 df = self . extract ()
161 df = self . clean ( df )
162 df , errors = self . validate ( df )
163
164 if self . config . get ( " transformations " ) :
165 df = self . transform ( df )
166
167 self . load ( df , errors )
168
169 return {
170 " success " : True ,
171 " stats " : self . stats ,
172 " error_count " : len ( errors )
173 }
174
175 except Exception as e :
176 return {
177 " success " : False ,
178 " error " : str ( e ) ,
179 " stats " : self . stats
180 }
181
182
183 # Usage
184 if __name__ == " __main__ " :
185
186 pipeline = Pr od uc tio nD at aPi pe li ne (
187 config_path = " pipeline_config . yaml " ,
188 log_path = " pipeline . log "
189 )
190
191 result = pipeline . run ()
192
193 if result [ " success " ]:
194 print ( " Pipeline completed successfully " )
195 print ( " Processed rows : " , result [ " stats " ][ " output_rows " ])
196 else :
197 print ( " Pipeline failed : " , result [ " error " ])

Summary for Quick Revision

Key Principles

• Modularity: Each cleaning operation should be implemented as a separate function.

• Configuration: Define pipeline rules in configuration files (YAML/JSON) instead of

hardcoding them inside the program.

67
68

• Idempotency: Running the pipeline multiple times with the same input should produce
the same output every time.

• Error Handling: Pipelines should fail gracefully and capture errors for debugging.

• Reporting: Maintain detailed reports of all transformations and operations.

Pipeline Components
• Extract: Read data from sources such as CSV, Excel, JSON, or databases.

• Clean: Remove duplicates, handle missing values, standardize text fields, and manage
outliers.

• Validate: Check the dataset against schema rules including required fields, data types,
ranges, and patterns.

• Transform: Perform feature engineering such as derived columns, encoding categorical

variables, and scaling numerical features.

• Load: Save the cleaned dataset along with logs, statistics, and error reports.

Implementation Patterns
• Function-based Pipeline: Uses a simple sequence of functions for data cleaning.

• Class-based Pipeline: Encapsulates state, configuration, and logic inside classes.

• Configuration-driven Pipeline: Uses YAML or JSON configuration files to control

pipeline behavior.

• Production-ready Pipeline: Includes logging, error handling, monitoring, and detailed

reporting.

Testing Strategies
• Unit Testing: Test individual cleaning functions separately.

• Integration Testing: Test the complete pipeline using small sample datasets.

• Validation Testing: Compare outputs against expected or known correct results.

Best Practices
• Always work on copies of data to avoid modifying the original dataset.

• Track statistics for every transformation step.

• Log all operations for debugging and reproducibility.

• Handle edge cases such as empty DataFrames or columns with all missing values.

• Plan for schema evolution as datasets and requirements change over time.

68
 69

Introduction to Data Pipeline Components

A data pipeline is a sequence of processes used to collect, process, and store data so that it can
be analyzed efficiently. In data science and data engineering, pipelines automate the movement
of data from its source to its final destination while ensuring data quality and consistency. The
main components of a typical data pipeline include data ingestion, data transformation, and
data storage.

1. Data Ingestion
Data ingestion is the process of collecting and importing data from various sources into a system
for further processing. The sources may include databases, CSV files, APIs, web applications,
sensors, or streaming platforms.
Data ingestion can be classified into two main types:

• Batch Ingestion: Data is collected and processed in large batches at scheduled intervals.
• Real-Time (Streaming) Ingestion: Data is continuously ingested as it is generated.

The goal of data ingestion is to ensure that raw data from different sources is reliably
transferred into the pipeline.

2. Data Transformation
Data transformation involves cleaning, organizing, and converting raw data into a format that
is suitable for analysis or machine learning tasks. This stage is crucial because raw data often
contains missing values, inconsistencies, or errors.
Common transformation operations include:

• Data cleaning (handling missing values and duplicates)

• Standardizing text and formats
• Feature engineering and data enrichment
• Encoding categorical variables
• Scaling or normalizing numerical features

The objective of this step is to produce high-quality and structured data that can be easily
used for analysis and modeling.

3. Data Storage
Data storage refers to storing the processed data in systems where it can be accessed for analysis,
reporting, or machine learning. Proper storage ensures efficient retrieval, scalability, and security
of data.
Common storage systems include:

• Relational databases (e.g., SQL databases)

• Data warehouses
• Data lakes
• Cloud storage systems

After storage, the data can be accessed by analytics tools, dashboards, or machine learning
pipelines.

69
70

Summary
A well-designed data pipeline ensures smooth data flow from ingestion to transformation and
finally to storage. By automating these stages, organizations can efficiently process large volumes
of data and support reliable data-driven decision making.

70
Chapter-4
Applied Statistics and Exploratory Analysis

4.1 Statistical Measures

Understanding the shape, spread, and relationships within your data is fundamental to exploratory analysis. This section covers
key statistical measures that quantify these aspects.

4.1.1 Covariance and Correlation

When analyzing two variables, it is crucial to understand if and how they are related.
Covariance is a measure of the joint variability of two random variables. It indicates the directional relationship between
them.

• Positive Covariance: When a larger value of one variable corresponds to a larger value of the other variable.

• Negative Covariance: When a larger value of one variable corresponds to a smaller value of the other variable.

Formula: For two samples Xi and Yi , the covariance is calculated as:

Pn
(Xi − X̄)(Yi − Ȳ )
Cov(X, Y ) = i=1 (1)
n−1
where X̄ and Ȳ are the sample means.
A key limitation is that its value is not standardized and is difficult to interpret.
Correlation (Coefficient) addresses the limitation of covariance by standardizing the measure. It is a dimensionless
quantity that always lies between −1 and +1.

• +1: Perfect positive linear relationship

• 0: No linear relationship

• −1: Perfect negative linear relationship

Formula (Pearson Correlation): The most common type, measuring the strength of a linear relationship.

Cov(X, Y )
rXY = (2)
sX sY
where sX and sY are the sample standard deviations of X and Y .
The cov2corr function is a standard utility to convert a covariance matrix into a correlation matrix by dividing by the outer
product of the standard deviations.

4.1.2 Skewness
Skewness is a measure of the asymmetry of the probability distribution of a real-valued random variable about its mean.

• Symmetrical Distribution: The distribution is bell-shaped and balanced (e.g., Normal distribution). Skewness is
approximately zero.

Positive Skew (Right-skewed): The distribution has a long tail on the right. The mean is typically greater than the
median.
The formula for sample skewness is often based on the third central moment:
1
Pn 3
n i=1 (Xi − X̄)
Skewness = (3)
s3
Negative Skew (Left-skewed): The distribution has a long tail on the left. The mean is typically less than the median.

4.1.3 Kurtosis
Kurtosis is a measure of the “tailedness” of the probability distribution. It describes the combined weight of the tails relative
to the center of the distribution.

• High Kurtosis (Leptokurtic): The distribution has heavy tails and a sharp, narrow peak. This indicates more outliers.

• Low Kurtosis (Platykurtic): The distribution has light tails and a flat, spread-out peak. This indicates fewer or no
outliers.

Formula: Kurtosis is based on the fourth central moment. It is often reported as excess kurtosis, which is the kurtosis value
minus 3 (the kurtosis of a normal distribution).
1
Pn 4
n i=1 (Xi − X̄)
Kurtosis = (4)
s4
1
 2

4.1 Statistical Measures: Python Programs

This section provides practical implementations of the concepts discussed above using Python’s core data science libraries.

Program 1: Calculating Covariance and Correlation

This program demonstrates how to calculate covariance and the Pearson correlation coefficient between two variables using
NumPy and Pandas.
1 import numpy as np
2 import pandas as pd
3
4 # Sample data : Height ( in cm ) and Weight ( in kg )
5 height = [170 , 175 , 168 , 180 , 178 , 182 , 165 , 172]
6 weight = [65 , 70 , 63 , 80 , 75 , 85 , 60 , 68]
7
8 print ( " --- Covariance and Correlation Calculation ---" )
9 print ( f " Height data : { height } " )
10 print ( f " Weight data : { weight }\ n " )
11

12 # Using NumPy
13 print ( " Using NumPy : " )
14 cov_numpy = np . cov ( height , weight , ddof =1) [0 , 1]
15 corr_numpy = np . corrcoef ( height , weight ) [0 , 1]
16
17 print ( f " Covariance : { cov_numpy :.2 f } " )
18 print ( f " Correlation ( Pearson r ) : { corr_numpy :.4 f }\ n " )
19
20 # Using Pandas
21 print ( " Using Pandas : " )
22 df = pd . DataFrame ({ ’ Height ’: height , ’ Weight ’: weight })
23

24 cov_pandas = df [ ’ Height ’ ]. cov ( df [ ’ Weight ’ ])

25 corr_pandas = df [ ’ Height ’ ]. corr ( df [ ’ Weight ’ ])
26
27 print ( f " Covariance : { cov_pandas :.2 f } " )
28 print ( f " Correlation ( Pearson r ) : { corr_pandas :.4 f } " )
29

30 # Interpretation
31 if abs ( corr_pandas ) > 0.7:
32 print ( " \ nInterpretation : Strong linear relationship . " )
33 else :
34 print ( " \ nInterpretation : Moderate or weak linear relationship . " )
Listing 1: Calculating Covariance and Correlation in Python

Explanation

• [Link]() and [Link]() compute the covariance and correlation matrices. The element [0,1] represents the
relationship between the two variables.

• The parameter ddof=1 ensures that the sample covariance is calculated.

• Pandas provides convenient methods .cov() and .corr() to compute these statistics directly from DataFrame columns.

The positive value confirms that height and weight increase together, and the correlation value close to 1 indicates a strong
positive linear relationship.

Program 2: Calculating Skewness and Kurtosis

This program demonstrates how to assess the shape of a distribution using SciPy.
1 import numpy as np
2 from scipy . stats import skew , kurtosis
3 import pandas as pd
4
5 np . random . seed (42)
6
7 # 1. Symmetrical distribution ( Normal )
8 normal_data = np . random . normal ( loc =100 , scale =10 , size =1000)
9
10 # 2. Positively skewed distribution
11 positive_skew = np . random . exponential ( scale =2 , size =1000)
12
13 # 3. Negatively skewed distribution
14 negative_skew = - np . random . exponential ( scale =2 , size =1000) + 5
15
16 print ( " --- Skewness and Kurtosis Calculation ---" )
17
18 print ( " \ n1 . Symmetrical ( Normal ) Data : " )
19 print ( f " Skewness : { skew ( normal_data ) :.4 f } " )
20 print ( f " Kurtosis : { kurtosis ( normal_data , fisher = True ) :.4 f } " )
21 print ( f " Kurtosis ( Pearson ) : { kurtosis ( normal_data , fisher = False ) :.4 f } " )
22
23 print ( " \ n2 . Positively Skewed Data : " )
24 print ( f " Skewness : { skew ( positive_skew ) :.4 f } " )
25 print ( f " Excess Kurtosis : { kurtosis ( positive_skew , fisher = True ) :.4 f } " )
26

27 print ( " \ n3 . Negatively Skewed Data : " )

28 print ( f " Skewness : { skew ( negative_skew ) :.4 f } " )
29 print ( f " Excess Kurtosis : { kurtosis ( negative_skew , fisher = True ) :.4 f } " )
30
31 print ( " \n - - - Pandas Summary Statistics ---" )
32

2
 3

33 df = pd . DataFrame ({
34 ’ Normal ’: normal_data ,
35 ’ Pos_Skew ’: positive_skew ,
36 ’ Neg_Skew ’: negative_skew
37 })
38

39 print ( df . describe () )
40
41 print ( " \n - - - Pandas Skew and Kurtosis ---" )
42 stats_df = df . agg ([ ’ skew ’ , ’ kurtosis ’ ]) . round (4)
43 print ( stats_df )
Listing 2: Calculating Skewness and Kurtosis in Python
Explanation

• [Link]() calculates the skewness of the data distribution.

• [Link]() measures the tailedness of the distribution.

• When fisher=True, the function returns excess kurtosis where a normal distribution has a value of 0.

• The Pandas method .agg() allows multiple statistical functions such as skew and kurtosis to be applied simultaneously
to DataFrame columns.

Numerical Example for Covariance, Correlation, Skewness, and Kurtosis

1. Numerical Example for Covariance and Correlation
Given Data

• X = {2, 4, 6, 8, 10}

• Y = {4, 6, 8, 10, 12}

• Number of observations: n = 5

Step 1: Compute Means

2 + 4 + 6 + 8 + 10
X̄ = =6
5
4 + 6 + 8 + 10 + 12
Ȳ = =8
5
Step 2: Compute Deviations

• X1 − X̄ = 2 − 6 = −4, Y1 − Ȳ = 4 − 8 = −4, product = 16

• X2 − X̄ = 4 − 6 = −2, Y2 − Ȳ = 6 − 8 = −2, product = 4

• X3 − X̄ = 6 − 6 = 0, Y3 − Ȳ = 8 − 8 = 0, product = 0

• X4 − X̄ = 8 − 6 = 2, Y4 − Ȳ = 10 − 8 = 2, product = 4

• X5 − X̄ = 10 − 6 = 4, Y5 − Ȳ = 12 − 8 = 4, product = 16

X
(X − X̄)(Y − Ȳ ) = 16 + 4 + 0 + 4 + 16 = 40
Step 3: Covariance
P
(X − X̄)(Y − Ȳ )
Cov(X, Y ) =
n−1
40
Cov(X, Y ) = = 10
4
Step 4: Standard Deviations
For X

(−4)2 + (−2)2 + 02 + 22 + 42

= 16 + 4 + 0 + 4 + 16 = 40

40 √
r
sX = = 10 = 3.162
4
For Y

(−4)2 + (−2)2 + 02 + 22 + 42 = 40

40 √
r
sY = = 10 = 3.162
4
Step 5: Correlation

Cov(X, Y )
r=
sX sY

3
4

10
r=
(3.162)(3.162)
10
r= =1
10
Result
• Correlation = 1
• Interpretation: Perfect positive linear relationship

2. Numerical Example for Skewness

Data

X = {2, 4, 6, 8, 10}
Step 1: Mean

X̄ = 6
Step 2: Deviations
• 2 − 6 = −4, (−4)3 = −64
• 4 − 6 = −2, (−2)3 = −8
• 6 − 6 = 0, 03 = 0
• 8 − 6 = 2, 23 = 8
• 10 − 6 = 4, 43 = 64

X
(X − X̄)3 = −64 − 8 + 0 + 8 + 64 = 0
Step 3: Standard Deviation
r
40
s= = 3.162
4
Step 4: Skewness
1
(X − X̄)3
P
n
Skewness =
s3
0/5
Skewness = =0
(3.162)3
Result
• Skewness = 0
• Interpretation: Distribution is symmetrical

3. Numerical Example for Kurtosis

Step 1: Fourth Power Deviations
• (−4)4 = 256
• (−2)4 = 16
• 04 = 0
• 24 = 16
• 44 = 256

X
(X − X̄)4 = 256 + 16 + 0 + 16 + 256 = 544
Step 2: Kurtosis
1
(X − X̄)4
P
n
Kurtosis =
s4

s = 3.162

s4 = 100

544/5
Kurtosis =
100
108.8
Kurtosis = = 1.088
100
Step 3: Excess Kurtosis

Excess Kurtosis = Kurtosis − 3

= 1.088 − 3 = −1.912
Result

4
 5

• Kurtosis = 1.088

• Excess Kurtosis = −1.912

• Interpretation: Distribution is Platykurtic (flat)

Final Results Summary

• Covariance = 10 (Positive relationship)

• Correlation = 1 (Perfect positive linear relationship)

• Skewness = 0 (Symmetrical distribution)

• Kurtosis = 1.088 (Platykurtic distribution)

4.2 Probability Review, Sampling, and Hypothesis Testing: Theory

This section covers the foundational concepts that allow us to move from describing a sample to making inferences about a
larger population.

4.2.1 Probability Review

Probability forms the mathematical foundation for statistics. A probability distribution defines the likelihood of different
outcomes for a random variable.

• Random Variable: A variable whose possible values are numerical outcomes of a random phenomenon.

• Probability Density Function (PDF): For continuous variables (e.g., Normal distribution), the PDF describes the
relative likelihood that the variable takes on a given value. The area under the PDF curve over an interval represents the
probability of the variable falling in that interval.

• Cumulative Distribution Function (CDF): The CDF gives the probability that a random variable takes a value less
than or equal to a specific point.

F (x) = P (X ≤ x)

• Common Distributions: The [Link] module provides a wide range of distributions, including:

– Normal distribution (norm)

– Exponential distribution (expon)
– Binomial distribution (binom)

Each distribution provides methods for calculating PDF, CDF, and generating random samples.

4.2.2 Sampling Concepts

Sampling is the process of selecting a subset of individuals from a population to estimate characteristics of the whole population.

• Population vs. Sample:

A parameter is a numerical summary of a population (e.g., population mean µ).
A statistic is a numerical summary calculated from a sample (e.g., sample mean x̄).
Statistics are used to estimate population parameters.

• Random Sample:
A sample drawn such that every member of the population has an equal and independent chance of being selected. This
is essential for making valid statistical inferences.

• Sampling with Replacement:

After an item is selected from the population, it is returned to the pool and can be chosen again. This method is commonly
used in simulation studies.

• Sampling without Replacement:

An item, once selected, is removed from the pool and cannot be chosen again. This method is commonly used in surveys
and real-world data collection.

• Sampling Distribution:
The probability distribution of a given statistic (such as the sample mean) obtained from many random samples drawn
from the same population.
The standard deviation of this distribution is called the standard error, which measures the precision of the sample statistic
as an estimate of the population parameter.

5
 6

4.2.3 Hypothesis Testing

Hypothesis testing is a formal statistical framework used to make decisions about population parameters using sample data.
Steps in Hypothesis Testing
1. State the Hypotheses

• Null Hypothesis (H0 ): A statement of no effect or no difference.

H0 : θ = θ0

• Alternative Hypothesis (Ha or H1 ): A statement that contradicts the null hypothesis.

Ha : θ ̸= θ0

2. Choose a Test Statistic

A test statistic is a single numerical value calculated from the sample data used to evaluate the null hypothesis. Examples
include the t-statistic and the χ2 statistic.
3. Determine the Sampling Distribution
This is the probability distribution of the test statistic assuming the null hypothesis is true.
4. Set the Significance Level
The significance level (α) is the probability of rejecting the null hypothesis when it is actually true (Type I error).
Commonly used value:

α = 0.05
5. Compute the p-value
The p-value is the probability of observing a test statistic as extreme as or more extreme than the observed value, assuming
the null hypothesis is true.

• If p-value ≤ α, reject H0 (statistically significant).

• If p-value > α, fail to reject H0 .

6. Critical Region
The critical region is the set of values of the test statistic that leads to rejection of the null hypothesis. The boundaries
of this region are known as critical values, determined by α and the sampling distribution.

Program 2: Sampling Techniques

This program demonstrates different ways to sample from a dataset using NumPy.
1 import numpy as np
2 import pandas as pd
3
4 print ( " --- Sampling Techniques ---" )
5
6 # Create a sample population dataset
7 np . random . seed (42)
8 population = pd . DataFrame ({
9 ’ ID ’: range (1 , 101) ,
10 ’ Value ’: np . random . normal (100 , 15 , 100)
11 })
12 print ( f " Population size : { len ( population ) } " )
13
14 # 1. Sampling with replacement ( default )
15 s a m p l e _ w i t h _ r e p l a c e m e n t = np . random . choice ( population [ ’ ID ’] , size =10 , replace = True )
16 print ( f " \ n1 . Sample with replacement ( IDs ) : { s a m p l e _ w i t h _ r e p l a c e m e n t } " )
17 print ( f " Note : IDs can appear multiple times " )
18
19 # 2. Sampling without replacement
20 s a m p l e _ w i t h o u t _ r e p l a c e m e n t = np . random . choice ( population [ ’ ID ’] , size =10 , replace = False )
21 print ( f " \ n2 . Sample without replacement ( IDs ) : { s a m p l e _ w i t h o u t _ r e p l a c e m e n t } " )
22 print ( f " Note : All IDs are unique " )
23
24 # 3. Sampling with probability weights
25 weights = population [ ’ Value ’] / population [ ’ Value ’ ]. sum ()
26
27 weighted_sample = np . random . choice (
28 population [ ’ ID ’] ,
29 size =5 ,
30 replace = False ,
31 p = weights
32 )
33
34 print ( f " \ n3 . Weighted sample ( IDs ) : { weighted_sample } " )
35 print ( f " Note : IDs with larger ’ Value ’ have higher selection probability " )
36
37 # 4. Random shuffling
38 print ( f " \ n4 . Random shuffling : " )
39 original_array = np . arange (10)
40 print ( f " Original : { original_array } " )
41
42 shuffled_array = np . random . permutation ( original_array )
43 print ( f " Permuted copy : { shuffled_array } " )
44
45 np . random . shuffle ( original_array )
46 print ( f " In - place shuffle : { original_array } " )
Listing 3: Sampling Techniques in Python

6
 7

Explanation

• [Link]() is the primary function for random sampling in NumPy.

• The parameter replace=True enables sampling with replacement.

• The parameter replace=False enables sampling without replacement.

• The parameter p allows assigning probability weights to each element.

• [Link]() returns a randomly permuted copy of a sequence.

• [Link]() randomly shuffles a sequence in-place.

Program 3: Hypothesis Testing – Binomial Test

This program demonstrates hypothesis testing by recreating the ESP study example, finding critical regions, and performing a
binomial test.
1 import numpy as np
2 import scipy . stats as stats
3 import matplotlib . pyplot as plt
4
5 print ( " --- Hypothesis Testing : ESP Study Example ---" )
6
7 # Parameters
8 n_trials = 100
9 n_simulations = 10000
10 alpha = 0.05
11
12 # Problem 0: Find critical region using simulation
13 print ( f " \ nProblem 0: Finding critical region ( alpha ={ alpha }) " )
14

15 np . random . seed (42)

16
17 # Simulate experiments under null hypothesis
18 si mulate d_resu lts = np . random . binomial ( n = n_trials , p =0.5 , size = n_simulations )
19
20 # Determine critical values
21 lower_critical = np . percentile ( simulated_results , 100 * alpha /2)
22 upper_critical = np . percentile ( simulated_results , 100 * (1 - alpha /2) )
23
24 print ( f " Critical region : X <= { lower_critical :.0 f } or X >= { upper_critical :.0 f } " )
25 print ( f " ( based on { n_simulations } simulations ) " )
26

27 # Problem 1: Effect of sample size

28 print ( f " \ nProblem 1: Effect of sample size on critical region " )
29
30 for N in [100 , 200 , 500]:
31 sim_props = np . random . binomial ( n =N , p =0.5 , size = n_simulations ) / N
32 lower_crit_prop = np . percentile ( sim_props , 100 * alpha /2)
33 upper_crit_prop = np . percentile ( sim_props , 100 * (1 - alpha /2) )
34
35 print ( f " N ={ N }: Critical proportion <= { lower_crit_prop :.3 f } or >= { upper_crit_prop :.3 f } " )
36
37 # Problem 2: Exact binomial test
38 print ( f " \ nProblem 2: Exact binomial test " )
39
40 obs er ve d_su cc es se s = 62
41
42 p_value = stats . binomtest (
43 observed_successes ,
44 n = n_trials ,
45 p =0.5 ,
46 alternative = ’two - sided ’
47 )
48
49 print ( f " Observed : { obs er ve d_ su cc es se s } correct out of { n_trials } " )
50 print ( f "P - value : { p_value . pvalue :.4 f } " )
51
52 if p_value . pvalue < alpha :
53 print ( f " Conclusion : Reject null hypothesis ( p < { alpha }) " )
54 print ( " Evidence suggests people guess better than chance " )
55 else :
56 print ( f " Conclusion : Fail to reject null hypothesis ( p >= { alpha }) " )
57 print ( " No significant evidence of better - than - chance guessing " )
Listing 4: Hypothesis Testing using Binomial Test in Python

Explanation

• Simulation-based critical region: Thousands of experiments are simulated under the null hypothesis to construct the
sampling distribution.

• Critical values: The boundaries of the rejection region correspond to the percentiles defined by the significance level α.

• Effect of sample size: Increasing sample size reduces the width of the critical region, showing that larger samples
produce more precise estimates.

• [Link]() performs an exact binomial hypothesis test and returns the corresponding p-value for the observed
data.

7
 8

Program 4: Common Statistical Tests with SciPy and Statsmodels

This program demonstrates various hypothesis tests available in SciPy and statsmodels.
1 import numpy as np
2 import scipy . stats as stats
3 import statsmodels . api as sm
4 from statsmodels . formula . api import ols
5
6 print ( " --- Common Statistical Tests ---" )
7
8 # Generate sample data
9 np . random . seed (42)
10 group1 = np . random . normal (100 , 10 , 30)
11 group2 = np . random . normal (105 , 10 , 30) # Slightly different mean
12 group3 = np . random . normal (100 , 10 , 30)
13
14 # 1. Independent t - test
15 print ( " \ n1 . Independent t - test : " )
16 t_stat , p_value = stats . ttest_ind ( group1 , group2 )
17 print ( f " t - statistic : { t_stat :.4 f } " )
18 print ( f " p - value : { p_value :.4 f } " )
19 print ( f " Conclusion : { ’ Significant difference ’ if p_value < 0.05 else ’ No significant difference ’} " )
20
21 # 2. Mann - Whitney U test
22 print ( " \ n2 . Mann - Whitney U test : " )
23 u_stat , p_value = stats . mannwhitneyu ( group1 , group2 , alternative = ’two - sided ’)
24 print ( f " U - statistic : { u_stat :.4 f } " )
25 print ( f " p - value : { p_value :.4 f } " )
26
27 # 3. Chi - square goodness - of - fit test
28 print ( " \ n3 . Chi - square goodness - of - fit test : " )
29 observed = [25 , 35 , 40]
30 expected = [33.3 , 33.3 , 33.4]
31
32 chi2_stat , p_value = stats . chisquare ( observed , expected )
33 print ( f " Chi - square statistic : { chi2_stat :.4 f } " )
34 print ( f " p - value : { p_value :.4 f } " )
35
36 # 4. Correlation tests
37 print ( " \ n4 . Correlation tests : " )
38 x = np . random . normal (0 , 1 , 50)
39 y = x * 0.7 + np . random . normal (0 , 0.5 , 50)
40
41 pearson_r , p_pearson = stats . pearsonr (x , y )
42 spearman_r , p_spearman = stats . spearmanr (x , y )
43
44 print ( f " Pearson correlation : r ={ pearson_r :.4 f } , p ={ p_pearson :.4 f } " )
45 print ( f " Spearman correlation : rho ={ spearman_r :.4 f } , p ={ p_spearman :.4 f } " )
46
47 # 5. One - way ANOVA using statsmodels
48 print ( " \ n5 . One - way ANOVA : " )
49
50 import pandas as pd
51

52 df_anova = pd . DataFrame ({
53 ’ value ’: np . concatenate ([ group1 , group2 , group3 ]) ,
54 ’ group ’: [ ’A ’ ]*30 + [ ’B ’ ]*30 + [ ’C ’ ]*30
55 })
56
57 model = ols ( ’ value ~ group ’ , data = df_anova ) . fit ()
58 anova_table = sm . stats . anova_lm ( model , typ =2)
59
60 print ( anova_table )
Listing 5: Common Statistical Tests in Python

Explanation

• [Link] ind() performs an independent samples t-test to compare the means of two independent groups.

• [Link]() is a non-parametric alternative to the t-test used when the data does not meet normality assump-
tions.

• [Link]() performs a chi-square goodness-of-fit test to compare observed frequencies with expected frequencies.

• [Link]() calculates the Pearson correlation coefficient and its p-value, measuring linear relationships between
variables.

• [Link]() computes the Spearman rank correlation coefficient, which measures monotonic relationships and is
more robust to outliers.

• statsmodels provides advanced statistical modeling tools. The OLS function fits a linear model, and anova lm() generates
an ANOVA table for testing differences among multiple groups.

4.3 Regression and Trend Analysis using Statsmodels

Based on the reference materials, this section introduces regression and trend analysis along with their theoretical foundations
and practical implementation using the statsmodels library in Python.

8
 9

4.3.1 Theoretical Foundations

What is Regression Analysis?
Regression analysis is a statistical method used for modeling the relationship between a dependent variable (target) and
one or more independent variables (predictors). It helps us understand how the dependent variable changes when one of the
independent variables changes while the others remain fixed.
Key Concepts

• Dependent Variable (Y ): The outcome variable that we want to predict or explain.

• Independent Variables (X): Predictor variables used to explain variation in the dependent variable.

• Regression Coefficients (β): Parameters representing the change in Y for a one-unit change in X.

• Residuals (ϵ): The difference between observed and predicted values.

Ordinary Least Squares (OLS) Regression

Ordinary Least Squares (OLS) is the most widely used method for estimating regression parameters. It estimates the regression
coefficients by minimizing the sum of squared residuals.
Mathematical Formulation
For simple linear regression with one predictor:

Yi = β0 + β1 Xi + ϵi
where

• β0 is the intercept

• β1 is the slope coefficient

• ϵi is the random error term

For multiple linear regression with k predictors:

Yi = β0 + β1 Xi1 + β2 Xi2 + · · · + βk Xik + ϵi

The OLS estimator minimizes the sum of squared residuals:
n
X
(Yi − Ŷi )2
i=1
n 
X 2
= Yi − (β̂0 + β̂1 Xi1 + · · · + β̂k Xik )
i=1

Key Regression Statistics

Several statistical measures are used to evaluate the performance and reliability of a regression model.

• R-squared (R2 )

SSres
R2 = 1 −
SStot
Measures the proportion of variance in the dependent variable explained by the regression model. Values range from 0 to
1.

• Adjusted R-squared

2 n−1
Radj = 1 − (1 − R2 )
n−k−1
Adjusted R2 penalizes the model for including too many predictors and provides a more reliable measure when multiple
variables are used.

• F-statistic

M Sreg
F =
M Sres
Tests whether the regression model explains a significant amount of variance in the dependent variable.

• t-statistic

β̂j
t=
SE(β̂j )

Used to test whether an individual regression coefficient is significantly different from zero.

• Standard Error of Coefficient

s
σ̂ 2
SE(β̂) = P
(Xi − X̄)2

Measures the precision of the estimated regression coefficient.

9
 10

Model Assumptions
For valid statistical inference, Ordinary Least Squares (OLS) regression relies on several key assumptions:

• Linearity: The relationship between the independent variable(s) X and the dependent variable Y is linear.

• Independence: Observations in the dataset are independent of each other.

• Homoscedasticity: The variance of the residuals remains constant across all levels of the independent variables.

• Normality: Residuals are normally distributed. This assumption is particularly important for statistical inference such
as hypothesis testing and confidence intervals.

Trend Analysis
Trend analysis examines patterns and long-term movements in data over time. It is commonly used in economics, finance, and
time series analysis to identify systematic changes in a variable.
Common Types of Trend Models

• Linear Trend

Yt = β0 + β1 t + εt

This model assumes a constant absolute change in the dependent variable for each unit increase in time.

• Quadratic Trend

Yt = β0 + β1 t + β2 t2 + εt

This model allows for acceleration or deceleration in the trend over time.

Seasonal-Trend Decomposition using LOESS (STL)

Seasonal-Trend Decomposition using LOESS (STL) is a technique used to decompose a time series into three main components.

• Trend (Tt ): Represents the long-term progression of the time series.

• Seasonal (St ): Represents regular patterns that repeat over fixed time intervals.

• Residual (Rt ): Represents the irregular or random component remaining after removing trend and seasonal effects.

The STL decomposition model can be represented as:

Yt = Tt + St + Rt

where:

• Tt represents the trend component

• St represents the seasonal component

• Rt represents the residual (remainder) component

4.3.2 Python Implementation with Statsmodels

Installation and Imports
1 # Installation ( if needed )
2 # pip install statsmodels
3
4 # Required imports
5 import numpy as np
6 import pandas as pd
7 import matplotlib . pyplot as plt
8 import statsmodels . api as sm
9 import statsmodels . formula . api as smf
10 from statsmodels . tsa . seasonal import STL
11 from statsmodels . tsa . arima . model import ARIMA
12 from statsmodels . tsa . forecasting . stl import STLForecast
13
14 # Set visualization style
15 plt . style . use ( ’ seaborn - v0_8 - darkgrid ’)
Listing 6: Installing and Importing Required Libraries

10
 11

Example 1: Simple Linear Regression

This example demonstrates the relationship between education and income.
1 # Create sample dataset
2 np . random . seed (42)
3 n = 100
4 education = np . random . normal (12 , 3 , n ) # Years of education
5 true_intercept = 15000
6 true_slope = 3500
7 error = np . random . normal (0 , 10000 , n )
8 income = true_intercept + true_slope * education + error
9
10 # Create DataFrame
11 df = pd . DataFrame ({
12 ’ education ’: education ,
13 ’ income ’: income
14 })
15
16 # Method 1: Using formula API ( recommended )
17 model_formula = smf . ols ( ’ income ~ education ’ , data = df )
18 results_formula = model_formula . fit ()
19
20 print ( " === Simple Linear Regression Results ( Formula API ) === " )
21 print ( results_formula . summary () . tables [1])
22

23 # Method 2: Using array API

24 X = sm . add_constant ( df [ ’ education ’ ])
25 y = df [ ’ income ’]
26
27 model_array = sm . OLS (y , X )
28 results_array = model_array . fit ()
29
30 print ( " \ n === Coefficient Comparison === " )
31 print ( f " Formula API - Intercept : { results_formula . params [ ’ Intercept ’]:.2 f } " )
32 print ( f " Formula API - Slope : { results_formula . params [ ’ education ’]:.2 f } " )
33 print ( f " Array API - Intercept : { results_array . params [0]:.2 f } " )
34 print ( f " Array API - Slope : { results_array . params [1]:.2 f } " )
35
36 # Calculate predicted values
37 df [ ’ predicted ’] = results_formula . predict ( df )
38 df [ ’ residuals ’] = results_formula . resid
39
40 # Visualization
41 fig , axes = plt . subplots (1 , 2 , figsize =(14 , 5) )
42
43 # Scatter plot with regression line
44 axes [0]. scatter ( df [ ’ education ’] , df [ ’ income ’] , alpha =0.6 , label = ’ Observed ’)
45 axes [0]. plot ( df [ ’ education ’] , df [ ’ predicted ’] , ’r - ’ , label = ’ Fitted line ’ , linewidth =2)
46

47 axes [0]. set_xlabel ( ’ Education ( years ) ’)

48 axes [0]. set_ylabel ( ’ Income ( $ ) ’)
49 axes [0]. set_title ( ’ Simple Linear Regression : Income vs Education ’)
50 axes [0]. legend ()
51 axes [0]. grid ( True , alpha =0.3)
52

53 # Residual plot
54 axes [1]. scatter ( df [ ’ predicted ’] , df [ ’ residuals ’] , alpha =0.6)
55 axes [1]. axhline ( y =0 , color = ’r ’ , linestyle = ’ -- ’ , linewidth =1)
56
57 axes [1]. set_xlabel ( ’ Predicted Values ’)
58 axes [1]. set_ylabel ( ’ Residuals ’)
59 axes [1]. set_title ( ’ Residual Plot ’)
60 axes [1]. grid ( True , alpha =0.3)
61
62 plt . tight_layout ()
63 plt . show ()
64

65 # Model diagnostics
66 print ( " \ n === Model Diagnostics === " )
67 print ( f "R - squared : { results_formula . rsquared :.4 f } " )
68 print ( f " Adjusted R - squared : { results_formula . rsquared_adj :.4 f } " )
69 print ( f "F - statistic : { results_formula . fvalue :.2 f } " )
70 print ( f "F - statistic p - value : { results_formula . f_pvalue :.4 e } " )
71 print ( f " Mean of residuals : { results_formula . resid . mean () :.4 e } " )
Listing 7: Simple Linear Regression using Statsmodels

Expected Output Interpretation

• The slope coefficient (approximately 3500) indicates that each additional year of education is associated with roughly
$3500 higher income.

• The intercept represents the estimated income when education is zero (mainly used for model positioning).

• The R2 value around 0.50–0.60 suggests that education explains approximately 50–60% of the variation in income.

• The residual plot helps assess assumptions such as homoscedasticity and independence.

• The mean of residuals should be close to zero, confirming that the model provides unbiased estimates.

Example 2: Multiple Regression with Non-linear Terms

This example demonstrates how to model non-linear relationships in multiple regression using quadratic terms.

11
 12

1 # Extend the dataset with age

2 np . random . seed (42)
3 n = 500
4 education = np . random . normal (13 , 3 , n )
5 age = np . random . uniform (25 , 65 , n )
6

7 # True relationship : income increases with age but at decreasing rate

8 true_income = ( -50000 + 3500 * education +
9 2500 * age - 25 * age **2 +
10 np . random . normal (0 , 15000 , n ) )
11
12 df_multi = pd . DataFrame ({
13 ’ education ’: education ,
14 ’ age ’: age ,
15 ’ income ’: true_income
16 })
17
18 # Create quadratic terms
19 df_multi [ ’ age_squared ’] = df_multi [ ’ age ’] ** 2
20 df_multi [ ’ ed ucatio n_squa red ’] = df_multi [ ’ education ’] ** 2
21
22 # Multiple regression model
23 model_multi = smf . ols ( ’ income ~ education + age + age_squared ’ , data = df_multi )
24 results_multi = model_multi . fit ()
25
26 print ( " === Multiple Regression with Quadratic Terms === " )
27 print ( results_multi . summary () . tables [1])
Listing 8: Multiple Regression with Quadratic Terms
Prediction and Visualization
1 # Predictions for visualization
2 age_range = pd . DataFrame ({
3 ’ age ’: np . linspace (25 , 65 , 50) ,
4 ’ age_squared ’: np . linspace (25 , 65 , 50) **2 ,
5 ’ education ’: [12] * 50 # Hold education constant at 12 years
6 })
7
8 # Predict income for each age
9 age_range [ ’ predicted_income ’] = results_multi . predict ( age_range )
10
11 # Calculate mean income by age group
12 df_multi [ ’ age_group ’] = pd . cut ( df_multi [ ’ age ’] , bins =20)
13 mean_by_age = df_multi . groupby ( ’ age_group ’) [ ’ income ’ ]. mean ()
14 age_centers = [( interval . left + interval . right ) /2 for interval in mean_by_age . index ]
15
16 # Visualization
17 plt . figure ( figsize =(10 , 6) )
18 plt . scatter ( df_multi [ ’ age ’] , df_multi [ ’ income ’] , alpha =0.3 , label = ’ Observed data ’)
19 plt . plot ( age_centers , mean_by_age . values , ’ bo ’ , markersize =8 , label = ’ Mean by age group ’)
20 plt . plot ( age_range [ ’ age ’] , age_range [ ’ predicted_income ’] , ’r - ’ , linewidth =3 ,
21 label = ’ Quadratic fit ( educ =12) ’)
22 plt . xlabel ( ’ Age ( years ) ’)
23 plt . ylabel ( ’ Income ( $ ) ’)
24 plt . title ( ’ Multiple Regression : Non - linear Age Effect ’)
25 plt . legend ()
26 plt . grid ( True , alpha =0.3)
27 plt . show ()
Listing 9: Predictions and Visualization
Interpreting the Quadratic Effect
1 # Interpret quadratic effect
2 age_coef = results_multi . params [ ’ age ’]
3 age2_coef = results_multi . params [ ’ age_squared ’]
4

5 # Find age where income peaks ( first derivative = 0)

6 peak_age = - age_coef / (2 * age2_coef )
7 print ( f " Income peaks at age : { peak_age :.1 f } years " )
8 print ( f " Age coefficient : { age_coef :.2 f } ( positive - initial increase ) " )
9 print ( f " A g e coefficient : { age2_coef :.2 f } ( negative - diminishing returns ) " )
10

11 # Residual diagnostics
12 residuals = results_multi . resid
13 fitted = results_multi . fittedvalues
14
15 fig , axes = plt . subplots (1 , 2 , figsize =(14 , 5) )
16

17 # Residuals vs fitted
18 axes [0]. scatter ( fitted , residuals , alpha =0.6)
19 axes [0]. axhline ( y =0 , color = ’r ’ , linestyle = ’ -- ’)
20 axes [0]. set_xlabel ( ’ Fitted Values ’)
21 axes [0]. set_ylabel ( ’ Residuals ’)
22 axes [0]. set_title ( ’ Residuals vs Fitted ’)
23 axes [0]. grid ( True , alpha =0.3)
24
25 # Q - Q plot for normality
26 sm . qqplot ( residuals , line = ’s ’ , ax = axes [1])
27 axes [1]. set_title ( ’Q - Q Plot ( Normality Check ) ’)
28

29 plt . tight_layout ()
30 plt . show ()
Listing 10: Quadratic Effect Interpretation and Residual Diagnostics
Interpretation:
• The positive coefficient for age indicates income initially increases with age.

12
 13

• The negative coefficient for age squared shows diminishing returns, leading to a peak in income at a certain age.

• Using the formula peak age = − 2βageβage

squared
, the model estimates the age at which income is highest.

• Residual plots and Q-Q plots are used to check model assumptions: homoscedasticity and normality of residuals.

Example 3: Interaction Effects

Interaction terms capture situations where the effect of one variable depends on the level of another variable.
1 # Create dataset with interaction
2 np . random . seed (42)
3 n = 300
4 education = np . random . normal (13 , 3 , n )
5 experience = np . random . uniform (0 , 40 , n )
6
7 # True model : education effect is stronger with more experience
8 true_income = (25000 + 2000 * education + 500 * experience +
9 150 * education * experience + # Interaction term
10 np . random . normal (0 , 8000 , n ) )
11

12 df_interact = pd . DataFrame ({
13 ’ education ’: education ,
14 ’ experience ’: experience ,
15 ’ income ’: true_income
16 })
17

18 # Model with interaction

19 model_interact = smf . ols ( ’ income ~ education * experience ’ , data = df_interact )
20 # Equivalent to : income ~ education + experience + education : experience
21 results_interact = model_interact . fit ()
22
23 print ( " === Regression with Interaction Term === " )
24 print ( results_interact . summary () . tables [1])
Listing 11: Regression with Interaction Term

Visualization of Interaction Effects

1 education_levels = [10 , 13 , 16] # Low , medium , high education
2 experience_range = np . linspace (0 , 40 , 50)
3
4 plt . figure ( figsize =(10 , 6) )
5

6 for edu in education_levels :

7 # Create prediction data
8 pred_data = pd . DataFrame ({
9 ’ education ’: [ edu ] * len ( experience_range ) ,
10 ’ experience ’: experience_range
11 })
12 pred_data [ ’ education : experience ’] = pred_data [ ’ education ’] * pred_data [ ’ experience ’]
13
14 # Predict income
15 pred_income = ( results_interact . params [ ’ Intercept ’] +
16 results_interact . params [ ’ education ’] * edu +
17 results_interact . params [ ’ experience ’] * experience_range +
18 results_interact . params [ ’ education : experience ’] * edu * experience_range )
19
20 plt . plot ( experience_range , pred_income , linewidth =2 ,
21 label = f ’ Education = { edu } years ’)
22
23 plt . xlabel ( ’ Experience ( years ) ’)
24 plt . ylabel ( ’ Predicted Income ( $ ) ’)
25 plt . title ( ’ Interaction Effect : Education Modifies Experience Effect ’)
26 plt . legend ()
27 plt . grid ( True , alpha =0.3)
28 plt . show ()
Listing 12: Visualizing Interaction Effects

Interpretation of Interaction Effect

1 print ( " \ n === Interaction Interpretation === " )
2 print ( " The effect of experience on income depends on education level : " )
3 for edu in education_levels :
4 effect = results_interact . params [ ’ experience ’] + results_interact . params [ ’ education : experience ’] *
edu
5 print ( f " At { edu } years education : 1 year experience $ { effect :.0 f } income increase " )
Listing 13: Quantifying Interaction Effect

• The positive coefficient for the interaction term (education:experience) indicates that the effect of experience on income
grows as education increases.

• For example, at 10 years of education, one additional year of experience increases income by a smaller amount compared
to 16 years of education.

• The plot shows how income curves steepen for higher education levels, demonstrating that education amplifies the return
to experience.

Example 4: STL Decomposition for Trend Analysis

This example demonstrates Seasonal-Trend decomposition using LOESS (STL) to analyze long-term trends and seasonal
patterns in time series data.

13
 14

1 # Load CO2 dataset ( monthly atmospheric CO2 measurements )

2 from statsmodels . datasets import co2
3
4 data = co2 . load () . data
5 co2_series = data [ ’ co2 ’]
6 co2_series . index = pd . date_range ( ’ 1958 -03 -29 ’ , periods = len ( co2_series ) , freq = ’W - WED ’)
7
8 # Fill missing values
9 co2_series = co2_series . fillna ( co2_series . interpolate () )
10
11 print ( " === CO2 Data Summary === " )
12 print ( f " Date range : { co2_series . index [0]} to { co2_series . index [ -1]} " )
13 print ( f " Number of observations : { len ( co2_series ) } " )
14 print ( f " Mean CO2 : { co2_series . mean () :.2 f } ppm " )
15 print ( f " Std CO2 : { co2_series . std () :.2 f } ppm " )
16
17 # STL Decomposition
18 # Seasonal period = 52 for weekly data with annual cycle
19 stl = STL ( co2_series , period =52 , robust = True )
20 result = stl . fit ()
21
22 # Plot decomposition
23 fig = result . plot ()
24 fig . set_size_inches (12 , 10)
25 plt . suptitle ( ’ STL Decomposition of Atmospheric CO2 ’ , fontsize =16)
26 plt . tight_layout ()
27 plt . show ()
28
29 # Extract components
30 trend = result . trend
31 seasonal = result . seasonal
32 residual = result . resid
33
34 print ( " \ n === STL Decomposition Results === " )
35 print ( f " Trend component variance : { trend . var () :.2 f } " )
36 print ( f " Seasonal component variance : { seasonal . var () :.2 f } " )
37 print ( f " Residual component variance : { residual . var () :.2 f } " )
38 print ( f " Proportion explained by trend : { trend . var () / co2_series . var () :.3 f } " )
39 print ( f " Proportion explained by seasonality : { seasonal . var () / co2_series . var () :.3 f } " )
40
41 # Analyze the trend
42 print ( f " \ nTrend start (1958) : { trend . iloc [0]:.2 f } ppm " )
43 print ( f " Trend end (2001) : { trend . iloc [ -1]:.2 f } ppm " )
44 total_increase = trend . iloc [ -1] - trend . iloc [0]
45 years = ( co2_series . index [ -1] - co2_series . index [0]) . days / 365.25
46 print ( f " Total increase over { years :.1 f } years : { total_increase :.2 f } ppm " )
47 print ( f " Average annual increase : { total_increase / years :.2 f } ppm / year " )
48

49 # Check residual patterns

50 plt . figure ( figsize =(12 , 4) )
51 plt . plot ( residual . index , residual . values , ’b - ’ , alpha =0.7)
52 plt . axhline ( y =0 , color = ’r ’ , linestyle = ’ -- ’)
53 plt . axhline ( y =2* residual . std () , color = ’ gray ’ , linestyle = ’: ’ , label = ’ 2 ’)
54 plt . axhline ( y = -2* residual . std () , color = ’ gray ’ , linestyle = ’: ’)
55 plt . xlabel ( ’ Year ’)
56 plt . ylabel ( ’ Residual ( ppm ) ’)
57 plt . title ( ’ STL Residuals - Should Show No Pattern ’)
58 plt . legend ()
59 plt . grid ( True , alpha =0.3)
60 plt . show ()
Listing 14: STL Decomposition of CO2 Data
Interpretation of STL Decomposition:
• The STL decomposition separates the CO2 time series into trend, seasonal, and residual components: Yt = Tt +St +Rt .

• Trend Component: Captures the long-term upward progression of atmospheric CO2. In this dataset, CO2 increased
from approximately 315 ppm (1958) to 370 ppm (2001), averaging ∼ 1.1 ppm/year.

• Seasonal Component: Shows regular yearly fluctuations in CO2 concentrations.

• Residual Component: Should contain no systematic patterns; it represents random variation after removing trend and
seasonality. The residual plot confirms minimal autocorrelation.

• Variance explained: The trend and seasonal components account for the majority of variation in the CO2 series, while
residuals are small.

Example 5: Forecasting with STL and ARIMA

This example demonstrates how to combine STL decomposition with ARIMA modeling for forecasting seasonal time series
data, using electrical equipment production as an example.
1 # Load dataset
2 from statsmodels . datasets import elec_equip as ds
3 elec_equip = ds . load () . data
4 elec_equip . index = pd . date_range ( ’ 1995 -01 -01 ’ , periods = len ( elec_equip ) , freq = ’M ’)
5 elec_equip . columns = [ ’ production ’]
6
7 # STL + ARIMA forecasting
8 from statsmodels . tsa . seasonal import STL
9 from statsmodels . tsa . forecasting . stl import STLForecast
10 from statsmodels . tsa . arima . model import ARIMA
11
12 stlf = STLForecast (
13 elec_equip [ ’ production ’] ,

14
 15

14 ARIMA ,
15 model_kwargs = dict ( order =(1 ,1 ,1) , trend = ’t ’)
16 )
17 stlf_res = stlf . fit ()
18
19 forecast_steps = 24
20 forecast = stlf_res . forecast ( forecast_steps )
21
22 # Visualization
23 plt . figure ( figsize =(14 ,8) )
24 plt . plot ( elec_equip . index [ -60:] , elec_equip [ ’ production ’ ][ -60:] , ’b - ’ , label = ’ Historical ’ , linewidth =2)
25 plt . plot ( forecast . index , forecast , ’r - ’ , label = ’ Forecast ’ , linewidth =2)
26 plt . xlabel ( ’ Date ’) ; plt . ylabel ( ’ Production ’)
27 plt . title ( ’ STL + ARIMA Forecast of Electrical Equipment Production ’)
28 plt . legend () ; plt . grid ( True , alpha =0.3)
29 plt . show ()
Listing 15: STL + ARIMA Forecasting

Key Points:

• STL decomposes the series into trend, seasonal, and residual components.

• ARIMA models the deseasonalized (trend + residual) component.

• The final forecast is the sum of ARIMA predictions and seasonal component.

• Model evaluation: AIC, BIC, and log-likelihood help assess fit.

Example 6: Multiple Seasonal Decomposition (MSTL)

For time series with multiple seasonal periods, MSTL can decompose trend, multiple seasonal components, and residuals.
1 # Create synthetic time series with multiple seasonalities
2 import numpy as np ; import pandas as pd
3 from statsmodels . tsa . seasonal import MSTL
4
5 np . random . seed (42)
6 t = np . arange (1 , 2000)
7 h ou rl y_ se as on al it y = 5 * np . sin (2* np . pi * t /24)
8 w ee kl y_ se as on al it y = 10 * np . sin (2* np . pi * t /(24*7) )
9 trend = 0.001 * t **1.5
10 noise = np . random . normal (0 ,2 , len ( t ) )
11 y = trend + hou rl y_ se as on al it y + we ekl y_ se as on al it y + noise
12
13 ts_index = pd . date_range ( start = " 2020 -01 -01 " , periods = len ( t ) , freq = " H " )
14 ts_data = pd . Series (y , index = ts_index , name = " value " )
15
16 # MSTL decomposition with daily and weekly periods
17 mstl = MSTL ( ts_data , periods =[24 ,168] , iterate =3)
18 mstl_result = mstl . fit ()
19
20 # Variance decomposition
21 total_var = ts_data . var ()
22 trend_var = mstl_result . trend . var ()
23 daily_var = mstl_result . seasonal [ ’ seasonal_24 ’ ]. var ()
24 weekly_var = mstl_result . seasonal [ ’ seasonal_168 ’ ]. var ()
25 resid_var = mstl_result . resid . var ()
26

27 print ( f " Trend : { trend_var / total_var :.3 f } " )

28 print ( f " Daily seasonal : { daily_var / total_var :.3 f } " )
29 print ( f " Weekly seasonal : { weekly_var / total_var :.3 f } " )
30 print ( f " Residual : { resid_var / total_var :.3 f } " )
Listing 16: MSTL Decomposition

Interpretation:

• MSTL separates the time series into trend, multiple seasonalities (daily, weekly), and residuals.

• Variance decomposition quantifies how much of the total variation is explained by each component.

• Useful for high-frequency data (hourly, minute-level) with nested seasonal patterns.

Exploratory Data Analysis (EDA)

1. Introduction to EDA
Exploratory Data Analysis (EDA) is an approach to analyzing datasets to summarize their main characteristics, often using
visual methods. As John Tukey, the pioneer of EDA, stated:

“Exploratory data analysis is detective work—numerical detective work—or graphical detective work.”

Key Distinction: EDA differs from confirmatory data analysis:

• Exploratory (EDA): Detective work—finding clues, generating hypotheses, understanding structure.

• Confirmatory (Inferential): Judicial work—evaluating evidence, testing hypotheses.

15
16

2. The Two Pillars: Descriptive and Inferential Methods in EDA

Descriptive Methods
Descriptive methods summarize features of a dataset and answer “what happened?”.
Purpose:
• Understand data distribution and central tendencies.
• Detect outliers and anomalies.
• Identify patterns and relationships.
• Assess data quality and completeness.
Key Descriptive Measures:
Measure Type Specific Measures Purpose
Central Tendency Mean, Median, Mode Identify typical values
Dispersion Range, Variance, Standard Deviation, IQR Understand spread
Shape Skewness, Kurtosis Understand distribution shape
Position Percentiles, Quartiles Understand relative standing

Inferential Methods
Inferential methods use sample data to make generalizations about populations. In EDA, inferential thinking helps:
• Assess whether observed patterns are likely to be real.
• Quantify uncertainty in exploratory findings.
• Guide hypothesis generation for confirmatory analysis.
Key Concepts: Sampling, uncertainty, and generalizability.

3. Types of Data Analysis Questions

Analysis Type Goal Characteristics
Descriptive Describe a set of data First analysis performed; cannot generalize without modeling
Exploratory Find relationships unknown Discover connections; correlation ̸= causation
Inferential Use sample to infer population Estimate quantity + uncertainty; depends on population and sampling

4. The EDA Process

1. Data Acquisition and Understanding
• Load data and examine structure.
• Identify data types (numerical, categorical, temporal).
2. Data Quality Assessment
• Check missing values.
• Identify outliers.
• Ensure completeness and reliability.
3. Univariate Analysis
• Examine each variable individually.
• Summary statistics: mean, median, mode, standard deviation.
• Visualizations: histograms, box plots.
4. Bivariate/Multivariate Analysis
• Explore relationships between variables.
• Correlation analysis, scatter plots, heatmaps.
5. Generate Hypotheses
• Formulate questions for further investigation.
• Consider sampling and generalizability.

5. Descriptive Statistics Detailed

Measures of Central Tendency:
• Mode: Most frequent value; only measure for nominal data.
• Mean: Arithmetic average P P
X x
Population mean: µ = , Sample mean: X̄ =
N n
• Median: Middle value; robust to outliers.
Measures of Dispersion:
• Range: Max − Min
• Variance: Average squared deviation from mean
√
• Standard deviation: SD = Variance
• Interquartile Range (IQR): Q3 − Q1

16
 17

6. Inferential Statistics in EDA Context

Key Concepts:

• Population vs. Sample: Entire group vs. subset.

• Random Variable: Value determined by chance.

Sampling Strategies:

• Random sampling: Equal chance for each member.

• Stratified sampling: Proportional representation of subgroups.

• Cluster sampling: Random selection of entire clusters.

• Convenience sampling: Easily accessible individuals.

Goal: Make inferences from a sample about a larger population, estimating both value and uncertainty.

7. Importance of EDA Before Modeling

• Detect patterns, outliers, and data quality issues.

• Ensure models are built on reliable foundations.

• Generate hypotheses for further analysis.

8. Summary: Descriptive vs. Inferential in EDA

Aspect Descriptive Methods Inferential Thinking
Purpose Summarize sample data Consider generalization beyond sample
Application Univariate/bivariate analysis Hypothesis generation
Tools Mean, median, plots, tables Sampling concepts, uncertainty assessment
Output Summary statistics, visualizations Questions for confirmatory analysis

4.5 Automation of EDA Workflows Using Python

1. Introduction to EDA Automation
Exploratory Data Analysis (EDA) is the critical first step in any data analysis project. Automating EDA workflows allows data
scientists to quickly understand datasets, detect patterns, and make decisions about further analysis without writing repetitive
code for each new dataset.
Core Philosophy: Automation of EDA means creating reusable workflows that systematically explore data, generate
summaries, and produce visualizations with minimal manual intervention. This aligns with the “detective work” approach to
understanding data before formal modeling.

2. The Python Ecosystem for EDA Automation

Python provides a rich set of libraries for automating EDA:
Library Purpose in EDA Automation Reference
pandas Core data structures (DataFrame, Series); data loading, cleaning, manipulation McKinney
NumPy Numerical computations; array operations; mathematical functions VanderPlas
Matplotlib Foundation for data visualization; customizable plots VanderPlas
Seaborn Statistical visualizations; high-level interface for attractive plots VanderPlas
IPython/Jupyter Interactive computing environment; iterative exploration VanderPlas

3. Key Components of Automated EDA

3.1 Data Loading and Initial Inspection
Automated EDA begins with programmatically loading data from various sources and performing initial inspection. Tasks to
automate include:

• Display first/last rows of data

• Check data types of each column

• Identify missing values

• Assess dataset dimensions

3.2 Data Quality Assessment

Evaluate data quality systematically:

• Missing value detection: identify columns with null values and calculate percentages

• Duplicate detection: report duplicate rows

• Data type validation: ensure columns have appropriate types

• Outlier identification: flag potential outliers using IQR or z-score

17
18

3.3 Univariate Analysis Automation

Generate descriptive statistics and visualizations for individual variables:

Descriptive statistics:

• Measures of central tendency: mean, median, mode

• Measures of dispersion: standard deviation, variance, range, IQR

• Shape statistics: skewness, kurtosis

• Percentiles and quartiles

Automated visualizations:

• Histograms for distribution

• Box plots for outlier detection

• Bar charts for categorical variables

3.4 Bivariate and Multivariate Analysis Automation

Explore relationships between variables:

• Correlation matrices: compute and visualize correlations

• Pair plots: scatterplot matrices

• Cross-tabulations: for categorical relationships

• Grouped summaries: statistics by categorical groups

4. The EDA Automation Workflow

1. Data Acquisition: Read data from CSV, Excel, SQL, APIs; handle file formats and encodings.

2. Data Cleaning: Handle missing values, remove duplicates, standardize data types.

3. Univariate Analysis: Generate summary statistics; create appropriate visualizations; flag potential issues.

4. Bivariate Analysis: Analyze relationships; generate correlation matrices, scatter plots, cross-tabs.

5. Report Generation: Compile findings into a structured report with key visualizations and patterns.

5. Benefits of EDA Automation

Benefit Description
Efficiency Eliminates repetitive coding for each new dataset
Consistency Ensures thorough analysis across projects
Reproducibility Creates documented, repeatable workflows
Scalability Handles large datasets efficiently
Collaboration Standardized outputs facilitate teamwork

6. Integration with Other Data Science Libraries

Automated EDA integrates with Python libraries for advanced analysis:

• SciPy: Statistical tests and advanced math functions

• scikit-learn: Machine learning preprocessing and modeling

• statsmodels: Statistical modeling and hypothesis testing

7. Practical Considerations for Automation

• Flexibility: Customization based on data types and goals

• Scalability: Efficient handling of datasets of varying sizes

• Extensibility: Easily add new analysis components

• Interpretability: Outputs should be clear and shareable

18
 19

8. Summary
Automation of EDA workflows using Python leverages libraries such as pandas, NumPy, Matplotlib, and Seaborn to system-
atically explore datasets. This approach embodies the “detective work” philosophy of EDA while making the process efficient,
reproducible, and scalable for real-world data analysis tasks. listings xcolor
1 import pandas as pd
2 import numpy as np
3 import matplotlib . pyplot as plt
4 import seaborn as sns
5 from scipy import stats
6 import warnings
7 warnings . fil terwarni ngs ( ’ ignore ’)
8
9 plt . style . use ( ’ seaborn - v0_8 - darkgrid ’)
10 sns . set_palette ( " husl " )
11
12 class AutomatedEDA :
13 def __init__ ( self , df , target = None ) :
14 self . df = df . copy ()
15 self . target = target
16 self . numeric_cols = df . select_dtypes ( include =[ np . number ]) . columns . tolist ()
17 self . c a t e g o r i c a l _ c o l s = df . select_dtypes ( include =[ ’ object ’ , ’ category ’ ]) . columns . tolist ()
18 self . datetime_cols = df . select_dtypes ( include =[ ’ datetime64 ’ ]) . columns . tolist ()
19
20 def r u n_ c o m p l e t e _ e d a ( self ) :
21 print ( " = " *70)
22 print ( " AUTOMATED EXPLORATORY DATA ANALYSIS REPORT " )
23 print ( " = " *70)
24 self . data_overview ()
25 self . d a t a _ q u a l i t y _ c h e c k ()
26 self . u n i v a r i a t e _ a n a l y s i s ()
27 self . b i v a r i a t e _ a n a l y s i s ()
28 self . g e n e r a t e _ s u m m a r y ()
29
30 def data_overview ( self ) :
31 print ( " \ n1 . DATASET OVERVIEW " )
32 print ( " -" *40)
33 print ( f " Dataset Shape : { self . df . shape [0]} rows { self . df . shape [1]} columns " )
34 print ( f " \ nFirst 5 rows : " )
35 print ( self . df . head () )
36 print ( f " \ nLast 5 rows : " )
37 print ( self . df . tail () )
38 print ( f " \ nColumn Names : " )
39 for col in self . df . columns :
40 print ( f " { col } " )
41
42 def d a t a _ q u a l i t y _ c h e c k ( self ) :
43 print ( " \ n2 . DATA QUALITY ASSESSMENT " )
44 print ( " -" *40)
45 print ( " \ n2 .1 Data Types : " )
46 dtype_counts = self . df . dtypes . value_counts ()
47 for dtype , count in dtype_counts . items () :
48 print ( f " { dtype }: { count } columns " )
49
50 print ( " \ n2 .2 Missing Values : " )
51 missing = self . df . isnull () . sum ()
52 missing_pct = ( missing / len ( self . df ) ) * 100
53 missing_df = pd . DataFrame ({ ’ Missing_Count ’: missing , ’ Mi ss in g _P er ce n t ’: missing_pct }) . sort_values ( ’ Mi ss in g _P er ce n t ’ ,
ascending = False )
54 m i ss i n g _ w i t h _ d a t a = missing_df [ missing_df [ ’ Missing_Count ’] > 0]
55 if len ( m i s s i n g _ w i t h _ d a t a ) > 0:
56 print ( m i s s i n g _ w i t h _ d a t a )
57 print ( f " \ nTotal Missing Values : { self . df . isnull () . sum () . sum () } " )
58 else :
59 print ( " No missing values found " )
60
61 print ( " \ n2 .3 Duplicate Rows : " )
62 duplicates = self . df . duplicated () . sum ()
63 if duplicates > 0:
64 print ( f " Found { duplicates } duplicate rows ({ duplicates / len ( self . df ) *100:.2 f }%) " )
65 else :
66 print ( " No duplicate rows found " )
67
68 print ( " \ n2 .4 Memory Usage : " )
69 memory_usage = self . df . memory_usage ( deep = True ) . sum () / 1024**2
70 print ( f " Total memory : { memory_usage :.2 f } MB " )
71
72 def u n i v a r i a t e _ a n a l y s i s ( self ) :
73 print ( " \ n3 . UNIVARIATE ANALYSIS " )
74 print ( " -" *40)
75 if self . numeric_cols :
76 print ( " \ n3 .1 Numeric Variables Summary : " )
77 print ( self . df [ self . numeric_cols ]. describe () . round (3) )
78 print ( " \ nShape Statistics : " )
79 for col in self . numeric_cols [:5]:
80 data = self . df [ col ]. dropna ()
81 skewness = stats . skew ( data )
82 kurt = stats . kurtosis ( data )
83 print ( f " \ n { col }: " )
84 print ( f " Skewness : { skewness :.3 f } " )
85 print ( f " Kurtosis : { kurt :.3 f } " )
86 print ( f " Range : { data . max () - data . min () :.3 f } " )
87 print ( f " IQR : { data . quantile (0.75) - data . quantile (0.25) :.3 f } " )
88
89 if self . c a t e g o r i c a l _ c o l s :
90 print ( " \ n3 .2 Categorical Variables Summary : " )
91 for col in self . ca t e g o r i c a l _ c o l s [:5]:
92 print ( f " \ n { col }: " )
93 print ( f " Unique values : { self . df [ col ]. nunique () } " )
94 print ( f " Top value : { self . df [ col ]. mode () [0]} " )
95 print ( f " Top frequency : { self . df [ col ]. value_counts () . iloc [0]} " )
96
97 self . pl o t_ un iv a ri at e ()
98
99 def pl ot_ un iv a ri at e ( self ) :
100 n_numeric = len ( self . numeric_cols )
101 n_categorical = len ( self . c a t e g o r i c a l _ c o l s )
102
103 if n_numeric > 0:
104 n_cols = min (3 , n_numeric )
105 n_rows = ( min ( n_numeric , 6) + n_cols - 1) // n_cols
106 fig , axes = plt . subplots ( n_rows , n_cols , figsize =(5* n_cols , 4* n_rows ) )
107 axes = axes . flatten () if n_rows > 1 else [ axes ] if n_cols == 1 else axes
108 for idx , col in enumerate ( self . numeric_cols [:6]) :
109 if idx < len ( axes ) :
110 self . df [ col ]. hist ( ax = axes [ idx ] , bins =30 , alpha =0.7 , color = ’ steelblue ’ , edgecolor = ’ black ’)
111 axes [ idx ]. axvline ( self . df [ col ]. mean () , color = ’ red ’ , linestyle = ’ -- ’ , label = ’ Mean ’)
112 axes [ idx ]. axvline ( self . df [ col ]. median () , color = ’ green ’ , linestyle = ’ - ’ , label = ’ Median ’)
113 axes [ idx ]. set_title ( f ’ Distribution of { col } ’)

19
 20

114 axes [ idx ]. legend ()

115 plt . suptitle ( ’ Univariate Analysis - Numeric Variables ’ , fontsize =14)
116 plt . tight_layout ()
117 plt . show ()
118
119 if n_categorical > 0:
120 n_cols = min (3 , n_categorical )
121 n_rows = ( min ( n_categorical , 6) + n_cols - 1) // n_cols
122 fig , axes = plt . subplots ( n_rows , n_cols , figsize =(5* n_cols , 4* n_rows ) )
123 axes = axes . flatten () if n_rows > 1 or n_categorical > 1 else [ axes ]
124 for idx , col in enumerate ( self . c a t e g o r i c a l _ c o l s [:6]) :
125 if idx < len ( axes ) :
126 value_counts = self . df [ col ]. value_counts () . head (10)
127 value_counts . plot ( kind = ’ bar ’ , ax = axes [ idx ] , color = ’ coral ’ , edgecolor = ’ black ’)
128 axes [ idx ]. set_title ( f ’ Top Categories in { col } ’)
129 axes [ idx ]. set_xlabel ( col )
130 axes [ idx ]. set_ylabel ( ’ Count ’)
131 axes [ idx ]. tick_params ( axis = ’x ’ , rotation =45)
132 plt . suptitle ( ’ Univariate Analysis - Categorical Variables ’ , fontsize =14)
133 plt . tight_layout ()
134 plt . show ()
135
136 def b i v a r i a t e _ a n a l y s i s ( self ) :
137 print ( " \ n4 . BIVARIATE ANALYSIS " )
138 print ( " -" *40)
139
140 if len ( self . numeric_cols ) > 1:
141 print ( " \ n4 .1 Correlation Matrix : " )
142 corr_matrix = self . df [ self . numeric_cols ]. corr ()
143 print ( corr_matrix . round (3) )
144 corr_pairs = []
145 for i in range ( len ( self . numeric_cols ) ) :
146 for j in range ( i +1 , len ( self . numeric_cols ) ) :
147 corr_pairs . append ({ ’ var1 ’: self . numeric_cols [ i ] , ’ var2 ’: self . numeric_cols [ j ] , ’ correlation ’: corr_matrix .
iloc [i , j ]})
148 corr_pairs = sorted ( corr_pairs , key = lambda x : abs ( x [ ’ correlation ’ ]) , reverse = True )
149 print ( " \ nTop 5 Strongest Correlations : " )
150 for pair in corr_pairs [:5]:
151 print ( f " { pair [ ’ var1 ’]} vs { pair [ ’ var2 ’]}: { pair [ ’ correlation ’]:.3 f } " )
152
153 plt . figure ( figsize =(10 , 8) )
154 sns . heatmap ( corr_matrix , annot = True , cmap = ’ coolwarm ’ , center =0 , square = True , linewidths =1 , cbar_kws ={ " shrink " :
0.8})
155 plt . title ( ’ Correlation Matrix Heatmap ’)
156 plt . tight_layout ()
157 plt . show ()
158
159 if self . target and self . target in self . numeric_cols :
160 print ( f " \ n4 .2 Relationships with Target Variable : { self . target } " )
161 features = [ col for col in self . numeric_cols if col != self . target ]
162 fig , axes = plt . subplots (2 , 3 , figsize =(15 , 10) )
163 axes = axes . flatten ()
164 for idx , feature in enumerate ( features [:6]) :
165 if idx < len ( axes ) :
166 axes [ idx ]. scatter ( self . df [ feature ] , self . df [ self . target ] , alpha =0.5)
167 z = np . polyfit ( self . df [ feature ] , self . df [ self . target ] , 1)
168 p = np . poly1d ( z )
169 x_range = np . linspace ( self . df [ feature ]. min () , self . df [ feature ]. max () , 100)
170 axes [ idx ]. plot ( x_range , p ( x_range ) , ’r - ’ , linewidth =2)
171 corr = self . df [ feature ]. corr ( self . df [ self . target ])
172 axes [ idx ]. set_title ( f ’{ feature } vs { self . target }\ nr = { corr :.3 f } ’)
173 axes [ idx ]. set_xlabel ( feature )
174 axes [ idx ]. set_ylabel ( self . target )
175 plt . suptitle ( f ’ Relationships with Target Variable : { self . target } ’ , fontsize =14)
176 plt . tight_layout ()
177 plt . show ()
178
179 if len ( self . numeric_cols ) <= 5:
180 print ( " \ n4 .3 Pair Plot Analysis : " )
181 sns . pairplot ( self . df [ self . numeric_cols ] , diag_kind = ’ kde ’)
182 plt . suptitle ( ’ Pair Plot of Numeric Variables ’ , y =1.02)
183 plt . show ()
184
185 def g e ne r a t e _ s u m m a r y ( self ) :
186 print ( " \ n5 . EDA SUMMARY REPORT " )
187 print ( " -" *40)
188 print ( " \ n5 .1 Dataset Ch a ra ct er i st ic s : " )
189 print ( f " Total observations : { self . df . shape [0]: ,} " )
190 print ( f " Total features : { self . df . shape [1]} " )
191 print ( f " Numeric features : { len ( self . numeric_cols ) } " )
192 print ( f " Categorical features : { len ( self . c a t e g o r i c a l _ c o l s ) } " )
193 print ( f " DateTime features : { len ( self . datetime_cols ) } " )
194
195 missing_total = self . df . isnull () . sum () . sum ()
196 missing_pct = ( missing_total / ( self . df . shape [0]* self . df . shape [1]) ) *100
197 print ( f " \ n5 .2 Data Quality : " )
198 print ( f " Complete cases : { self . df . dropna () . shape [0]: ,} ({ self . df . dropna () . shape [0]/ self . df . shape [0]*100:.1 f }%) " )
199 print ( f " Missing values : { missing_total : ,} ({ missing_pct :.2 f }% of all cells ) " )
200
201 duplicates = self . df . duplicated () . sum ()
202 if duplicates > 0:
203 print ( f " Duplicate rows : { duplicates : ,} ({ duplicates / self . df . shape [0]*100:.1 f }%) " )
204
205 print ( f " \ n5 .3 Feature Types : " )
206 for col in self . df . columns :
207 dtype = str ( self . df [ col ]. dtype )
208 missing = self . df [ col ]. isnull () . sum ()
209 unique = self . df [ col ]. nunique ()
210 print ( f " { col }: { dtype } , { unique } unique values , { missing } missing " )
211
212 print ( " \ n " + " = " *70)
213 print ( " AUTOMATED EDA COMPLETED SUCCESSFULLY " )
214 print ( " = " *70)
215
216 # --- Example 1: Iris Dataset ---
217 from sklearn . datasets import load_iris
218 iris = load_iris ()
219 df_iris = pd . DataFrame ( iris . data , columns = iris . feature_names )
220 df_iris [ ’ species ’] = pd . Categorical . from_codes ( iris . target , iris . target_names )
221 eda = AutomatedEDA ( df_iris , target = ’ sepal length ( cm ) ’)
222 eda . r u n _ c o m p l e t e_ e d a ()
223
224 # --- Example 2: Titanic Dataset ---
225 np . random . seed (42)
226 n = 500
227 df_titanic = pd . DataFrame ({
228 ’ passenger_id ’: range (1 , n +1) ,
229 ’ survived ’: np . random . choice ([0 ,1] , n , p =[0.6 ,0.4]) ,
230 ’ pclass ’: np . random . choice ([1 ,2 ,3] , n , p =[0.2 ,0.3 ,0.5]) ,
231 ’ name ’: [ f ’ Passenger_ { i } ’ for i in range (1 , n +1) ] ,
232 ’ sex ’: np . random . choice ([ ’ male ’ , ’ female ’] , n , p =[0.6 ,0.4]) ,

20
 21

233 ’ age ’: np . random . normal (30 ,15 , n ) . clip (0 ,80) . round (1) ,
234 ’ sibsp ’: np . random . choice ([0 ,1 ,2 ,3 ,4 ,5] , n , p =[0.6 ,0.2 ,0.1 ,0.05 ,0.03 ,0.02]) ,
235 ’ parch ’: np . random . choice ([0 ,1 ,2 ,3 ,4 ,5] , n , p =[0.7 ,0.15 ,0.08 ,0.04 ,0.02 ,0.01]) ,
236 ’ fare ’: np . random . gamma (5 ,10 , n ) . round (2) ,
237 ’ embarked ’: np . random . choice ([ ’C ’ , ’Q ’ , ’S ’] , n , p =[0.3 ,0.2 ,0.5])
238 })
239 df_titanic . loc [ np . random . choice (n ,50) , ’ age ’] = np . nan
240 df_titanic . loc [ np . random . choice (n ,20) , ’ embarked ’] = np . nan
241 eda2 = AutomatedEDA ( df_titanic , target = ’ survived ’)
242 eda2 . r u n _ c o m p l e t e _ e d a ()
243
244 # --- Example 3: Custom Dataset ---
245 np . random . seed (42)
246 n = 1000
247 df_custom = pd . DataFrame ({
248 ’ customer_id ’: range (1000 ,1000+ n ) ,
249 ’ age ’: np . random . normal (40 ,15 , n ) . clip (18 ,90) . astype ( int ) ,
250 ’ income ’: np . random . gamma (5 ,10000 , n ) . round (0) ,
251 ’ credit_score ’: np . random . normal (700 ,50 , n ) . clip (300 ,850) . astype ( int ) ,
252 ’ p ur ch as e _a mo un t ’: np . random . exponential (100 , n ) . round (2) ,
253 ’ frequency ’: np . random . poisson (5 , n ) ,
254 ’ satisfaction ’: np . random . choice ([1 ,2 ,3 ,4 ,5] , n , p =[0.1 ,0.15 ,0.25 ,0.3 ,0.2]) ,
255 ’ region ’: np . random . choice ([ ’ North ’ , ’ South ’ , ’ East ’ , ’ West ’] , n ) ,
256 ’ member_type ’: np . random . choice ([ ’ Bronze ’ , ’ Silver ’ , ’ Gold ’ , ’ Platinum ’] , n , p =[0.4 ,0.3 ,0.2 ,0.1]) ,
257 ’ last_purchase ’: pd . date_range ( ’ 2023 -01 -01 ’ , periods =n , freq = ’D ’)
258 })
259 eda3 = AutomatedEDA ( df_custom , target = ’ pu r ch as e _a mo un t ’)
260 eda3 . r u n _ c o m p l e t e _ e d a ()

Listing 17: Full AutomatedEDA Python Workflow

21
Chapter-5
Data Visualization and Storytelling

0.1 Principles of Effective Visualization and Dashboard Design

0.1.1 Introduction to Data Visualization
Data visualization is the graphical representation of information and data. By using visual elements like charts, graphs, and
maps, data visualization tools provide an accessible way to see and understand trends, outliers, and patterns in data. Effective
visualization is not just about making “pretty pictures” but about communicating information clearly and efficiently [?, ?].

0.1.2 Why Visualize Data?

Graphics are very effective for communicating numerical information quickly. Many design choices are based on subjective
measures such as personal taste or disciplinary conventions rather than objective criteria. Understanding principles of effective
visualization helps move from subjective to objective design decisions.

0.1.3 Perceptual Principles in Visualization

Preattentive Features
Preattentive features are visual properties processed very rapidly by the human brain without conscious effort. They can
highlight important elements in a visualization.
Key Preattentive Attributes:

• Color hue: Different colors processed preattentively.

• Color intensity: Lighter vs. darker shades.

• Orientation: Horizontal, vertical, diagonal lines.

• Shape: Circles, squares, triangles.

• Size: Larger vs. smaller objects.

• Motion: Moving vs. stationary elements.

• Spatial position: Grouping and proximity.

Gestalt Heuristics
Gestalt principles describe how humans naturally perceive visual elements as organized patterns and wholes.

Principle Description Application in Visual-

ization
Proximity Objects near each other are per- Group related data points
ceived as a group together
Similarity Similar objects are perceived as Use consistent colors for
related same categories
Enclosure Objects with a boundary are seen Use boxes to group related
as a group charts
Closure Mind fills in missing information Use partial borders effec-
tively
Continuity Aligned objects are perceived as Line charts show trends
continuous smoothly
Connection Connected objects are seen as re- Network diagrams, con-
lated nected scatter plots

Table 1: Gestalt Principles in Visualization

0.1.4 Principles of Effective Visualization

Show the Data Clearly
Present data accurately and without distortion:

• Choose appropriate chart types

• Avoid misleading scales

• Ensure high data-ink ratio (minimize non-data ink)

1
2

Maximize Data-Ink Ratio

Edward Tufte’s concept of data-ink ratio:
Data-Ink
Data-Ink Ratio =
Total Ink Used
Remove non-data ink wherever possible, e.g., unnecessary gridlines, borders, or decorations.

Avoid Chartjunk
Chartjunk includes unnecessary visual elements that distract from the data:
• Heavy grid lines
• Unnecessary 3D effects
• Excessive decorations or patterns

Use Appropriate Chart Types

Data Relationship Recommended Chart

Comparison Bar charts, column charts
Trends over time Line charts, area charts
Distribution Histograms, box plots, density plots
Correlation Scatter plots, bubble charts
Composition Pie charts (limited), stacked bars
Geospatial Maps, choropleth maps

Table 2: Recommended Chart Types for Different Data Relationships

Color Usage Principles

• Use color purposefully
• Consider colorblind-friendly palettes
• Sequential palettes for ordered data
• Diverging palettes for data with meaningful midpoint
• Qualitative palettes for categorical data

0.1.5 Dashboard Design Principles

What is a Dashboard?
A dashboard is a visual display of the most important information needed to achieve objectives, consolidated on a single screen
for at-a-glance monitoring.

Key Design Principles

1. Know Your Audience: Understand who will use the dashboard and what decisions they need to make.
2. Establish a Visual Hierarchy: Most important info should be most prominent; use size, position, and color.
3. Maintain Consistency: Consistent colors, fonts, labels, and chart types.
4. Provide Context: Show comparisons, benchmarks, and goals; include explanatory titles.
5. Limit Scope: One dashboard should answer related questions; use tabs for multiple aspects.
6. Ensure Accessibility: Consider screen sizes, colorblind viewers, and readable text.

0.1.6 Common Visualization Mistakes

Mistake Why Problematic Better Approach

3D charts Distorts perception Use 2D charts
Truncated y-axis Exaggerates differences Start axis at zero
Too many colors Confuses viewer Limit color palette
Pie charts with many Hard to compare Use bar charts
slices
Missing labels Context lost Always label axes
Inconsistent scales Misleads comparisons Keep scales consistent

Table 3: Common Visualization Mistakes and Corrections

0.1.7 Storytelling Aspect

Effective visualization tells a story:
• Narrative: Guide the viewer through the data
• Context: Explain why the data matters
• Insight: Highlight key findings
• Call to action: Suggest what to do

2
0.2. VISUALIZATION WITH MATPLOTLIB: LINE, BAR, HISTOGRAM, SCATTER, SUBPLOTS 3

0.1.8 Testing Visualization Effectiveness

• Design choices should be tested with users
• What seems clear to the designer may not be clear to viewers
• Iterative refinement based on feedback improves effectiveness

0.1.9 Summary of Key Principles

Principle Key Takeaway

Preattentive Features Use color, size, position to guide attention automatically
Gestalt Principles Leverage natural perception of groups and patterns
Data-Ink Ratio Maximize information, minimize decoration
Appropriate Charts Match chart type to data and message
Dashboard Design Know audience, establish hierarchy, provide context
Storytelling Guide viewers to insights and action

Table 4: Summary of Visualization and Dashboard Principles

0.2 Visualization with Matplotlib: Line, Bar, Histogram, Scatter, Subplots

0.2.1 Introduction to Matplotlib
Matplotlib is the primary plotting library in Python, designed to create high-quality visualizations. Developed by John Hunter
in 2003, it serves as the foundation for most Python visualization libraries. Matplotlib allows intuitive plotting while providing
complete control over figure appearance [?, ?].

0.2.2 The Two Interfaces: Pyplot vs. Object-Oriented

Pyplot Interface (MATLAB-style)
• Simple, state-based interface
• Good for quick, interactive plotting
• Uses commands like [Link](), [Link](), etc.
• Matplotlib tracks the “current figure” and “current axes”

Object-Oriented Interface
• Provides more control and customization
• Explicit creation of Figure and Axes objects
• Better for complex plots and subplots
• Recommended for production code and reusability

0.2.3 Anatomy of a Matplotlib Figure

Understanding the components of a figure is essential for customization:

• Figure: The overall window or page that contains all elements

• Axes: The plotting area containing x and y axes
• Axis: Number-line objects that set scale and limits
• Artist: Everything visible in the figure (lines, text, patches)

Figure
Title
Axes
Plot Area
X-axis Label
Y-axis Label

0.2.4 Basic Plot Types

Line Plot
• Purpose: Show trends over time or ordered categories
• When to use: Time series data, trends, patterns
• Key parameters: linestyle=’-’, marker=’o’, linewidth=2

Bar Plot
• Purpose: Compare quantities across categories
• Types: Vertical, horizontal, stacked, grouped
• When to use: Frequency counts, categorical comparison, survey responses

3
 4

Histogram
• Purpose: Show distribution of continuous variables

• When to use: Understanding data distribution, normality, skewness

• Key concepts:

– Bins: Intervals to group data

– Frequency: Count of observations
– Density: Normalized area under histogram

Scatter Plot
• Purpose: Show relationship between two continuous variables

• When to use: Correlations, clusters, outliers, data density

• Variations: Bubble charts (size as third variable), colored scatter (categorical data as color)

Subplots
• Purpose: Display multiple plots in one figure

• Layout options: Regular grids ([Link]()), GridSpec, nested subplots

0.2.5 Customization Elements

Element Purpose Common Functions

Title Describe the plot [Link](), [Link] title()
Axis Labels Explain axes [Link](), [Link]()
Legend Identify data series [Link]()
Grid Improve readability [Link]()
Ticks Control axis markings [Link](), [Link]()
Colors Enhance distinction color=’red’, cmap=’viridis’
Annotations Highlight points [Link](), [Link]()

Table 5: Common Plot Customization Elements

0.2.6 Saving Figures

To save a figure in Matplotlib:

[Link](’[Link]’, dpi=300, bbox_inches=’tight’)

Common formats:

• PNG: Raster graphics

• PDF/SVG: Vector graphics

• JPG: Photographs or web images

0.3 Program 1: Line Plots

0.3.1 Introduction
Line plots are primarily used to show trends over time or ordered categories. They are ideal for continuous data, time series, or
comparing multiple sequences. The following examples demonstrate different ways to use line plots in Matplotlib.

0.3.2 Python Code Examples

1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 # Sample data
5 x = np . linspace (0 , 10 , 100)
6 y1 = np . sin ( x )
7 y2 = np . cos ( x )
8 y3 = np . sin ( x ) * np . exp ( - x /3) # Damped sine wave
9
10 # 1. Basic Line Plot
11 plt . figure ( figsize =(10 , 6) )
12 plt . plot (x , y1 , ’b - ’ , linewidth =2 , label = ’ sin ( x ) ’)
13 plt . plot (x , y2 , ’r - - ’ , linewidth =2 , label = ’ cos ( x ) ’)
14 plt . plot (x , y3 , ’g -. ’ , linewidth =2 , label = ’ damped sin ( x ) ’)
15 plt . xlabel ( ’X axis ( radians ) ’)
16 plt . ylabel ( ’Y axis ( amplitude ) ’)
17 plt . title ( ’ Basic Line Plot - Trigonometric Functions ’)
18 plt . legend ()
19 plt . grid ( True , alpha =0.3)
20 plt . show ()
Listing 1: Basic Line Plots with Matplotlib

4
 0.4. PROGRAM 2: BAR PLOTS 5

1 # 2. Line Plot with Markers

2 plt . figure ( figsize =(10 , 6) )
3 x_markers = np . arange (0 , 11 , 1)
4 y_markers = x_markers ** 2
5 plt . plot ( x_markers , y_markers , ’ro - ’ , markersize =8 , markerfacecolor = ’ white ’ ,
6 markeredgewidth =2 , linewidth =2 , label = ’y = x ’)
7 plt . xlabel ( ’X values ’)
8 plt . ylabel ( ’Y values ’)
9 plt . title ( ’ Line Plot with Markers ’)
10 plt . legend ()
11 plt . grid ( True , alpha =0.3)
12 plt . show ()
Listing 2: Line Plot with Markers

1 # 3. Multiple Lines with Different Styles

2 plt . figure ( figsize =(12 , 6) )
3 styles = [ ’ - ’ , ’ -- ’ , ’ -. ’ , ’: ’]
4 for i , style in enumerate ( styles ) :
5 y = ( i +1) * x + np . random . normal (0 , 0.5 , len ( x ) )
6 plt . plot (x , y , style , linewidth =2 , label = f ’ Line { i +1} ({ style }) ’)
7 plt . xlabel ( ’X axis ’)
8 plt . ylabel ( ’Y axis ’)
9 plt . title ( ’ Different Line Styles ’)
10 plt . legend ()
11 plt . grid ( True , alpha =0.3)
12 plt . show ()
Listing 3: Multiple Lines with Different Styles

1 # 4. Fill Between ( Area Plot )

2 plt . figure ( figsize =(10 , 6) )
3 x = np . linspace (0 , 2* np . pi , 100)
4 y_upper = np . sin ( x ) + 0.2
5 y_lower = np . sin ( x ) - 0.2
6 plt . plot (x , np . sin ( x ) , ’b - ’ , linewidth =2 , label = ’ sin ( x ) ’)
7 plt . fill_between (x , y_lower , y_upper , alpha =0.3 , color = ’ blue ’ , label = ’ 0 .2 range ’)
8 plt . xlabel ( ’X axis ’)
9 plt . ylabel ( ’Y axis ’)
10 plt . title ( ’ Line Plot with Confidence Interval ’)
11 plt . legend ()
12 plt . grid ( True , alpha =0.3)
13 plt . show ()
Listing 4: Line Plot with Fill Between (Confidence Interval)

0.3.3 Explanation of Examples

• Basic Line Plot: Demonstrates multiple lines with different colors and styles.

• Line Plot with Markers: Adds markers to highlight individual data points.

• Multiple Lines with Styles: Shows how to plot several lines with varying linestyles and random noise.

• Fill Between (Area Plot): Illustrates how to display a confidence interval or range around a line using fill between.

0.4 Program 2: Bar Plots

0.4.1 Introduction
Bar plots are used to compare quantities across different categories. They are ideal for displaying categorical data, showing
frequency counts, or comparing multiple series. Both vertical and horizontal orientations are possible, and additional elements
such as grouped bars, stacked bars, and error bars can be included for clarity.

0.4.2 Python Code Examples

1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 # Sample data
5 categories = [ ’ Product A ’ , ’ Product B ’ , ’ Product C ’ , ’ Product D ’ , ’ Product E ’]
6 sales_2024 = [52 , 68 , 45 , 85 , 61]
7
8 # 1. Basic Vertical Bar Plot
9 plt . figure ( figsize =(10 , 6) )
10 x_pos = np . arange ( len ( categories ) )
11 plt . bar ( x_pos , sales_2024 , color = ’ steelblue ’ , edgecolor = ’ black ’ , alpha =0.7)
12 plt . xlabel ( ’ Products ’)
13 plt . ylabel ( ’ Sales ( thousands ) ’)
14 plt . title ( ’ Product Sales 2024 ’)
15 plt . xticks ( x_pos , categories , rotation =45)
16 plt . grid ( True , alpha =0.3 , axis = ’y ’)
17 plt . show ()
Listing 5: Basic Vertical Bar Plot

1 # 2. Horizontal Bar Plot

2 plt . figure ( figsize =(10 , 6) )
3 y_pos = np . arange ( len ( categories ) )
4 plt . barh ( y_pos , sales_2024 , color = ’ coral ’ , edgecolor = ’ black ’ , alpha =0.7)
5 plt . ylabel ( ’ Products ’)
6 plt . xlabel ( ’ Sales ( thousands ) ’)

5
 6

7 plt . title ( ’ Product Sales 2024 - Horizontal ’)

8 plt . yticks ( y_pos , categories )
9 plt . grid ( True , alpha =0.3 , axis = ’x ’)
10 plt . show ()
Listing 6: Horizontal Bar Plot

1 # 3. Grouped Bar Plot ( Comparing Two Years )

2 sales_2023 = [45 , 62 , 38 , 71 , 53]
3 x = np . arange ( len ( categories ) )
4 width = 0.35
5 plt . figure ( figsize =(12 , 6) )
6 plt . bar ( x - width /2 , sales_2023 , width , label = ’ 2023 ’ , color = ’ lightblue ’ , edgecolor = ’ black ’)
7 plt . bar ( x + width /2 , sales_2024 , width , label = ’ 2024 ’ , color = ’ steelblue ’ , edgecolor = ’ black ’)
8 plt . xlabel ( ’ Products ’)
9 plt . ylabel ( ’ Sales ( thousands ) ’)
10 plt . title ( ’ Product Sales Comparison : 2023 vs 2024 ’)
11 plt . xticks (x , categories , rotation =45)
12 plt . legend ()
13 plt . grid ( True , alpha =0.3 , axis = ’y ’)
14 plt . show ()
Listing 7: Grouped Bar Plot Comparing Two Years

1 # 4. Stacked Bar Plot

2 plt . figure ( figsize =(10 , 6) )
3 plt . bar ( categories , sales_2023 , label = ’ 2023 ’ , color = ’ lightblue ’ , edgecolor = ’ black ’)
4 plt . bar ( categories , sales_2024 , bottom = sales_2023 , label = ’ 2024 ’ , color = ’ steelblue ’ , edgecolor = ’ black ’)
5 plt . xlabel ( ’ Products ’)
6 plt . ylabel ( ’ Total Sales ( thousands ) ’)
7 plt . title ( ’ Stacked Bar Plot - Cumulative Sales ’)
8 plt . legend ()
9 plt . grid ( True , alpha =0.3 , axis = ’y ’)
10 plt . xticks ( rotation =45)
11 plt . show ()
Listing 8: Stacked Bar Plot

1 # 5. Bar Plot with Error Bars

2 errors = [3 , 4 , 3 , 5 , 4]
3 plt . figure ( figsize =(10 , 6) )
4 plt . bar ( categories , sales_2024 , yerr = errors , capsize =5 ,
5 color = ’ lightgreen ’ , edgecolor = ’ black ’ , alpha =0.7 ,
6 error_kw ={ ’ linewidth ’: 2 , ’ ecolor ’: ’ red ’ })
7 plt . xlabel ( ’ Products ’)
8 plt . ylabel ( ’ Sales ( thousands ) ’)
9 plt . title ( ’ Bar Plot with Error Bars ( Standard Deviation ) ’)
10 plt . grid ( True , alpha =0.3 , axis = ’y ’)
11 plt . xticks ( rotation =45)
12 plt . show ()
Listing 9: Bar Plot with Error Bars

0.4.3 Explanation of Examples

• Basic Vertical Bar Plot: Shows sales for each product as vertical bars.

• Horizontal Bar Plot: Useful when category names are long, easier to read.

• Grouped Bar Plot: Compares sales across two years side by side for each product.

• Stacked Bar Plot: Displays cumulative totals by stacking 2024 sales on top of 2023.

• Bar Plot with Error Bars: Includes variability or uncertainty in the data using error bars.

0.5 Program 3: Histograms

0.5.1 Introduction
Histograms are used to visualize the distribution of continuous data by grouping values into intervals (bins) and plotting the
frequency of each bin. They are useful for identifying the shape of the data, such as normality, skewness, modality, and detecting
outliers. Variations include cumulative histograms, density plots, and 2D histograms for bivariate data.

0.5.2 Python Code Examples

1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 # Generate normal distribution
5 np . random . seed (42)
6 normal_data = np . random . normal (100 , 15 , 1000)
7
8 # 1. Basic Histogram
9 plt . figure ( figsize =(10 , 6) )
10 plt . hist ( normal_data , bins =30 , color = ’ steelblue ’ , edgecolor = ’ black ’ , alpha =0.7)
11 plt . xlabel ( ’ Values ’)
12 plt . ylabel ( ’ Frequency ’)
13 plt . title ( ’ Basic Histogram - Normal Distribution ’)
14 plt . grid ( True , alpha =0.3 , axis = ’y ’)
15 plt . show ()
Listing 10: Basic Histogram

6
 0.5. PROGRAM 3: HISTOGRAMS 7

1 # 2. Histogram with Different Bin Sizes

2 fig , axes = plt . subplots (2 , 2 , figsize =(12 , 10) )
3 bin_sizes = [5 , 10 , 30 , 50]
4
5 for idx , bins in enumerate ( bin_sizes ) :
6 row , col = idx // 2 , idx % 2
7 axes [ row , col ]. hist ( normal_data , bins = bins , color = ’ steelblue ’ ,
8 edgecolor = ’ black ’ , alpha =0.7)
9 axes [ row , col ]. set_title ( f ’ Bins = { bins } ’)
10 axes [ row , col ]. set_xlabel ( ’ Values ’)
11 axes [ row , col ]. set_ylabel ( ’ Frequency ’)
12 axes [ row , col ]. grid ( True , alpha =0.3 , axis = ’y ’)
13
14 plt . suptitle ( ’ Effect of Bin Size on Histogram ’ , fontsize =14)
15 plt . tight_layout ()
16 plt . show ()
Listing 11: Histogram with Different Bin Sizes

1 # 3. Multiple Distributions Comparison

2 exponential_data = np . random . exponential (2 , 1000)
3
4 plt . figure ( figsize =(12 , 6) )
5 plt . hist ( normal_data , bins =30 , alpha =0.5 , label = ’ Normal ’ , color = ’ blue ’ , edgecolor = ’ black ’)
6 plt . hist ( exponential_data , bins =30 , alpha =0.5 , label = ’ Exponential ’ , color = ’ red ’ , edgecolor = ’ black ’)
7 plt . xlabel ( ’ Values ’)
8 plt . ylabel ( ’ Frequency ’)
9 plt . title ( ’ Comparing Two Distributions ’)
10 plt . legend ()
11 plt . grid ( True , alpha =0.3 , axis = ’y ’)
12 plt . show ()
Listing 12: Comparing Multiple Distributions

1 # 4. Histogram with Density ( KDE ) Overlay

2 from scipy import stats
3
4 plt . figure ( figsize =(10 , 6) )
5 counts , bins , patches = plt . hist ( normal_data , bins =30 , density = True ,
6 alpha =0.7 , color = ’ steelblue ’ ,
7 edgecolor = ’ black ’ , label = ’ Histogram ’)
8
9 # KDE overlay
10 kde = stats . gaussian_kde ( normal_data )
11 x_range = np . linspace ( normal_data . min () , normal_data . max () , 200)
12 plt . plot ( x_range , kde ( x_range ) , ’r - ’ , linewidth =2 , label = ’ KDE ’)
13
14 plt . xlabel ( ’ Values ’)
15 plt . ylabel ( ’ Density ’)
16 plt . title ( ’ Histogram with KDE Overlay ’)
17 plt . legend ()
18 plt . grid ( True , alpha =0.3)
19 plt . show ()
Listing 13: Histogram with Density (KDE) Overlay

1 # 5. Cumulative Histogram
2 plt . figure ( figsize =(10 , 6) )
3 plt . hist ( normal_data , bins =30 , cumulative = True , density = True ,
4 color = ’ steelblue ’ , edgecolor = ’ black ’ , alpha =0.7 ,
5 label = ’ Cumulative Distribution ’)
6 plt . xlabel ( ’ Values ’)
7 plt . ylabel ( ’ Cumulative Probability ’)
8 plt . title ( ’ Cumulative Histogram ( CDF ) ’)
9 plt . grid ( True , alpha =0.3)
10 plt . show ()
Listing 14: Cumulative Histogram

1 # 6. 2 D Histogram ( Hexbin )
2 x = np . random . normal (0 , 1 , 5000)
3 y = x + np . random . normal (0 , 0.5 , 5000)
4
5 plt . figure ( figsize =(10 , 8) )
6 plt . hexbin (x , y , gridsize =30 , cmap = ’ YlOrRd ’)
7 plt . colorbar ( label = ’ Count ’)
8 plt . xlabel ( ’X values ’)
9 plt . ylabel ( ’Y values ’)
10 plt . title ( ’2 D Histogram ( Hexbin Plot ) ’)
11 plt . show ()
Listing 15: 2D Histogram (Hexbin Plot)

0.5.3 Explanation of Examples

• Basic Histogram: Shows frequency distribution of a single continuous variable.

• Different Bin Sizes: Demonstrates how bin width affects histogram appearance and interpretation.

• Multiple Distributions: Allows visual comparison of two or more distributions.

• Histogram with KDE Overlay: Adds smooth density curve to highlight distribution shape.

• Cumulative Histogram: Represents cumulative distribution function (CDF) of the data.

• 2D Histogram (Hexbin): Visualizes the joint distribution of two continuous variables.

7
 8

0.6 Program 4: Scatter Plots

0.6.1 Introduction
Scatter plots are used to visualize relationships between two continuous variables. They help identify correlations, clusters,
outliers, and patterns. Variations include bubble charts (size as a third variable), color-coded categories, and regression lines
to show trends.

0.6.2 Python Code Examples

1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 # Generate positively correlated data
5 np . random . seed (42)
6 n = 200
7 x = np . random . normal (0 , 1 , n )
8 y = 0.7 * x + np . random . normal (0 , 0.5 , n )
9
10 # 1. Basic Scatter Plot
11 plt . figure ( figsize =(10 , 6) )
12 plt . scatter (x , y , alpha =0.6 , color = ’ steelblue ’ , edgecolor = ’ black ’ , s =50)
13 plt . xlabel ( ’X variable ’)
14 plt . ylabel ( ’Y variable ’)
15 plt . title ( ’ Basic Scatter Plot - Positive Correlation ’)
16 plt . grid ( True , alpha =0.3)
17 plt . show ()
Listing 16: Basic Scatter Plot

1 # 2. Scatter Plot for Positive , Negative , and No Correlation

2 x2 = np . random . normal (0 , 1 , n )
3 y2 = -0.6 * x2 + np . random . normal (0 , 0.5 , n ) # Negative correlation
4 x3 = np . random . normal (0 , 1 , n )
5 y3 = np . random . normal (0 , 1 , n ) # No correlation
6
7 fig , axes = plt . subplots (1 , 3 , figsize =(15 , 5) )
8

9 axes [0]. scatter (x , y , alpha =0.6 , color = ’ steelblue ’ , edgecolor = ’ black ’)

10 axes [0]. set_title ( ’ Positive Correlation ( r 0.7) ’)
11 axes [0]. set_xlabel ( ’X ’) ; axes [0]. set_ylabel ( ’Y ’)
12 axes [0]. grid ( True , alpha =0.3)
13
14 axes [1]. scatter ( x2 , y2 , alpha =0.6 , color = ’ coral ’ , edgecolor = ’ black ’)
15 axes [1]. set_title ( ’ Negative Correlation ( r -0.6) ’)
16 axes [1]. set_xlabel ( ’X ’) ; axes [1]. set_ylabel ( ’Y ’)
17 axes [1]. grid ( True , alpha =0.3)
18
19 axes [2]. scatter ( x3 , y3 , alpha =0.6 , color = ’ green ’ , edgecolor = ’ black ’)
20 axes [2]. set_title ( ’ No Correlation ( r 0) ’)
21 axes [2]. set_xlabel ( ’X ’) ; axes [2]. set_ylabel ( ’Y ’)
22 axes [2]. grid ( True , alpha =0.3)
23
24 plt . suptitle ( ’ Scatter Plots - Different Correlation Types ’ , fontsize =14)
25 plt . tight_layout ()
26 plt . show ()
Listing 17: Scatter Plot with Different Correlation Types

1 # 3. Bubble Chart
2 n = 100
3 x = np . random . uniform (0 , 100 , n )
4 y = np . random . uniform (0 , 100 , n )
5 population = np . random . gamma (2 , 10 , n ) * 1000 # Bubble size
6 colors = np . random . uniform (0 , 1 , n ) # Color scale
7
8 scatter = plt . scatter (x , y , s = population /100 , c = colors , alpha =0.6 ,
9 cmap = ’ viridis ’ , edgecolor = ’ black ’ , linewidth =0.5)
10 plt . colorbar ( scatter , label = ’ Color scale ’)
11 plt . xlabel ( ’X coordinate ’)
12 plt . ylabel ( ’Y coordinate ’)
13 plt . title ( ’ Bubble Chart - Size = Population , Color = Category ’)
14 plt . grid ( True , alpha =0.3)
15 plt . show ()
Listing 18: Bubble Chart (Size

1 # 4. Scatter Plot with Regression Line

2 plt . figure ( figsize =(10 , 6) )
3 plt . scatter (x , y , alpha =0.6 , color = ’ steelblue ’ , edgecolor = ’ black ’ , label = ’ Data ’)
4

5 # Regression line
6 z = np . polyfit (x , y , 1)
7 p = np . poly1d ( z )
8 x_line = np . linspace ( x . min () , x . max () , 100)
9 plt . plot ( x_line , p ( x_line ) , ’r - ’ , linewidth =2 ,
10 label = f ’ Regression line ( y ={ z [0]:.2 f } x +{ z [1]:.2 f }) ’)
11
12 plt . xlabel ( ’X variable ’)
13 plt . ylabel ( ’Y variable ’)
14 plt . title ( ’ Scatter Plot with Regression Line ’)
15 plt . legend ()
16 plt . grid ( True , alpha =0.3)

8
 0.7. PROGRAM 5: SUBPLOTS 9

17 plt . show ()
Listing 19: Scatter Plot with Regression Line

1 # 5. Scatter Plot with Color by Category

2 categories = np . random . choice ([ ’ Group A ’ , ’ Group B ’ , ’ Group C ’] , n )
3 colors = { ’ Group A ’: ’ red ’ , ’ Group B ’: ’ blue ’ , ’ Group C ’: ’ green ’}
4
5 plt . figure ( figsize =(10 , 6) )
6 for category in colors :
7 mask = categories == category
8 plt . scatter ( x [ mask ] , y [ mask ] , c = colors [ category ] , label = category ,
9 alpha =0.6 , edgecolor = ’ black ’ , s =80)
10 plt . xlabel ( ’X variable ’)
11 plt . ylabel ( ’Y variable ’)
12 plt . title ( ’ Scatter Plot - Colored by Category ’)
13 plt . legend ()
14 plt . grid ( True , alpha =0.3)
15 plt . show ()
Listing 20: Scatter Plot Colored by Category

0.6.3 Explanation of Examples

• Basic Scatter Plot: Shows the relationship between two continuous variables.

• Different Correlation Types: Demonstrates positive, negative, and no correlation visually.

• Bubble Chart: Introduces a third variable via bubble size and a fourth via color.

• Regression Line: Adds a trend line to indicate the linear relationship between variables.

• Color by Category: Highlights categorical distinctions in the scatter plot.

0.7 Program 5: Subplots

0.7.1 Introduction
Subplots allow multiple plots to be displayed within a single figure. They are useful for comparing different datasets, visualizing
related variables, and creating dashboards. Matplotlib provides flexible methods for creating regular grids, irregular layouts,
shared axes, and nested plots using GridSpec.

0.7.2 Python Code Examples

1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 # Generate sample data
5 x = np . linspace (0 , 10 , 100)
6 categories = [ ’A ’ , ’B ’ , ’C ’ , ’D ’]
7 values = [23 , 45 , 56 , 78]
8 data1 = np . random . normal (0 , 1 , 1000)
9 data2 = np . random . normal (2 , 1.5 , 1000)
10
11 # 1. Basic Subplot (2 x2 grid )
12 fig , axes = plt . subplots (2 , 2 , figsize =(12 , 10) )
13
14 # Line plot
15 axes [0 , 0]. plot (x , np . sin ( x ) , ’b - ’ , linewidth =2)
16 axes [0 , 0]. set_title ( ’ Line Plot - sin ( x ) ’)
17 axes [0 , 0]. set_xlabel ( ’x ’)
18 axes [0 , 0]. set_ylabel ( ’ sin ( x ) ’)
19 axes [0 , 0]. grid ( True , alpha =0.3)
20
21 # Bar plot
22 axes [0 , 1]. bar ( categories , values , color = ’ steelblue ’ , edgecolor = ’ black ’)
23 axes [0 , 1]. set_title ( ’ Bar Plot ’)
24 axes [0 , 1]. set_xlabel ( ’ Categories ’)
25 axes [0 , 1]. set_ylabel ( ’ Values ’)
26 axes [0 , 1]. grid ( True , alpha =0.3 , axis = ’y ’)
27
28 # Histogram
29 axes [1 , 0]. hist ( data1 , bins =30 , alpha =0.7 , color = ’ green ’ , edgecolor = ’ black ’)
30 axes [1 , 0]. set_title ( ’ Histogram ’)
31 axes [1 , 0]. set_xlabel ( ’ Values ’)
32 axes [1 , 0]. set_ylabel ( ’ Frequency ’)
33 axes [1 , 0]. grid ( True , alpha =0.3 , axis = ’y ’)
34
35 # Scatter plot
36 axes [1 , 1]. scatter ( data1 , data2 , alpha =0.5 , color = ’ coral ’ , edgecolor = ’ black ’)
37 axes [1 , 1]. set_title ( ’ Scatter Plot ’)
38 axes [1 , 1]. set_xlabel ( ’X values ’)
39 axes [1 , 1]. set_ylabel ( ’Y values ’)
40 axes [1 , 1]. grid ( True , alpha =0.3)
41

42 plt . suptitle ( ’2 2 Subplot Grid - Different Plot Types ’ , fontsize =16)

43 plt . tight_layout ()
44 plt . show ()
Listing 21: Basic 2x2 Subplot Grid

9
 10

1 # 2. Irregular Subplot Layout

2 fig = plt . figure ( figsize =(15 , 10) )
3 ax1 = plt . subplot (2 , 2 , 1) # Top - left
4 ax2 = plt . subplot (2 , 2 , 2) # Top - right
5 ax3 = plt . subplot (2 , 1 , 2) # Bottom ( spans both columns )
6

7 ax1 . plot (x , np . sin ( x ) , ’b - ’ , linewidth =2) ; ax1 . set_title ( ’ sin ( x ) ’) ; ax1 . grid ( True , alpha =0.3)
8 ax2 . plot (x , np . cos ( x ) , ’r - ’ , linewidth =2) ; ax2 . set_title ( ’ cos ( x ) ’) ; ax2 . grid ( True , alpha =0.3)
9 ax3 . plot (x , np . sin ( x ) * np . cos ( x ) , ’g - ’ , linewidth =2) ; ax3 . set_title ( ’ sin ( x ) cos ( x ) ’) ; ax3 . set_xlabel (
’x ’) ; ax3 . grid ( True , alpha =0.3)
10
11 plt . suptitle ( ’ Irregular Subplot Layout ’ , fontsize =16)
12 plt . tight_layout ()
13 plt . show ()
Listing 22: Irregular Subplot Layout

1 # 3. Subplots with Shared X - Axis

2 fig , axes = plt . subplots (3 , 1 , figsize =(10 , 12) , sharex = True )
3 axes [0]. plot (x , np . sin ( x ) , ’b - ’ , linewidth =2) ; axes [0]. set_ylabel ( ’ sin ( x ) ’) ; axes [0]. set_title ( ’ Three
Plots with Shared X - Axis ’) ; axes [0]. grid ( True , alpha =0.3)
4 axes [1]. plot (x , np . cos ( x ) , ’r - ’ , linewidth =2) ; axes [1]. set_ylabel ( ’ cos ( x ) ’) ; axes [1]. grid ( True , alpha
=0.3)
5 axes [2]. plot (x , np . sin ( x ) + np . cos ( x ) , ’g - ’ , linewidth =2) ; axes [2]. set_xlabel ( ’X axis ( shared ) ’) ; axes
[2]. set_ylabel ( ’ sin ( x ) + cos ( x ) ’) ; axes [2]. grid ( True , alpha =0.3)
6
7 plt . tight_layout ()
8 plt . show ()
Listing 23: Subplots with Shared Axes

1 # 4. Nested Subplots ( GridSpec )

2 from matplotlib . gridspec import GridSpec
3 fig = plt . figure ( figsize =(12 , 8) )
4 gs = GridSpec (3 , 3 , figure = fig )
5

6 ax1 = fig . add_subplot ( gs [0 , :]) # Top row , all columns

7 ax2 = fig . add_subplot ( gs [1 , : -1]) # Middle row , first two columns
8 ax3 = fig . add_subplot ( gs [1: , -1]) # Right column , last two rows
9 ax4 = fig . add_subplot ( gs [2 , 0]) # Bottom - left
10 ax5 = fig . add_subplot ( gs [2 , 1]) # Bottom - middle
11

12 ax1 . plot (x , np . sin ( x ) , ’b - ’ , linewidth =2) ; ax1 . set_title ( ’ Wide Plot ( Top ) ’) ; ax1 . grid ( True , alpha =0.3)
13 ax2 . plot (x , np . cos ( x ) , ’r - ’ , linewidth =2) ; ax2 . set_title ( ’ Medium Plot ’) ; ax2 . grid ( True , alpha =0.3)
14 ax3 . barh ( categories , values , color = ’ steelblue ’) ; ax3 . set_title ( ’ Tall Plot ’) ; ax3 . grid ( True , alpha =0.3 ,
axis = ’x ’)
15 ax4 . hist ( data1 , bins =30 , color = ’ green ’ , edgecolor = ’ black ’) ; ax4 . set_title ( ’ Small Plot 1 ’) ; ax4 . grid (
True , alpha =0.3)
16 ax5 . scatter ( data1 , data2 , alpha =0.5 , color = ’ coral ’) ; ax5 . set_title ( ’ Small Plot 2 ’) ; ax5 . grid ( True ,
alpha =0.3)
17
18 plt . suptitle ( ’ Complex GridSpec Layout ’ , fontsize =16)
19 plt . tight_layout ()
20 plt . show ()
Listing 24: Nested Subplots Using GridSpec

0.7.3 Explanation of Examples

• Basic 2x2 Grid: Shows multiple plot types together for comparison.

• Irregular Layouts: Combines subplots of different sizes in one figure.

• Shared Axes: Synchronizes axes for easier comparison of trends.

• Nested Subplots/GridSpec: Provides full control over layout and subplot sizes.

0.8 Program 6: Complete Customization Example

0.8.1 Introduction
This program demonstrates a fully customized Matplotlib plot, incorporating multiple elements such as multiple datasets,
regression lines, annotations, text boxes, secondary axes, gridlines, legends, and axis customization. It serves as a reference for
creating publication-quality visualizations.

0.8.2 Python Code Example

1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 # Generate data
5 x = np . linspace (0 , 10 , 50)
6 y1 = 2* x + 1 + np . random . normal (0 , 1 , 50)
7 y2 = 3* x - 2 + np . random . normal (0 , 1.5 , 50)
8

9 # Create figure and axis

10 fig , ax = plt . subplots ( figsize =(12 , 8) )
11
12 # Scatter plots
13 ax . scatter (x , y1 , color = ’ blue ’ , marker = ’o ’ , s =80 , alpha =0.7 , edgecolor = ’ black ’ , linewidth =1 , label = ’
Dataset A ’)

10
 0.9. PROGRAM 6: COMPLETE CUSTOMIZATION EXAMPLE 11

14 ax . scatter (x , y2 , color = ’ red ’ , marker = ’s ’ , s =80 , alpha =0.7 , edgecolor = ’ black ’ , linewidth =1 , label = ’
Dataset B ’)
15
16 # Regression lines
17 z1 = np . polyfit (x , y1 , 1)
18 z2 = np . polyfit (x , y2 , 1)
19 p1 = np . poly1d ( z1 )
20 p2 = np . poly1d ( z2 )
21 ax . plot (x , p1 ( x ) , ’b - - ’ , linewidth =2 , label = f ’A trend ( slope ={ z1 [0]:.2 f }) ’)
22 ax . plot (x , p2 ( x ) , ’r - - ’ , linewidth =2 , label = f ’B trend ( slope ={ z2 [0]:.2 f }) ’)
23
24 # Axis customization
25 ax . set_xlabel ( ’ Time ( hours ) ’ , fontsize =12 , fontweight = ’ bold ’)
26 ax . set_ylabel ( ’ Measurement ( units ) ’ , fontsize =12 , fontweight = ’ bold ’)
27 ax . set_title ( ’ Complete Customized Plot Example \ nwith All Elements ’ , fontsize =16 , fontweight = ’ bold ’ , pad
=20)
28 ax . set_xlim ( -0.5 , 10.5)
29 ax . set_ylim ( -5 , 35)
30 ax . set_xticks ( np . arange (0 , 11 , 1) )
31 ax . set_yticks ( np . arange ( -5 , 36 , 5) )
32 ax . tick_params ( axis = ’ both ’ , which = ’ major ’ , labelsize =10)
33
34 # Grid , legend , and annotations
35 ax . grid ( True , alpha =0.3 , linestyle = ’ -- ’ , linewidth =0.5)
36 ax . legend ( loc = ’ upper left ’ , fontsize =10 , framealpha =0.9)
37 ax . annotate ( ’ Important \ nobservation ’ , xy =(5 , 15) , xytext =(6 , 25) ,
38 arrowprops = dict ( facecolor = ’ black ’ , shrink =0.05 , width =1.5) ,
39 fontsize =11 , ha = ’ center ’ , bbox = dict ( boxstyle = ’ round ’ , facecolor = ’ wheat ’ , alpha =0.5) )
40 ax . text (0.02 , 0.98 , ’ Correlation : 0.85 ’ , transform = ax . transAxes ,
41 fontsize =11 , ve rtical alignm ent = ’ top ’ ,
42 bbox = dict ( boxstyle = ’ round ’ , facecolor = ’ lightblue ’ , alpha =0.8) )
43
44 # Horizontal and vertical lines
45 ax . axhline ( y =0 , color = ’ gray ’ , linestyle = ’ - ’ , linewidth =0.5 , alpha =0.5)
46 ax . axvline ( x =5 , color = ’ gray ’ , linestyle = ’ - ’ , linewidth =0.5 , alpha =0.5)
47

48 # Secondary y - axis
49 ax2 = ax . twinx ()
50 ax2 . set_ylabel ( ’ Secondary Scale ’ , fontsize =12 , color = ’ purple ’)
51 ax2 . tick_params ( axis = ’y ’ , labelcolor = ’ purple ’)
52 ax2 . set_ylim ( -10 , 70)
53

54 plt . tight_layout ()
55 plt . show ()
Listing 25: Complete Customization with Scatter Plots and Regression Lines

0.8.3 Explanation of Customizations

• Multiple datasets: Visualize two related datasets on the same plot with distinct markers and colors.

• Regression lines: Add trend lines to highlight patterns.

• Annotations and text boxes: Highlight important observations and display contextual information.

• Secondary y-axis: Display an additional measurement scale.

• Grid, limits, ticks, and legend: Improve readability and interpretability.

0.8.4 Summary Table: When to Use Each Plot Type

Plot Type Best Use Case Example Code

Line Plot Time series, trends [Link](x, y)
Bar Plot Category comparisons [Link](x, height)
Histogram Distribution analysis [Link](data, bins)
Scatter Plot Correlation analysis [Link](x, y)
Subplots Multiple comparisons [Link](rows, cols)

Table 6: Overview of Matplotlib Plot Types and Best Use Cases

0.9 Program 6: Complete Customization Example

0.9.1 Introduction
This program demonstrates a fully customized Matplotlib plot, incorporating multiple elements such as multiple datasets,
regression lines, annotations, text boxes, secondary axes, gridlines, legends, and axis customization. It serves as a reference for
creating publication-quality visualizations.

0.9.2 Python Code Example

1 import numpy as np
2 import matplotlib . pyplot as plt
3
4 # Generate data
5 x = np . linspace (0 , 10 , 50)
6 y1 = 2* x + 1 + np . random . normal (0 , 1 , 50)
7 y2 = 3* x - 2 + np . random . normal (0 , 1.5 , 50)

11
 12

8
9 # Create figure and axis
10 fig , ax = plt . subplots ( figsize =(12 , 8) )
11
12 # Scatter plots
13 ax . scatter (x , y1 , color = ’ blue ’ , marker = ’o ’ , s =80 , alpha =0.7 , edgecolor = ’ black ’ , linewidth =1 , label = ’
Dataset A ’)
14 ax . scatter (x , y2 , color = ’ red ’ , marker = ’s ’ , s =80 , alpha =0.7 , edgecolor = ’ black ’ , linewidth =1 , label = ’
Dataset B ’)
15
16 # Regression lines
17 z1 = np . polyfit (x , y1 , 1)
18 z2 = np . polyfit (x , y2 , 1)
19 p1 = np . poly1d ( z1 )
20 p2 = np . poly1d ( z2 )
21 ax . plot (x , p1 ( x ) , ’b - - ’ , linewidth =2 , label = f ’A trend ( slope ={ z1 [0]:.2 f }) ’)
22 ax . plot (x , p2 ( x ) , ’r - - ’ , linewidth =2 , label = f ’B trend ( slope ={ z2 [0]:.2 f }) ’)
23

24 # Axis customization
25 ax . set_xlabel ( ’ Time ( hours ) ’ , fontsize =12 , fontweight = ’ bold ’)
26 ax . set_ylabel ( ’ Measurement ( units ) ’ , fontsize =12 , fontweight = ’ bold ’)
27 ax . set_title ( ’ Complete Customized Plot Example \ nwith All Elements ’ , fontsize =16 , fontweight = ’ bold ’ , pad
=20)
28 ax . set_xlim ( -0.5 , 10.5)
29 ax . set_ylim ( -5 , 35)
30 ax . set_xticks ( np . arange (0 , 11 , 1) )
31 ax . set_yticks ( np . arange ( -5 , 36 , 5) )
32 ax . tick_params ( axis = ’ both ’ , which = ’ major ’ , labelsize =10)
33
34 # Grid , legend , and annotations
35 ax . grid ( True , alpha =0.3 , linestyle = ’ -- ’ , linewidth =0.5)
36 ax . legend ( loc = ’ upper left ’ , fontsize =10 , framealpha =0.9)
37 ax . annotate ( ’ Important \ nobservation ’ , xy =(5 , 15) , xytext =(6 , 25) ,
38 arrowprops = dict ( facecolor = ’ black ’ , shrink =0.05 , width =1.5) ,
39 fontsize =11 , ha = ’ center ’ , bbox = dict ( boxstyle = ’ round ’ , facecolor = ’ wheat ’ , alpha =0.5) )
40 ax . text (0.02 , 0.98 , ’ Correlation : 0.85 ’ , transform = ax . transAxes ,
41 fontsize =11 , ve rtical alignm ent = ’ top ’ ,
42 bbox = dict ( boxstyle = ’ round ’ , facecolor = ’ lightblue ’ , alpha =0.8) )
43
44 # Horizontal and vertical lines
45 ax . axhline ( y =0 , color = ’ gray ’ , linestyle = ’ - ’ , linewidth =0.5 , alpha =0.5)
46 ax . axvline ( x =5 , color = ’ gray ’ , linestyle = ’ - ’ , linewidth =0.5 , alpha =0.5)
47
48 # Secondary y - axis
49 ax2 = ax . twinx ()
50 ax2 . set_ylabel ( ’ Secondary Scale ’ , fontsize =12 , color = ’ purple ’)
51 ax2 . tick_params ( axis = ’y ’ , labelcolor = ’ purple ’)
52 ax2 . set_ylim ( -10 , 70)
53
54 plt . tight_layout ()
55 plt . show ()
Listing 26: Complete Customization with Scatter Plots and Regression Lines

0.9.3 Explanation of Customizations

• Multiple datasets: Visualize two related datasets on the same plot with distinct markers and colors.

• Regression lines: Add trend lines to highlight patterns.

• Annotations and text boxes: Highlight important observations and display contextual information.

• Secondary y-axis: Display an additional measurement scale.

• Grid, limits, ticks, and legend: Improve readability and interpretability.

0.9.4 Summary Table: When to Use Each Plot Type

Plot Type Best Use Case Example Code

Line Plot Time series, trends [Link](x, y)
Bar Plot Category comparisons [Link](x, height)
Histogram Distribution analysis [Link](data, bins)
Scatter Plot Correlation analysis [Link](x, y)
Subplots Multiple comparisons [Link](rows, cols)

Table 7: Overview of Matplotlib Plot Types and Best Use Cases

5.3 Seaborn for Statistical Visualization: Box Plot, Pair Plot, Heat Map
1. Introduction to Seaborn
Seaborn is a Python data visualization library based on Matplotlib that provides a high-level interface for drawing statistical
graphics. It integrates closely with pandas DataFrames and offers sensible defaults for statistical visualizations.
Key Features:

• Built-in themes for attractive graphics

• Functions for visualizing distributions and relationships

12
 0.9. PROGRAM 6: COMPLETE CUSTOMIZATION EXAMPLE 13

• Tools for visualizing linear regression models

• Support for multi-plot grids

• Integration with pandas DataFrames

2. Statistical Visualizations in Seaborn

Seaborn specializes in statistical visualizations that help understand distributions and relationships between variables, incorpo-
rating statistical estimations and aggregations.

3. Box Plot
3.1 What is a Box Plot?
A box plot displays the distribution of data based on a five-number summary: minimum, Q1, median, Q3, and maximum.
Useful for identifying outliers and comparing distributions across categories.

3.2 Components of a Box Plot

Component Formula Description

Median (Q2) Middle value Line inside box (50th percentile)
First Quartile (Q1) 25th percentile Bottom of box
Third Quartile (Q3) 75th percentile Top of box
Interquartile Range (IQR) Q3 - Q1 Height of box (middle 50% of data)
Lower Whisker Q1 - 1.5 × IQR Extends to smallest non-outlier
Upper Whisker Q3 + 1.5 × IQR Extends to largest non-outlier
Outliers Points beyond whiskers Individual points shown separately

3.3 Statistical Interpretation

• Median centered: symmetric distribution

• Median near bottom: right-skewed

• Median near top: left-skewed

• Wider box: more variability

• Outliers: unusual values

3.4 When to Use Box Plots

• Comparing distributions across categories

• Identifying outliers

• Understanding spread and symmetry

• Showing median and quartiles

• Detecting skewness

3.5 Seaborn Box Plot Functions

Function Purpose
[Link]() Standard box plot
[Link]() Letter-value plot for large datasets
[Link]() Box plot + kernel density estimate
[Link]() Mean with error bars

3.6 Key Parameters

1 sns . boxplot (
2 x = None , y = None , hue = None , data = None ,
3 order = None , palette = None , showfliers = True
4 )

4. Pair Plot
4.1 What is a Pair Plot?
Creates a grid showing pairwise relationships, with scatter plots for numeric pairs and distributions on the diagonal.

4.2 Structure of a Pair Plot

• Diagonal: distribution of each variable

• Off-diagonal: scatter plots of variable pairs

• Hue: color-code categories

13
 14

4.3 What Pair Plots Reveal

Feature What to Look For

Correlations Linear patterns
Clusters Groups of points
Outliers Points far from main cloud
Distributions Histogram shape on diagonal
Skewness Asymmetric distributions
Modality Multiple peaks
Non-linear relationships Curved scatter patterns

4.4 When to Use Pair Plots

• Initial dataset exploration

• Identifying relationships

• Checking multicollinearity

• Discovering patterns and anomalies

4.5 Limitations
• Cluttered with many variables (¿7)

• Computationally intensive

• Only shows pairwise relationships

4.6 Key Parameters

1 sns . pairplot (
2 data , hue = None , vars = None ,
3 kind = ’ scatter ’ , diag_kind = ’ hist ’ ,
4 palette = None , markers = None ,
5 height =2.5 , aspect =1 , corner = False
6 )

5. Heat Map
5.1 What is a Heat Map?
Graphical representation of data where values are shown by color, useful for matrices (correlation, confusion, etc.)

5.2 Structure
• Grid cells: values

• Color mapping: magnitude

• Color bar: legend

• Row/column labels

• Optional annotations

5.3 Common Applications

Application Purpose
Correlation Matrix Show variable relationships
Confusion Matrix Classification performance
Missing Data Pattern Show missing data
Time Series Patterns across time
Genomic Data Gene expression patterns

5.4 Interpreting Correlation Heat Maps

• Red/Blue: positive/negative correlation

• Color intensity: strength

• Dark diagonal: perfect self-correlation

• Patterns: correlated variable groups

5.5 When to Use Heat Maps

• Visualizing correlation matrices

• Confusion matrices

• Missing data patterns

• Identifying clusters

• Comparing multiple variables

14
 0.9. PROGRAM 6: COMPLETE CUSTOMIZATION EXAMPLE 15

5.6 Key Parameters

1 sns . heatmap (
2 data , annot = False , fmt = ’ .2 g ’ , cmap = ’ coolwarm ’ ,
3 center = None , vmin = None , vmax = None , cbar = True ,
4 square = False , linewidths =0 , linecolor = ’ white ’ ,
5 xticklabels = ’ auto ’ , yticklabels = ’ auto ’ ,
6 mask = None , ax = None
7 )

5.7 Color Maps for Heat Maps

Color Map Best For

coolwarm Diverging data
viridis Sequential data
RdBu r Red-Blue diverging
YlOrRd Yellow-Orange-Red intensity
Blues Single-color sequential

6. Comparison of Seaborn Statistical Plots

Feature Box Plot Pair Plot Heat Map
Purpose Distribution comparison Multi-variable relationships Matrix visualization
Data Type Numeric + Categorical Multiple numeric 2D Matrix
Shows Quartiles, outliers Pairwise relationships Color-coded values
Best For Group comparisons Initial exploration Correlation/confusion
Limitation Limited detail Cluttered ¿7 vars Needs structured data

7. Summary of Key Functions

Plot Type Function Basic Syntax
Box Plot [Link]() [Link](x=’category’, y=’value’, data=df)
Pair Plot [Link]() [Link](df, hue=’category’)
Heat Map [Link]() [Link]([Link](), annot=True)

5.5 Interactive Visualization using Plotly

1. Introduction to Plotly
Plotly is an open-source graphing library for creating interactive, publication-quality visualizations. Unlike static libraries like
Matplotlib, Plotly generates web-based visualizations that allow zooming, panning, hovering, and clicking interactions. Built
on [Link], Plotly for Python brings interactive web graphics directly into Python.

2. The Plotly Ecosystem

Plotly provides two primary interfaces for visualization:

2.1 Plotly Express

Plotly Express (px) is a high-level API for creating charts with concise syntax and sensible defaults. It is ideal for exploratory
data analysis and rapid prototyping.
Key characteristics:
• One-line chart creation
• Direct integration with Pandas DataFrames
• Automatic interactive features
• Pre-styled charts

2.2 Graph Objects

Graph Objects (go) is a low-level interface offering fine-grained control over every aspect of a visualization.
Key characteristics:
• Complete customization
• Multi-layered visualizations
• Modify figures after creation
• Access to advanced chart types

3. Core Concepts in Plotly

3.1 Figure Structure
Every Plotly visualization is built around a Figure object:
Component Description Contents
data Visual representation List of traces (scatter, bar, etc.)
layout Chart environment Titles, axes, legends, annotations

15
 16

3.2 Traces
A trace is the fundamental building block of a figure. Each trace represents one dataset and its visualization. Multiple traces
can be combined in one figure.
Common trace types:

• Scatter: points, lines, or both

• Bar: vertical or horizontal bars

• Box: statistical box plots

• Heatmap: color-coded matrices

3.3 Interactivity Features

Feature Description User Benefit

Hover Tooltips Display values on hover Immediate access to data
Zoom & Pan Scale and move across plot Examine details
Legend Toggle Show/hide traces via legend Focus on specific series
Range Sliders Select ranges dynamically Filter data interactively
Download Options Save as PNG or SVG Share or include in reports

4. The Two APIs: Express vs. Graph Objects

4.1 Plotly Express (px)

1 import plotly . express as px

2 fig = px . scatter ( df , x = ’ column1 ’ , y = ’ column2 ’ , color = ’ category ’)
3 fig . show ()

Best for:

• Quick exploratory analysis

• Standard chart types

• Beginners and rapid prototyping

4.2 Graph Objects (go)

1 import plotly . graph_objects as go

2 fig = go . Figure ()
3 fig . add_trace ( go . Scatter ( x = df [ ’x ’] , y = df [ ’y ’] , mode = ’ markers ’) )
4 fig . update_layout ( title = ’ Custom Title ’)
5 fig . show ()

Best for:

• Complex, multi-trace visualizations

• Fine-tuned customization

• Specialized chart types

5. Chart Types in Plotly

Category Chart Types

Basic Charts Scatter, line, bar, pie, histogram
Statistical Charts Box plot, violin plot, histogram, heatmap
Scientific Charts Contour, error bars, ternary plots
Financial Charts Time series, candlestick, OHLC
Maps Choropleth, scatter geo, line geo
3D Charts 3D scatter, 3D surface, 3D mesh

6. Integration with Data Analysis Workflow

• With Pandas: Plotly Express accepts DataFrames directly

• With Jupyter: Figures render inline, supporting interactive exploration

• With Dash: Plotly provides the visualization foundation for interactive web apps

7. Customization Capabilities
7.1 Visual Styling
• Templates: ’plotly’, ’plotlyd ark ′ ,′ ggplot2′ ,′ seaborn′ ,′ simplew hite′
• Color scales: sequential, diverging, qualitative

• Fonts: size, family, color

16
0.9. PROGRAM 6: COMPLETE CUSTOMIZATION EXAMPLE 17

7.2 Layout Control

• Margins: control spacing

• Annotations: text, shapes, arrows

• Axes: tick marks, grids, ranges

7.3 Hover Information

• Tooltips: control displayed data

• Hover mode: single, closest, compare

8. Export and Sharing

Format Method Use Case
HTML [Link] html(’[Link]’) Share interactive charts
Static Images [Link] image(’[Link]’) Include in reports
Plotly Cloud [Link](renderer=”plotly mimetype”) Host on Plotly’s platform

9. Advantages of Interactive Visualization

• Deeper Data Exploration: Probe points, zoom, and filter dynamically

• Enhanced Communication: Users explore data at their own pace

• Efficient EDA Workflow: Rapid iteration and hypothesis testing

10. Summary: Key Takeaways

Concept Key Point
Plotly Express High-level API for simple charts
Graph Objects Low-level API for full control
Interactivity Hover, zoom, pan, toggle built-in
Figure Structure Data (traces) + Layout (environment)
Integration Works with Pandas and Jupyter
Export HTML for sharing, images for reports
listings hyperref tikz
margin=1in

5.6 Visualization Driven Insight Generation

1. Introduction to Insight Generation
Data visualization is not merely about creating charts; it is about generating insights that drive decision-making. Visualization-
driven insight generation uses graphical representations to discover patterns, trends, and anomalies leading to actionable un-
derstanding.

2. The Insight Generation Process

Data → Visualization → Pattern Recognition → Hypothesis → Insight → Action

Stage Description
Data Raw information collected from various sources
Visualization Graphical representation of data
Pattern Recognition Identifying trends, clusters, outliers
Hypothesis Formulating explanations for patterns
Insight Deep understanding with business implications
Action Decisions based on insights

3. Types of Insights from Visualizations

3.1 Descriptive Insights
Answers ”What happened?” based on historical data. Examples:

• Sales peaked in December over the last three years

• Customer churn is highest among users in their first 90 days

• Product A consistently outsells Product B across regions

3.2 Diagnostic Insights

Answers ”Why did it happen?” by revealing relationships and causes. Examples:

• Sales decline correlates with competitor price reductions

• High churn coincides with poor customer service

• Regional variations linked to local economic indicators

17
18

3.3 Predictive Insights

Answers ”What might happen?” using trends. Examples:

• Upward trend in website traffic suggests continued growth

• Seasonal patterns indicate next peak in July

• Leading indicators point to potential downturn

3.4 Prescriptive Insights

Answers ”What should we do?” combining patterns with domain knowledge. Examples:

• Focus marketing on high-growth segments

• Adjust inventory based on visualized demand

• Target interventions where risk factors cluster

4. Visual Cues That Generate Insights

4.1 Patterns

Pattern What It Reveals Example Insight

Trend Directional movement ”Sales increased 15% annually”
Seasonality Predictable patterns ”December sees 30% sales spike”
Cyclicality Long-term waves ”Market correction every 5-7 years”
Correlation Variables move together ”Satisfaction correlates with retention”
Clustering Groups of points ”High-value customers cluster in urban areas”

4.2 Anomalies

Anomaly What It Reveals Example Insight

Outliers Points far from norm ”One transaction is 10× larger than average”
Structural Breaks Sudden pattern changes ”Website traffic dropped after redesign”
Gaps Missing expected data ”No sales from a key region”
Unexpected Stability No change where expected ”Prices didn’t drop despite competitor sale”

4.3 Comparisons

Comparison What It Reveals Example Insight

Benchmarking Against standards ”Performance 20% below industry avg”
Before/After Impact of interventions ”Satisfaction improved after training”
Segments Differences between groups ”Younger users prefer mobile”
Actual vs Forecast Prediction accuracy ”Overestimate Q1 sales by 15%”

5. Role of Visualization Types in Insight Generation

5.1 Time Series Visualizations

Chart Type Insight Generated

Line Chart Long-term trends, rate of change
Best for: trend, seasonality, cyclical insights Area Chart Magnitude of change over time
Seasonal Plot Year-over-year patterns
Time Series Decomposition Separating trend from seasonality

5.2 Comparison Visualizations

Chart Type Insight Generated

Bar Chart Relative performance across categories
Best for: ranking, benchmarking, segment insights Grouped Bar Chart Multi-factor comparisons
Diverging Bar Chart Deviation from baseline
Dot Plot Precise comparisons

5.3 Distribution Visualizations

Chart Type Insight Generated

Histogram Shape of distribution, modality
Best for: spread, central tendency, outlier insights Box Plot Quartiles, spread, outliers
Violin Plot Distribution shape + density
Density Plot Smooth distribution visualization

18
0.9. PROGRAM 6: COMPLETE CUSTOMIZATION EXAMPLE 19

5.4 Relationship Visualizations

Chart Type Insight Generated

Scatter Plot Correlation strength and direction
Best for: correlation, clustering, interaction Bubble Chart Three-variable relationships
Heatmap Correlation matrix patterns
Pair Plot Multi-variable relationships

5.5 Composition Visualizations

Chart Type Insight Generated

Stacked Bar Chart Category composition across groups
Best for: part-to-whole, proportion Pie/Donut Chart Simple proportions
Treemap Hierarchical proportions
Waterfall Chart Sequential composition changes

6. Insight Generation Framework

6.1 Ask the Right Questions

Question Type Examples

Descriptive What happened? How much? How many?
Temporal When did it happen? Has it changed?
Comparative How does this compare to that?
Relational What is connected to what?
Causal What causes what?

6.2 Choose the Right Visualization

Question Visualization
”How has this changed over time?” Line chart
”How do categories compare?” Bar chart
”What is the distribution?” Histogram, box plot
”What is the relationship?” Scatter plot, heatmap
”What is the composition?” Stacked bar, treemap

6.3 Look for Patterns Systematically

Scan visualizations with intention:

• Overall pattern: What is the big picture?

• Exceptions: What doesn’t fit the pattern?

• Groups: Are there natural clusters?

• Trends: Direction over time?

• Relationships: Do variables move together?

6.4 Generate Hypotheses

Transform observations into testable statements:

• ”Sales drop in January → Hypothesis: Post-holiday spending reduction”

• ”Churn spikes after 30 days → Hypothesis: Onboarding is insufficient”

• ”Regional variations → Hypothesis: Local competitors affect pricing”

6.5 Validate and Refine

• Drill down into specific segments

• Add context variables

• Analyze different time scales

• Compare to external benchmarks

7. Common Insight Patterns

• Unexpected Finding: Data contradicts expectation → Shift focus

• Hidden Relationship: Two unrelated variables correlate → Identify drivers

• Tipping Point: Gradual change accelerates → Prepare for scaling

• Silent Majority: Large segment behaves differently → Reallocate resources

• Leading Indicator: Metric predicts another → Build forecasting

19
 20

8. From Insight to Action

Insight Action
”Sales peak in November” Increase inventory in October
”Customers churn after poor support” Improve support training
”Product A outperforms in region X” Increase marketing in region X
”Price sensitivity varies by segment” Implement segmented pricing

9. Common Pitfalls in Insight Generation

Pitfall Description Avoidance
Confirmation Bias Seeing only confirming evidence Seek disconfirming evidence
Overinterpretation Reading noise as signal Use statistical significance
Correlation/Causation Assuming related implies caused Investigate mechanisms
Missing Context Insights without business context Connect to business questions
Analysis Paralysis Too many insights, no action Prioritize actionable findings

11. Key Takeaways

Principle Description
Purpose Visualization exists to generate insights, not just show data
Process Systematic: visualize → detect patterns → hypothesize → validate → act
Patterns Look for trends, outliers, clusters, correlations, comparisons
Questions Start with clear questions to guide visualization choice
Validation Verify insights with additional analysis
Action Insights must lead to decisions

Case Study: End-to-End Visualization and Reporting Project

Project Title: E-commerce Sales Performance Dashboard
This case study simulates a real-world scenario where a data analyst uses Python to build an interactive reporting solution for a
company’s sales team. The goal is to transform raw transactional data into actionable insights through a structured workflow.

Phase 1: Problem Definition and Data Acquisition

Objective: The sales director requires a centralized dashboard to monitor key performance indicators (KPIs) such as revenue,
profit, and customer acquisition across different channels and regions. The aim is to replace manual Excel reports with an
automated, interactive solution.
Data Source: Raw transactional data from CSV files or SQL database with columns:

• order id

• order date

• customer id

• product category

• revenue

• cost

• country

• acquisition channel

Phase 2: Data Ingestion and Cleaning (using Pandas)

Process: Load the raw data into a Pandas DataFrame. Address missing values, inconsistent types, and duplicate records.
Implementation:
1 import pandas as pd
2

3 # Load data
4 df = pd . read_csv ( ’ sales_data . csv ’)
5

6 # Overview
7 df . info ()
8 df . describe ()
9

10 # Handle missing values

11 df [ ’ revenue ’ ]. fillna ( df [ ’ revenue ’ ]. mean () , inplace = True )
12 df [ ’ cost ’ ]. fillna ( df [ ’ cost ’ ]. median () , inplace = True )
13

14 # Remove duplicates
15 df . drop_duplicates ( subset = ’ order_id ’ , inplace = True )
16

17 # Convert date column

18 df [ ’ order_date ’] = pd . to_datetime ( df [ ’ order_date ’ ])

20
 0.9. PROGRAM 6: COMPLETE CUSTOMIZATION EXAMPLE 21

Phase 3: Exploratory Data Analysis (EDA)

Process: Understand data structure, distributions, and relationships to guide dashboard design.
Implementation:

• Univariate Analysis: Histograms and box plots for revenue and cost to detect outliers.

• Bivariate Analysis: Pair plots and correlation heatmaps using Seaborn.

• Feature Engineering:

– Extract month and year from order date

– Calculate profit: revenue - cost

Phase 4: Metrics Definition and Calculation

Process: Define KPIs to display on the dashboard.
Implementation:
1 # Total Revenue
2 total_revenue = df [ ’ revenue ’ ]. sum ()
3

4 # Total Profit
5 df [ ’ profit ’] = df [ ’ revenue ’] - df [ ’ cost ’]
6 total_profit = df [ ’ profit ’ ]. sum ()
7

8 # Average Order Value

9 aov = df [ ’ revenue ’ ]. mean ()
10

11 # Profit Margin
12 profit_margin = ( total_profit / total_revenue ) * 100

Phase 5: Static and Interactive Visualization

Process: Create plots to answer key business questions.
Implementation:

• Line Chart (Matplotlib/Seaborn): Monthly revenue trend with trend line.

• Bar Chart (Seaborn): Revenue by product category or acquisition channel.

• Treemap (Plotly): Hierarchical revenue contribution by product categories.

• Geographical Map (Plotly): Choropleth map of total revenue by country.

Phase 6: Dashboard Development and Automation

Process: Combine visualizations and KPIs into an interactive dashboard.
Implementation (Streamlit):
1 import streamlit as st
2 import plotly . express as px
3

4 st . title ( "E - commerce Sales Dashboard " )

5

6 # Sidebar filters
7 start_date = st . sidebar . date_input ( " Start Date " )
8 end_date = st . sidebar . date_input ( " End Date " )
9 category_filter = st . sidebar . multiselect ( " Category " , df [ ’ product_category ’ ]. unique () )
10

11 # KPI cards
12 st . metric ( " Total Revenue " , f " $ { total_revenue : ,.2 f } " )
13 st . metric ( " Total Profit " , f " $ { total_profit : ,.2 f } " )
14 st . metric ( " Profit Margin " , f " { profit_margin :.2 f }% " )
15

16 # Visualizations
17 fig_line = px . line ( df , x = ’ order_date ’ , y = ’ revenue ’ , title = ’ Monthly Revenue ’)
18 st . plotly_chart ( fig_line )

Phase 7: Deployment and Insights Communication

Deployment: Streamlit app deployed on Streamlit Cloud for team access.
Final Insights:

• Overall revenue reached $9.8M with an average margin of 30.9%.

• Web and Mobile App channels contributed 75% of total sales.

• Electronics category is the dominant revenue driver, followed by Home Goods.

21
Chapter-6
Data Engineering and Automation

6.1 Overview of Data Engineering in Applied Data Science

1. Introduction to Data Engineering
Data engineering is the foundation upon which data science is built. While data scientists focus on analysis, modeling, and
deriving insights, data engineering is concerned with the practical aspects of acquiring, preparing, and maintaining data for
these tasks. It involves the systems and processes that enable data to be accessible, reliable, and usable at scale.

2. The Role of Data Engineering in the Data Science Lifecycle

In an applied data science project, data engineering occupies the critical early stages and supports the entire lifecycle. The
typical workflow includes:

• Data Acquisition: Getting raw data from various sources such as databases, files, APIs, or streams.

• Data Preparation: Cleaning, transforming, and structuring data for analysis. This often represents the bulk of data
engineering work.

• Data Modeling & Analysis: The core work of a data scientist, which relies entirely on the quality of the prepared
data.

• Deployment & Monitoring: Putting models into production, which requires robust and reliable data pipelines to feed
the system.

3. Key Concepts and Components of Data Engineering

The following concepts are central to data engineering in applied data science:

• Data Pipelines and ETL/ELT: Moving data from source systems to destinations for analysis.

– ETL (Extract, Transform, Load): Transform data before loading.

– ELT (Extract, Load, Transform): Load first, transform in place, common with modern cloud data warehouses.

• Data Wrangling: Hands-on cleaning and transforming of raw data into structured, usable formats. Includes:

– Handling missing values

– Standardizing formats
– Correcting errors
– Merging datasets

• Data Automation: Using technology to execute repetitive tasks without manual intervention.

– Improves efficiency and reliability

– Automates quality checks and regular updates

• Imperative vs. Declarative Approaches in Automation:

– Imperative: Explicitly define step-by-step process (like a detailed recipe).

– Declarative: Specify desired outcome and let the system determine how to achieve it.

4. The Data Engineer and Data Scientist Relationship

The responsibilities of data engineers and data scientists are distinct but highly complementary:

• Data Engineer:

– Primary Focus: Building and maintaining data infrastructure

– Typical Tasks: Design and implement ETL/ELT pipelines, ensure data quality, manage data storage, optimize
retrieval
– Key Tools: SQL, Spark, cloud platforms (AWS, GCP, Azure), workflow managers (Airflow)

• Data Scientist:

– Primary Focus: Analyzing data to solve business problems

– Typical Tasks: Perform statistical analysis, build machine learning models, create visualizations, communicate in-
sights
– Key Tools: Python (pandas, scikit-learn), R, Jupyter notebooks

The handoff between these roles is a critical point in any project. A well-structured data engineering foundation allows data
scientists to focus on core competencies without being bogged down by data acquisition and cleaning.

1
2

6.2 Designing and Implementing ETL Pipelines

1. Introduction to ETL Pipelines
An ETL pipeline is a fundamental data engineering pattern that moves data from source systems to a destination data warehouse
or data lake. ETL stands for Extract, Transform, Load, representing the three core phases.
In applied data science, ETL pipelines are crucial because they ensure data scientists work with clean, reliable, and well-
structured data rather than spending most of their time on manual data wrangling.

2. The Three Phases of ETL

2.1 Extract Phase
The Extract phase involves retrieving data from various sources. This forms the foundation for the rest of the pipeline.
Common Data Sources:

• Relational databases: PostgreSQL, MySQL, SQL Server

• Files: CSV, Excel, JSON, Parquet, text files

• APIs: RESTful web services, SaaS platforms

• Data streams: Real-time event data from Kafka or similar platforms

Extraction Strategies:

• Full extraction: Pull all data each time

• Incremental extraction: Pull only changed data

• Batch processing: Process data in scheduled chunks

• Streaming: Continuous processing of incoming data

Key Python Considerations:

• requests for API calls

• pandas (read csv, read json, read excel) for file ingestion

• SQLAlchemy for database connections

• Handle API rate limits and authentication

2.2 Transform Phase

The Transform phase cleans, standardizes, and enriches data for analysis. This is typically the most complex stage.
Common Transformation Operations:

• Data cleaning: Remove duplicates, handle missing values (drop duplicates(), fillna(), dropna())

• Data standardization: Ensure consistent formats ([Link] datetime(), string operations)

• Data type conversion: Correct data types (astype(), infer objects())

• Filtering: Remove irrelevant records (Boolean indexing, query())

• Joining/merging: Combine datasets (merge(), concat())

• Aggregation: Summarize data (groupby(), agg())

• Feature engineering: Create derived columns (custom functions, apply())

• Validation: Check data quality (custom logic)

Transformation Approaches:

• In-Notebook: Interactive exploration (common in EDA)

• Scripted: Reusable Python functions and classes

• Distributed: Using Spark for large datasets

2.3 Load Phase

The Load phase writes the transformed data to its final destination.
Load Destinations:

• Data warehouses: Snowflake, Redshift, BigQuery

• Data lakes: S3, Azure Data Lake, Google Cloud Storage

• Databases: PostgreSQL, MySQL

• File systems: Parquet, CSV

Load Strategies:

• Full load: Replace entire destination table

• Incremental load: Append new records

2
 3

• Upsert: Update existing, insert new

• Partitioned load: Write partitioned by date/category

File Formats for Loading:

• Parquet: Columnar, compressed, preserves types

• CSV: Human-readable, widely compatible

• JSON: Flexible for nested data

3. ETL vs. ELT

Traditional ETL: Transform data before loading. Modern ELT: Load first, transform inside the warehouse.
Key Differences:

• Transform location: ETL on server, ELT in warehouse

• Performance: ETL limited, ELT leverages warehouse scalability

• Raw data availability: ETL loses raw, ELT preserves raw

• Use case: ETL for limited warehouse power, ELT for cloud warehouses

4. Designing Robust ETL Pipelines

Best Practices:

• Modular Architecture: Separate extraction, transformation, and loading components

• Data Quality Validation: Schema checks, completeness, uniqueness, range, and count validation

• Error Handling and Logging: Comprehensive logs, defensive reading, isolate bad data, alerting

• Incremental Processing: Process only new/changed data, track timestamps, ensure idempotency

• Orchestration: Use tools like Apache Airflow, Prefect, or cloud-native services for scheduling and dependency manage-
ment

5. Python Libraries for ETL

• Pandas: Core data manipulation and transformation

• NumPy: Numerical operations

• SQLAlchemy: Database connections and ORM

• Requests: API data extraction

• PySpark: Distributed processing for big data

• Apache Airflow: Workflow orchestration

• Great Expectations: Data validation and testing

• Boto3: AWS services integration (S3, Redshift)

7. Practical Example: A Simple ETL Pipeline in Python

While reference books do not provide a single complete example, the following pattern combines concepts from multiple sources.

Python ETL Pipeline Structure

1 import pandas as pd
2 import logging
3 from datetime import datetime
4
5 # Configure logging
6 logging . basicConfig ( level = logging . INFO )
7 logger = logging . getLogger ( __name__ )
8

9 def extract_data ( file_path ) :

10 " " " Extract data from source file " " "
11 logger . info ( f " Extracting data from { file_path } " )
12 df = pd . read_csv ( file_path ) # Could also be JSON , Excel , API , etc .
13 logger . info ( f " Extracted { len ( df ) } rows " )
14 return df
15
16 def transform_data ( df ) :
17 " " " Clean and transform the data " " "
18 logger . info ( " Starting transformation " )
19
20 # Remove duplicates
21 initial_count = len ( df )
22 df = df . drop_duplicates ()
23 logger . info ( f " Removed { initial_count - len ( df ) } duplicates " )
24
25 # Handle missing values
26 df = df . fillna ({

3
 4

27 ’ numeric_column ’: 0 ,
28 ’ cat eg or ic al _c ol um n ’: ’ Unknown ’
29 })
30
31 # Standardize date formats
32 df [ ’ date ’] = pd . to_datetime ( df [ ’ date ’] , errors = ’ coerce ’)
33
34 # Filter invalid records
35 df = df [ df [ ’ value ’] > 0]
36
37 # Create derived columns
38 df [ ’ processed_date ’] = datetime . now ()
39
40 logger . info ( f " Transformation complete . { len ( df ) } rows remain " )
41 return df
42
43 def validate_data ( df ) :
44 " " " Run quality checks " " "
45 logger . info ( " Validating data " )
46
47 # Schema validation
48 expected_columns = [ ’ id ’ , ’ date ’ , ’ value ’ , ’ category ’]
49 assert all ( col in df . columns for col in expected_columns ) , " Missing columns "
50

51 # Completeness check
52 null_counts = df . isnull () . sum ()
53 logger . info ( f " Null counts : { null_counts } " )
54
55 # Uniqueness check
56 assert df [ ’ id ’ ]. is_unique , " Duplicate IDs found "
57
58 logger . info ( " Validation passed " )
59 return True
60
61 def load_data ( df , output_path ) :
62 " " " Load to destination " " "
63 logger . info ( f " Loading { len ( df ) } rows to { output_path } " )
64 df . to_parquet ( output_path , index = False )
65 logger . info ( " Load complete " )
66
67 def run_pipeline ( input_file , output_file ) :
68 " " " Orchestrate the full ETL process " " "
69 try :
70 logger . info ( " Starting ETL pipeline " )
71
72 # Extract
73 data = extract_data ( input_file )
74

75 # Transform
76 transformed = transform_data ( data )
77
78 # Validate
79 validate_data ( transformed )
80

81 # Load
82 load_data ( transformed , output_file )
83
84 logger . info ( " Pipeline completed successfully " )
85
86 except Exception as e :
87 logger . error ( f " Pipeline failed : { str ( e ) } " )
88 raise
89
90 # Execute pipeline
91 if __name__ == " __main__ " :
92 run_pipeline ( " data / raw / input . csv " , " data / curated / output . parquet " )

Extract Phase

• Function: extract data(file path)

• Reads data from a CSV file (can be JSON, Excel, API, etc.)

• Logs the number of rows extracted

Transform Phase

• Function: transform data(df)

• Removes duplicates

• Handles missing values using fillna()

• Standardizes date formats ([Link] datetime)

• Filters invalid records (e.g., value > 0)

• Creates derived columns (e.g., processed date = [Link]())

• Logs row counts before and after transformation

4
 5

Validation Phase
• Function: validate data(df)

• Schema validation: checks for expected columns

• Completeness check: counts null values

• Uniqueness check: ensures unique IDs

• Logs validation results

Load Phase
• Function: load data(df, output path)

• Writes transformed data to Parquet file

• Logs completion of loading

Pipeline Orchestration
• Function: run pipeline(input file, output file)

• Steps executed in order: Extract → Transform → Validate → Load

• Logs start, success, and errors

8. Scaling Considerations
• Pandas: For datasets that fit in memory (up to a few GB)

• Dask: For larger-than-memory datasets on a single machine

• Spark: Distributed processing across clusters

• Cloud Services: Managed ETL (AWS Glue, Google Dataflow)

9. Testing ETL Pipelines

Reliable pipelines require comprehensive testing:

• Unit tests: Test individual transformation functions

• Integration tests: Test the flow between components

• Data quality tests: Validate against known good datasets

• Regression tests: Ensure changes don’t break existing functionality

10. Summary
• ETL Definition: Extract, Transform, Load – the three phases of data movement

• Extract: Get data from sources; choose full or incremental strategies

• Transform: Clean, standardize, enrich; the most complex phase

• Load: Write to destination; choose full or incremental loading

• Design Principles: Modularity, validation, error handling, logging

• Python Tools: Pandas for transformation, orchestration tools for workflow

• Scaling: Move from Pandas to Dask to Spark as data grows

6.3 Automating Workflows with Schedulers (CRON, Schedule)

1. Introduction to Workflow Automation
Workflow automation is the process of automating repetitive tasks and processes to improve efficiency, reduce errors, and ensure
consistency. In data engineering, schedulers execute data pipelines, scripts, and maintenance tasks at predetermined times
without manual intervention. This allows data professionals to focus on higher-value activities while routine operations run
reliably in the background.

2. CRON: The Unix/Linux Scheduler

2.1 What is CRON?
Cron is a time-based job scheduler in Unix-like operating systems. It runs as a daemon (background process) and executes
commands or scripts at specified dates and times. Cronjobs are stored in a crontab file, and Cron is commonly used on systems
like Ubuntu Linux.

5
6

2.2 Installing and Managing Cron

• Update package index: apt update

• Install Cron: apt install cron

• Ensure service is enabled and running: systemctl enable [Link]

Output example:

Synchronizing state of [Link] with SysV service script with /lib/systemd/systemd-sysv-install.

Executing: /lib/systemd/systemd-sysv-install enable cron

2.3 Understanding Crontab Structure

Crontab entries follow this order:

• MINUTE: 0-59

• HOUR: 0-23

• DAY OF MONTH: 1-31

• MONTH: 1-12 (or JAN-DEC)

• DAY OF WEEK: 0-6 (0 = Sunday, or SUN-SAT)

• COMMAND: Path to the script/command

Example: Run a backup every night at 4:30 AM:

30 04 * * * /var/www/websites/[Link]

2.4 Special Characters in Cron Syntax

• * : Wildcard representing ”all” (e.g., * * * * * runs every minute)

• , : List of values (e.g., 0,30 * * * * runs at minute 0 and 30)

• - : Range (e.g., 0-29 * * * * runs every minute for first 30 minutes)

• */ : Step value (e.g., */10 * * * * runs every 10 minutes)

• ? : No specific value (used in Quartz Scheduler)

2.5 Special Time Strings

• @reboot : Run once at startup

• @yearly / @annually : Run once a year

• @monthly : Run once a month

• @weekly : Run once a week

• @daily / @midnight : Run once a day

• @hourly : Run once an hour

2.6 Managing Crontabs

• crontab -e : Edit crontab (creates new if none exists)

• crontab -l : List current cronjobs

• crontab -r : Remove all cronjobs

• crontab -r -i : Remove with confirmation

2.7 Redirecting Output

To capture output and errors for logging:

0 5 * * * /path/to/[Link] >> /var/log/[Link] 2>&1

2.8 Practical Cron Examples

• 30 14 15 6 3 /path/to/[Link] : Run on Wednesday, June 15, at 2:30 PM

• 0 9-17/2 * * 1-5 /path/to/[Link] : Run every second hour between 9 AM-5 PM, Monday-Friday

• @reboot /path/to/[Link] : Run at system startup

2.9 Cron Daemon Management

• Check status (systemd): systemctl status cron

• Check status (older systems): service cron status

• Restart cron if needed: systemctl start cron

6
 0.1. SCHEDULE LIBRARY FOR WORKFLOW AUTOMATION 7

0.1 Schedule Library for Workflow Automation

0.1.1 Overview
The schedule library is a Python package that provides an in-process scheduler for periodic jobs. It offers a simple and readable
syntax for scheduling functions to run at specific intervals or times.

0.1.2 Installation and Setup

1 pip install schedule

0.1.3 Core Concepts

Job
A job is a task (function) scheduled to run at specific times. Each job includes:

• The function that will be executed

• The timing interval for execution

• Optional arguments passed to the function

Scheduler
The scheduler manages all registered jobs and determines when each task should be executed according to its schedule.

Job Runner
The run pending() function checks scheduled jobs and executes those whose execution time has arrived.

0.1.4 Basic Syntax and Structure

Fundamental Pattern

1 import schedule
2 import time
3
4 def my_task () :
5 print ( " Task executed " )
6
7 # Schedule the task
8 schedule . every (10) . minutes . do ( my_task )
9
10 # Keep the scheduler running
11 while True :
12 schedule . run_pending ()
13 time . sleep (1)

0.1.5 Time Units and Intervals

Seconds

1 schedule . every (10) . seconds . do ( task ) # Every 10 seconds

2 schedule . every () . second . do ( task ) # Every second

Minutes

1 schedule . every (5) . minutes . do ( task ) # Every 5 minutes

2 schedule . every () . minute . do ( task ) # Every minute

Hours

1 schedule . every (3) . hours . do ( task ) # Every 3 hours

2 schedule . every () . hour . do ( task ) # Every hour

Days

1 schedule . every (2) . days . do ( task ) # Every 2 days

2 schedule . every () . day . do ( task ) # Every day

Weeks

1 schedule . every () . week . do ( task ) # Every week

2 schedule . every (2) . weeks . do ( task ) # Every 2 weeks

7
 8

0.1.6 Specific Day Scheduling

Weekday Scheduling

1 # Specific weekdays
2 schedule . every () . monday . do ( task )
3 schedule . every () . tuesday . do ( task )
4 schedule . every () . wednesday . do ( task )
5 schedule . every () . thursday . do ( task )
6 schedule . every () . friday . do ( task )
7 schedule . every () . saturday . do ( task )
8 schedule . every () . sunday . do ( task )
9
10 # Weekdays only ( Monday to Friday )
11 schedule . every () . monday . to . friday . do ( task )
12

13 # Multiple days
14 schedule . every () . monday . and . wednesday . do ( task )

6.4 Logging, Monitoring, and Error Handling in Data Pipelines

Overview
In data engineering, logging, monitoring, and error handling are essential practices that ensure pipeline reliability, maintain-
ability, and observability. These mechanisms help detect problems early, diagnose failures quickly, and maintain data quality
throughout the workflow.

1. Logging Fundamentals
Python’s Logging Module
Python provides a built-in logging module that offers a flexible framework for generating log messages from applications.

Logging Levels
Python defines five standard logging levels arranged by increasing severity:

• DEBUG (10) – Detailed information used for diagnostic purposes.

• INFO (20) – Confirmation that operations are functioning as expected.

• WARNING (30) – Indicates unexpected situations or potential problems.

• ERROR (40) – A serious issue preventing a function from executing properly.

• CRITICAL (50) – A severe error indicating a system-level failure requiring immediate attention.

Basic Logging Configuration

1 import logging
2
3 # Basic configuration
4 logging . basicConfig (
5 level = logging . INFO ,
6 format = ’ %( asctime ) s - %( name ) s - %( levelname ) s - %( message ) s ’ ,
7 filename = ’ pipeline . log ’ ,
8 filemode = ’a ’
9 )
10
11 # Create different types of logs
12 logging . debug ( " Variable has value : % s " , path )
13 logging . info ( " Data has been transformed and will now be loaded " )
14 logging . warning ( " Unexpected number of rows detected : % d " , row_count )
15 logging . error ( " KeyError arose in execution : % s " , str ( e ) )

Example output:

INFO: Data has been transformed and will now be loaded

WARNING: Unexpected number of rows detected
ERROR: KeyError arose in execution

Module-Level Logging Pattern

For production pipelines, it is recommended to use named loggers rather than the root logger.
1 import logging
2

3 # Create module - level logger

4 logger = logging . getLogger ( __name__ )
5
6 def process_data () :
7 logger . info ( " Starting data processing " )
8 try :
9 # Processing logic
10 logger . debug ( " Processing row % d " , row_num )
11 except Exception as e :
12 logger . error ( " Processing failed : % s " , str ( e ) )
13 raise

8
 0.1. SCHEDULE LIBRARY FOR WORKFLOW AUTOMATION 9

2. Advanced Logging Configuration

Using Dictionary Configuration
For complex pipelines, centralized logging configuration can be implemented using dictConfig.
1 import logging . config
2
3 LOGGING_CONFIG = {
4 ’ version ’: 1 ,
5 ’ formatters ’: {
6 ’ detailed ’: {
7 ’ format ’: ’ %( asctime ) s - %( name ) s - %( levelname ) s - %( message ) s ’
8 },
9 ’ simple ’: {
10 ’ format ’: ’ %( levelname ) s : %( message ) s ’
11 }
12 },
13 ’ handlers ’: {
14 ’ console ’: {
15 ’ class ’: ’ logging . StreamHandler ’ ,
16 ’ level ’: ’ INFO ’ ,
17 ’ formatter ’: ’ simple ’ ,
18 ’ stream ’: ’ ext :// sys . stdout ’
19 },
20 ’ file ’: {
21 ’ class ’: ’ logging . handlers . R ot a t in g F il e H an d l er ’ ,
22 ’ level ’: ’ DEBUG ’ ,
23 ’ formatter ’: ’ detailed ’ ,
24 ’ filename ’: ’ pipeline . log ’ ,
25 ’ maxBytes ’: 10485760 ,
26 ’ backupCount ’: 5
27 },
28 ’ error_file ’: {
29 ’ class ’: ’ logging . handlers . R ot a t in g F il e H an d l er ’ ,
30 ’ level ’: ’ ERROR ’ ,
31 ’ formatter ’: ’ detailed ’ ,
32 ’ filename ’: ’ errors . log ’ ,
33 ’ maxBytes ’: 10485760 ,
34 ’ backupCount ’: 3
35 }
36 },
37 ’ loggers ’: {
38 ’ pipeline ’: {
39 ’ handlers ’: [ ’ console ’ , ’ file ’ , ’ error_file ’] ,
40 ’ level ’: ’ DEBUG ’ ,
41 ’ propagate ’: False
42 }
43 },
44 ’ root ’: {
45 ’ handlers ’: [ ’ console ’] ,
46 ’ level ’: ’ WARNING ’
47 }
48 }
49
50 logging . config . dictConfig ( LOGGING_CONFIG )
51 logger = logging . getLogger ( ’ pipeline ’)

Structured Logging with JSON

For better integration with log aggregation tools and monitoring platforms, log messages can be structured in JSON format.
Structured logging allows logs to be parsed, searched, and analyzed more easily by centralized logging systems.
1 import json
2 import logging
3 from datetime import datetime
4
5 class JSONFormatter ( logging . Formatter ) :
6 def format ( self , record ) :
7 log_record = {
8 ’ timestamp ’: datetime . utcnow () . isoformat () ,
9 ’ level ’: record . levelname ,
10 ’ logger ’: record . name ,
11 ’ message ’: record . getMessage () ,
12 ’ module ’: record . module ,
13 ’ function ’: record . funcName
14 }
15

16 # Add exception info if present

17 if record . exc_info :
18 log_record [ ’ exception ’] = self . formatException ( record . exc_info )
19
20 # Add custom attributes
21 if hasattr ( record , ’ pipeline_id ’) :
22 log_record [ ’ pipeline_id ’] = record . pipeline_id
23
24 return json . dumps ( log_record )
25
26 # Configure JSON logging
27 handler = logging . StreamHandler ()
28 handler . setFormatter ( JSONFormatter () )
29
30 logger = logging . getLogger ( ’ pipeline ’)
31 logger . addHandler ( handler )
32 logger . setLevel ( logging . INFO )
33

9
 10

34 # Usage with custom attributes

35 logger . info ( " Pipeline started " , extra ={ ’ pipeline_id ’: ’ daily_etl_001 ’ })

3. Error Handling Strategies

Try-Except Blocks
Basic exception handling combined with logging provides detailed context when failures occur during pipeline execution.
1 import logging
2 import pandas as pd
3
4 logger = logging . getLogger ( __name__ )
5
6 def extract_data ( source_path ) :
7 " " " Extract data from source with error handling " " "
8 try :
9 logger . info ( f " Extracting data from { source_path } " )
10 data = pd . read_csv ( source_path )
11 logger . info ( f " Successfully extracted { len ( data ) } rows " )
12 return data
13

14 except Fi leNotF oundEr ror as e :

15 logger . error ( f " Source file not found : { source_path } " )
16 raise # Re - raise after logging
17
18 except pd . errors . EmptyDataError as e :
19 logger . error ( f " Source file is empty : { source_path } " )
20 raise
21
22 except Exception as e :
23 logger . error (
24 f " Unexpected error during extraction : { str ( e ) } " ,
25 exc_info = True
26 )
27 raise

Handling Specific Exceptions

Different types of exceptions should be handled appropriately to ensure that the pipeline can either recover from errors or fail
gracefully with meaningful logs.
1 def transform_data ( raw_data ) :
2 " " " Transform data with specific error handling " " "
3 try :
4 # Try to filter by price_change
5 clean_data = raw_data [ raw_data [ ’ price_change ’ ]. notna () ]
6 logger . info ( " Successfully filtered DataFrame by ’ price_change ’" )
7 return clean_data
8
9 except KeyError as ke :
10 # Handle missing column by creating it
11 logger . warning ( f " { ke }: Cannot filter by ’ price_change ’, creating column " )
12 raw_data [ ’ price_change ’] = raw_data [ ’ close ’] - raw_data [ ’ open ’]
13 clean_data = raw_data [ raw_data [ ’ price_change ’ ]. notna () ]
14 return clean_data
15
16 except Exception as e :
17 logger . error ( f " Transformation failed : { str ( e ) } " , exc_info = True )
18 raise

Data Validation and Quality Checks

Data validation ensures that datasets meet expected quality standards before further processing. Performing validation early
in the pipeline helps detect data issues quickly and prevents downstream failures.
1 def validate_data ( df , expected_columns , min_rows =1) :
2 " " " Validate dataframe quality " " "
3 errors = []
4

5 # Check if dataframe is empty

6 if len ( df ) == 0:
7 errors . append ( " DataFrame is empty " )
8
9 # Check minimum rows
10 if len ( df ) < min_rows :
11 errors . append (
12 f " DataFrame has { len ( df ) } rows , minimum required : { min_rows } "
13 )
14
15 # Check expected columns
16 missing_columns = set ( expected_columns ) - set ( df . columns )
17 if missing_columns :
18 errors . append ( f " Missing columns : { missing_columns } " )
19
20 # Check for nulls in critical columns
21 for col in expected_columns :
22 null_count = df [ col ]. isnull () . sum ()
23 if null_count > 0:
24 errors . append (
25 f " Column ’{ col } ’ has { null_count } null values "
26 )
27
28 if errors :

10
 0.1. SCHEDULE LIBRARY FOR WORKFLOW AUTOMATION 11

29 error_msg = " ; " . join ( errors )

30 logger . error ( f " Data validation failed : { error_msg } " )
31 raise ValueError ( f " Data validation failed : { error_msg } " )
32
33 logger . info (
34 f " Data validation passed : { len ( df ) } rows , { len ( df . columns ) } columns "
35 )
36 return True
37
38 # Usage
39 try :
40 validate_data ( data ,
41 expected_columns =[ ’ id ’ , ’ date ’ , ’ value ’] ,
42 min_rows =100)
43 except ValueError as e :
44 # Handle validation failure
45 logger . critical (
46 f " Pipeline stopped due to validation error : { e } "
47 )
48 send_alert ( f " Pipeline validation failed : { e } " )
49 sys . exit (1)

4. Retry Mechanisms
Basic Retry Logic
Retry mechanisms are used to handle temporary or transient failures such as network timeouts, API rate limits, or temporary
service unavailability. A common approach is to retry the operation multiple times with an increasing delay between attempts,
known as exponential backoff.
1 import time
2 import logging
3

4 logger = logging . getLogger ( __name__ )

5
6 def retry_operation ( operation , max_retries =3 , initial_delay =1) :
7 """
8 Retry an operation with exponential backoff
9

10 Args :
11 operation : Callable to retry
12 max_retries : Maximum number of retry attempts
13 initial_delay : Initial delay in seconds ( doubles each retry )
14 """
15 for attempt in range ( max_retries ) :
16 try :
17 logger . info ( f " Attempt { attempt + 1}/{ max_retries } " )
18 return operation ()
19
20 except Exception as e :
21 wait_time = initial_delay * (2 ** attempt )
22
23 if attempt < max_retries - 1:
24 logger . warning (
25 f " Attempt { attempt + 1} failed : { str ( e ) }. "
26 f " Retrying in { wait_time } seconds ... "
27 )
28 time . sleep ( wait_time )
29 else :
30 logger . error ( f " All { max_retries } attempts failed " )
31 raise # Re - raise on final failure
32
33

34 # Usage example
35 def uns ta bl e_ op er at io n () :
36 # Simulate flaky operation
37 if random . random () < 0.7: # 70% failure rate
38 raise ConnectionError ( " Network timeout " )
39 return " Success "
40
41
42 try :
43 result = retry_operation ( unstable_operation , max_retries =5)
44 logger . info ( f " Operation succeeded : { result } " )
45 except Exception as e :
46 logger . error ( f " Operation failed after retries : { e } " )

Using Tenacity Library

The tenacity library provides advanced retry capabilities with configurable stopping conditions, exponential backoff strategies,
and logging integration. It is commonly used for handling transient failures such as API timeouts or network interruptions.
1 from tenacity import (
2 retry ,
3 stop_after_attempt ,
4 wait_exponential ,
5 retry_if_exception_type ,
6 before_log ,
7 after_log
8 )
9 import logging
10 import requests
11
12 logger = logging . getLogger ( __name__ )
13

11
 12

14 @retry (
15 stop = sto p_ af te r_ at te mp t (5) ,
16 wait = wait_exponential ( multiplier =1 , min =4 , max =30) ,
17 retry = r e t r y _ i f _ e x c e p t i o n _ t y p e ((
18 requests . exceptions . Timeout ,
19 requests . exceptions . ConnectionError
20 )),
21 before = before_log ( logger , logging . INFO ) ,
22 after = after_log ( logger , logging . INFO )
23 )
24 def f et c h _d a t a_ f r om _ a pi ( url ) :
25 " " " Fetch data with automatic retry for transient failures " " "
26 logger . info ( f " Fetching data from { url } " )
27 response = requests . get ( url , timeout =5)
28 response . raise_for_status ()
29 return response . json ()
30
31 # Usage
32 try :
33 data = f et c h _d a t a_ f r om _ a pi ( " https :// api . example . com / data " )
34 except Exception as e :
35 logger . error ( f " API request failed after retries : { e } " )

5. Monitoring and Observability

Monitoring and observability are essential for understanding how data pipelines behave in production. By tracking performance
metrics and execution details, engineers can identify bottlenecks, detect failures quickly, and improve pipeline reliability.

Pipeline Performance Monitoring

Execution time and resource usage can be tracked using decorators that wrap pipeline functions.
1 import time
2 import logging
3 from functools import wraps
4 from datetime import datetime
5
6 logger = logging . getLogger ( __name__ )
7
8 def monitor_pipeline ( func ) :
9 " " " Decorator to monitor pipeline execution " " "
10 @wraps ( func )
11 def wrapper (* args , ** kwargs ) :
12 start_time = time . time ()
13 start_datetime = datetime . now ()
14

15 logger . info ( f " Starting { func . __name__ } at { start_datetime } " )

16
17 try :
18 result = func (* args , ** kwargs )
19
20 end_time = time . time ()
21 duration = end_time - start_time
22
23 logger . info (
24 f " Completed { func . __name__ } in { duration :.2 f } seconds . "
25 f " Rows processed : { len ( result ) if hasattr ( result , ’ __len__ ’) else ’N / A ’} "
26 )
27
28 # Record metrics for monitoring
29 record_metric ( f " { func . __name__ }. duration " , duration )
30 record_metric ( f " { func . __name__ }. success " , 1)
31

32 return result
33
34 except Exception as e :
35 end_time = time . time ()
36 duration = end_time - start_time
37

38 logger . error (
39 f " Failed { func . __name__ } after { duration :.2 f } seconds : { str ( e ) } "
40 )
41 record_metric ( f " { func . __name__ }. duration " , duration )
42 record_metric ( f " { func . __name__ }. success " , 0)
43 raise
44
45 return wrapper
46
47
48 @m onitor _pipel ine
49 def etl_pipeline () :
50 # Pipeline logic here
51 pass

Custom Metrics Collection

Custom metrics collection helps track pipeline execution details such as runtime, processed rows, and errors. These metrics can
be stored in files or sent to monitoring systems for later analysis.
1 import json
2 import os
3 from datetime import datetime
4
5 class PipelineMetrics :

12
 0.1. SCHEDULE LIBRARY FOR WORKFLOW AUTOMATION 13

6 def __init__ ( self , pipeline_name ) :

7 self . pipeline_name = pipeline_name
8 self . metrics = {
9 ’ pipeline ’: pipeline_name ,
10 ’ start_time ’: None ,
11 ’ end_time ’: None ,
12 ’ duration ’: None ,
13 ’ status ’: None ,
14 ’ rows_read ’: 0 ,
15 ’ rows_written ’: 0 ,
16 ’ errors ’: []
17 }
18
19 def start ( self ) :
20 self . metrics [ ’ start_time ’] = datetime . now () . isoformat ()
21
22 def stop ( self , status = ’ success ’) :
23 self . metrics [ ’ end_time ’] = datetime . now () . isoformat ()
24 self . metrics [ ’ duration ’] = (
25 datetime . fromisoformat ( self . metrics [ ’ end_time ’ ]) -
26 datetime . fromisoformat ( self . metrics [ ’ start_time ’ ])
27 ) . total_seconds ()
28 self . metrics [ ’ status ’] = status
29

30 def add_error ( self , error ) :

31 self . metrics [ ’ errors ’ ]. append ({
32 ’ timestamp ’: datetime . now () . isoformat () ,
33 ’ message ’: str ( error )
34 })
35

36 def save ( self , metrics_dir = ’ metrics ’) :

37 os . makedirs ( metrics_dir , exist_ok = True )
38 filename = f " { metrics_dir }/{ self . pipeline_name } _ { datetime . now () . strftime ( ’% Y % m % d_ % H % M % S ’) }. json
"
39
40 with open ( filename , ’w ’) as f :
41 json . dump ( self . metrics , f , indent =2)
42
43 logger . info ( f " Metrics saved to { filename } " )
44
45
46 # Usage
47 metrics = PipelineMetrics ( ’ daily_etl ’)
48 metrics . start ()
49
50 try :
51 # Pipeline execution
52 metrics . metrics [ ’ rows_read ’] = 15000
53 metrics . metrics [ ’ rows_written ’] = 14980
54 metrics . stop ( ’ success ’)
55
56 except Exception as e :
57 metrics . add_error ( e )
58 metrics . stop ( ’ failure ’)
59 raise
60
61 finally :
62 metrics . save ()

6. Alerting Mechanisms
Alerting mechanisms notify engineers when a pipeline fails or encounters unusual conditions. Alerts can be delivered through
email, messaging platforms, or monitoring systems.

Simple Alert System

1 import smtplib
2 import requests
3 from email . mime . text import MIMEText
4 from email . mime . multipart import MIMEMultipart
5
6 class AlertManager :
7 def __init__ ( self , config ) :
8 self . config = config
9 self . logger = logging . getLogger ( __name__ )
10

11 def send_email_alert ( self , subject , message , severity = ’ ERROR ’) :

12 " " " Send email alert " " "
13 try :
14 msg = MIMEMultipart ()
15 msg [ ’ From ’] = self . config [ ’ email_from ’]
16 msg [ ’ To ’] = self . config [ ’ email_to ’]
17 msg [ ’ Subject ’] = f " [{ severity }] { subject } "
18
19 msg . attach ( MIMEText ( message , ’ plain ’) )
20
21 server = smtplib . SMTP (
22 self . config [ ’ smtp_server ’] ,
23 self . config [ ’ smtp_port ’]
24 )
25 server . starttls ()
26 server . login (
27 self . config [ ’ smtp_user ’] ,
28 self . config [ ’ smtp_password ’]

13
 14

29 )
30 server . send_message ( msg )
31 server . quit ()
32
33 self . logger . info ( f " Email alert sent : { subject } " )
34

35 except Exception as e :
36 self . logger . error ( f " Failed to send email alert : { e } " )
37
38 def send_slack_alert ( self , message , severity = ’ ERROR ’) :
39 " " " Send Slack alert via webhook " " "
40 try :
41 webhook_url = self . config [ ’ slack_webhook ’]
42
43 color = {
44 ’ INFO ’: ’ #36 a64f ’ ,
45 ’ WARNING ’: ’# ffcc00 ’ ,
46 ’ ERROR ’: ’# ff0000 ’ ,
47 ’ CRITICAL ’: ’ #8 b0000 ’
48 }. get ( severity , ’ #808080 ’)
49
50 payload = {
51 ’ attachments ’: [{
52 ’ color ’: color ,
53 ’ title ’: f ’ Pipeline Alert : { severity } ’ ,
54 ’ text ’: message ,
55 ’ fields ’: [
56 {
57 ’ title ’: ’ Pipeline ’ ,
58 ’ value ’: self . config . get ( ’ pipeline_name ’ , ’ unknown ’) ,
59 ’ short ’: True
60 },
61 {
62 ’ title ’: ’ Time ’ ,
63 ’ value ’: datetime . now () . strftime (
64 ’%Y -% m -% d % H :% M :% S ’
65 ),
66 ’ short ’: True
67 }
68 ]
69 }]
70 }
71
72 response = requests . post ( webhook_url , json = payload )
73 response . raise_for_status ()
74
75 self . logger . info (
76 f " Slack alert sent : { message [:50]}... "
77 )
78
79 except Exception as e :
80 self . logger . error (
81 f " Failed to send Slack alert : { e } "
82 )
83
84
85 # Usage
86 alert_config = {
87 ’ email_from ’: ’ pipeline@example . com ’ ,
88 ’ email_to ’: ’ team@example . com ’ ,
89 ’ smtp_server ’: ’ smtp . gmail . com ’ ,
90 ’ smtp_port ’: 587 ,
91 ’ smtp_user ’: ’ user@gmail . com ’ ,
92 ’ smtp_password ’: ’ password ’ ,
93 ’ slack_webhook ’: ’ https :// hooks . slack . com / services / xxx / yyy / zzz ’ ,
94 ’ pipeline_name ’: ’ daily_etl ’
95 }
96
97 alerts = AlertManager ( alert_config )
98
99 try :
100 # Pipeline execution
101 pass
102
103 except Exception as e :
104 alerts . send_slack_alert (
105 f " Pipeline failed : { str ( e ) } " ,
106 severity = ’ ERROR ’
107 )
108 alerts . send_email_alert (
109 " Pipeline Failure Alert " ,
110 f " The daily ETL pipeline failed with error :\ n \ n { str ( e ) } "
111 )

7. Comprehensive Pipeline Example

This section presents a complete ETL pipeline that integrates logging, monitoring, retry mechanisms, and error handling. The
example demonstrates how production pipelines can be structured to ensure reliability, observability, and maintainability.

Complete ETL Pipeline with Logging, Monitoring, and Error Handling

1 import logging
2 import logging . config
3 import time

14
 0.1. SCHEDULE LIBRARY FOR WORKFLOW AUTOMATION 15

4 import json
5 from datetime import datetime
6 from tenacity import retry , stop_after_attempt , wait_exponential
7 import pandas as pd
8 import requests
9

10 # Configure logging
11 LOGGING_CONFIG = {
12 ’ version ’: 1 ,
13 ’ formatters ’: {
14 ’ detailed ’: {
15 ’ format ’: ’ %( asctime ) s - %( name ) s - %( levelname ) s - %( message ) s ’
16 },
17 ’ json ’: {
18 ’ () ’: ’ pipeline_utils . JSONFormatter ’
19 }
20 },
21 ’ handlers ’: {
22 ’ console ’: {
23 ’ class ’: ’ logging . StreamHandler ’ ,
24 ’ level ’: ’ INFO ’ ,
25 ’ formatter ’: ’ detailed ’
26 },
27 ’ file ’: {
28 ’ class ’: ’ logging . handlers . R ot a t in g F il e H an d l er ’ ,
29 ’ level ’: ’ DEBUG ’ ,
30 ’ formatter ’: ’ json ’ ,
31 ’ filename ’: ’ pipeline . log ’ ,
32 ’ maxBytes ’: 10485760 ,
33 ’ backupCount ’: 5
34 },
35 ’ error_file ’: {
36 ’ class ’: ’ logging . handlers . R ot a t in g F il e H an d l er ’ ,
37 ’ level ’: ’ ERROR ’ ,
38 ’ formatter ’: ’ json ’ ,
39 ’ filename ’: ’ errors . log ’ ,
40 ’ maxBytes ’: 10485760 ,
41 ’ backupCount ’: 3
42 }
43 },
44 ’ loggers ’: {
45 ’ etl_pipeline ’: {
46 ’ handlers ’: [ ’ console ’ , ’ file ’ , ’ error_file ’] ,
47 ’ level ’: ’ DEBUG ’ ,
48 ’ propagate ’: False
49 }
50 }
51 }
52
53 logging . config . dictConfig ( LOGGING_CONFIG )
54 logger = logging . getLogger ( ’ etl_pipeline ’)
55
56 class PipelineMetrics :
57 " " " Track pipeline execution metrics " " "
58
59 def __init__ ( self , pipeline_name ) :
60 self . pipeline_name = pipeline_name
61 self . metrics = {
62 ’ pipeline ’: pipeline_name ,
63 ’ run_id ’: datetime . now () . strftime ( ’% Y % m % d_ % H % M % S ’) ,
64 ’ start_time ’: None ,
65 ’ end_time ’: None ,
66 ’ duration ’: None ,
67 ’ status ’: ’ running ’ ,
68 ’ stages ’: {} ,
69 ’ errors ’: []
70 }
71 self . logger = logging . getLogger ( ’ etl_pipeline . metrics ’)
72
73 def start_stage ( self , stage_name ) :
74 self . metrics [ ’ stages ’ ][ stage_name ] = {
75 ’ start_time ’: time . time () ,
76 ’ status ’: ’ running ’
77 }
78 self . logger . info ( f " Starting stage : { stage_name } " )
79
80 def end_stage ( self , stage_name , status = ’ success ’ , rows_processed = None ) :
81 stage = self . metrics [ ’ stages ’ ]. get ( stage_name , {})
82 if stage :
83 stage [ ’ end_time ’] = time . time ()
84 stage [ ’ duration ’] = stage [ ’ end_time ’] - stage . get ( ’ start_time ’ ,
85 stage [ ’ end_time ’ ])
86 stage [ ’ status ’] = status
87 if rows_processed :
88 stage [ ’ rows_processed ’] = rows_processed
89
90 self . logger . info (
91 f " Completed stage : { stage_name } - "
92 f " Duration : { stage [ ’ duration ’]:.2 f }s , Status : { status } "
93 )
94
95 def add_error ( self , stage , error ) :
96 error_record = {
97 ’ stage ’: stage ,
98 ’ timestamp ’: time . time () ,
99 ’ error_type ’: type ( error ) . __name__ ,

15
 16

100 ’ error_message ’: str ( error )

101 }
102 self . metrics [ ’ errors ’ ]. append ( error_record )
103 self . logger . error ( f " Error in { stage }: { error } " , exc_info = True )
104
105 def complete ( self , status = ’ success ’) :
106 self . metrics [ ’ status ’] = status
107 self . metrics [ ’ end_time ’] = time . time ()
108 self . metrics [ ’ duration ’] = self . metrics [ ’ end_time ’] - \
109 self . metrics . get ( ’ start_time ’ , 0)
110
111 self . _save_metrics ()
112
113 def _save_metrics ( self ) :
114 import os
115
116 metrics_dir = ’ metrics ’
117 os . makedirs ( metrics_dir , exist_ok = True )
118
119 filename = f " { metrics_dir }/{ self . pipeline_name } _ " \
120 f " { self . metrics [ ’ run_id ’]}. json "
121
122 with open ( filename , ’w ’) as f :
123 json . dump ( self . metrics , f , indent =2)
124
125 self . logger . info ( f " Metrics saved to { filename } " )
126
127
128 class ETLPipeline :
129 " " " Production ETL Pipeline with monitoring " " "
130
131 def __init__ ( self , config ) :
132 self . config = config
133 self . logger = logging . getLogger ( ’ etl_pipeline ’)
134 self . metrics = None
135

136 @retry ( stop = sto p_ af te r_ at te mp t (3) ,

137 wait = wait_exponential ( multiplier =1 , min =2 , max =10) )
138 def extract ( self , source ) :
139 self . logger . info ( f " Extracting data from { source } " )
140
141 if source . startswith ( ’ http ’) :
142 response = requests . get ( source , timeout =30)
143 response . raise_for_status ()
144 data = response . json ()
145 else :
146 data = pd . read_csv ( source )
147

148 rows = len ( data ) if hasattr ( data , ’ __len__ ’) else 1

149 self . logger . info ( f " Extracted { rows } records " )
150 return data
151
152 def validate ( self , data ) :
153 self . logger . info ( " Validating data quality " )
154
155 if data is None or ( hasattr ( data , ’ __len__ ’) and len ( data ) == 0) :
156 raise ValueError ( " Empty dataset " )
157
158 if isinstance ( data , pd . DataFrame ) :
159 required_cols = self . config . get ( ’ required_columns ’ , [])
160 missing = [ c for c in required_cols if c not in data . columns ]
161
162 if missing :
163 raise ValueError ( f " Missing columns : { missing } " )
164
165 self . logger . info ( " Data validation passed " )
166 return True
167
168 def transform ( self , data ) :
169 self . logger . info ( " Starting transformation " )
170
171 if isinstance ( data , pd . DataFrame ) :
172 original_rows = len ( data )
173 data = data . drop_duplicates ()
174
175 data = data . fillna ({
176 ’ numeric_col ’: 0 ,
177 ’ string_col ’: ’ unknown ’
178 })
179
180 rows_after = len ( data )
181 self . logger . info (
182 f " Transformation complete : { original_rows } -> { rows_after } "
183 )
184
185 return data
186
187 def load ( self , data , destination ) :
188 self . logger . info ( f " Loading data to { destination } " )
189
190 if destination . endswith ( ’. csv ’) and isinstance ( data , pd . DataFrame ) :
191 data . to_csv ( destination , index = False )
192
193 rows = len ( data ) if hasattr ( data , ’ __len__ ’) else 1
194 self . logger . info ( f " Loaded { rows } records " )
195

16
 0.1. SCHEDULE LIBRARY FOR WORKFLOW AUTOMATION 17

196 return True

197
198 def run ( self ) :
199 self . metrics = PipelineMetrics ( self . config . get ( ’ name ’ , ’ etl_pipeline ’) )
200 self . metrics . metrics [ ’ start_time ’] = time . time ()
201

202 try :
203 self . metrics . start_stage ( ’ extract ’)
204 data = self . extract ( self . config [ ’ source ’ ])
205 self . metrics . end_stage ( ’ extract ’)
206
207 self . metrics . start_stage ( ’ validate ’)
208 self . validate ( data )
209 self . metrics . end_stage ( ’ validate ’)
210
211 self . metrics . start_stage ( ’ transform ’)
212 data = self . transform ( data )
213 self . metrics . end_stage ( ’ transform ’)
214
215 self . metrics . start_stage ( ’ load ’)
216 self . load ( data , self . config [ ’ destination ’ ])
217 self . metrics . end_stage ( ’ load ’)
218
219 self . metrics . complete ( ’ success ’)
220 return data
221
222 except Exception as e :
223 self . metrics . add_error ( ’ pipeline ’ , e )
224 self . metrics . complete ( ’ failed ’)
225 raise
226
227
228 if __name__ == " __main__ " :
229
230 config = {
231 ’ name ’: ’ daily_sales_etl ’ ,
232 ’ source ’: ’ sales_data . csv ’ ,
233 ’ destination ’: ’ processed_sales . csv ’ ,
234 ’ required_columns ’: [ ’ order_id ’ , ’ customer_id ’ , ’ amount ’ , ’ date ’] ,
235 ’ critical_columns ’: [ ’ order_id ’ , ’ amount ’] ,
236 ’ max_null_pct ’: 10 ,
237 ’ alert_on_failure ’: True
238 }
239
240 pipeline = ETLPipeline ( config )
241
242 try :
243 result = pipeline . run ()
244 print ( " Pipeline executed successfully " )
245 except Exception as e :
246 print ( f " Pipeline failed : { e } " )

6.5 Data Storage and Retrieval Strategies for ETL Pipelines

In ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) pipelines, data storage is not merely the final des-
tination of processed data. The strategies used for storing and retrieving data significantly influence pipeline performance,
scalability, reliability, and overall architecture. Selecting appropriate storage systems allows pipelines to efficiently process large
volumes of data while controlling costs and ensuring long-term accessibility.

1. Storage Paradigms for Pipelines

Modern data pipelines typically interact with multiple storage systems, each designed for a specific role in the data lifecycle.

a) Data Warehouses
A data warehouse is a centralized repository designed for analytical processing (OLAP). It stores structured and processed data
optimized for reporting, dashboards, and business intelligence.
In traditional ETL pipelines, the data warehouse acts as the final destination. Data is extracted from operational systems,
transformed in staging environments, and then loaded into the warehouse using structured schemas such as star or snowflake
schemas. In ELT pipelines, raw data is first loaded into the warehouse and transformations are performed using SQL within
the warehouse itself.
Key Characteristics

• Schema-on-Write: Data structure is enforced during the loading stage.

• Optimized for Analytical Queries: Uses techniques such as indexing, partitioning, and columnar storage formats.

• ACID Compliance: Ensures data consistency and reliability.

Common Platforms
Amazon Redshift, Google BigQuery, Snowflake, and Azure Synapse Analytics.

b) Data Lakes
A data lake is a large-scale storage repository designed to store raw data in its native format. Unlike warehouses, data lakes
support structured, semi-structured, and unstructured data.
In ELT architectures, data lakes often function as the initial landing zone where data from multiple sources is stored with
minimal transformation. This approach allows analysts and data scientists to process and analyze raw data later.
Key Characteristics

17
 18

• Schema-on-Read: Structure is applied only when data is analyzed.

• Cost-Effective Storage: Built on scalable object storage systems.

• Flexible Data Formats: Supports formats such as JSON, XML, images, and log files.

Common Platforms
AWS S3, Azure Data Lake Storage (ADLS), and Google Cloud Storage (GCS).

c) Data Lakehouses
A data lakehouse is a hybrid architecture that combines the flexibility of data lakes with the data management features of data
warehouses. This is achieved by adding a transactional metadata layer on top of data lake storage.
In pipeline architectures, the lakehouse can serve both as a landing zone for raw data and as an analytical storage system.
This reduces system complexity by eliminating the need for separate data lakes and warehouses.
Key Characteristics

• ACID Transactions: Supports reliable concurrent reads and writes.

• Schema Evolution: Allows schema changes without breaking existing datasets.

• Time Travel: Enables querying previous versions of data.

• Open File Formats: Data is typically stored in open formats such as Parquet.

Common Technologies
Delta Lake, Apache Iceberg, and Apache Hudi.

2. Key Data Storage Strategies

When designing the storage layer of a data pipeline, several strategies help improve performance, reliability, and cost efficiency.
Data Partitioning
Data partitioning divides large datasets into smaller logical segments based on column values such as date, region, or
category. This enables query engines to scan only relevant partitions instead of the entire dataset.

• Without partitioning: Queries must scan the full dataset.

• With partitioning: Queries access only the required partitions.

Data Replication
Data replication involves maintaining multiple copies of data across systems or geographic locations.
Common uses include:

• Disaster recovery and fault tolerance

• Improved query performance through geographically distributed copies

• Separation between operational databases (OLTP) and analytical systems (OLAP)

Data Caching
Caching stores intermediate or frequently accessed data in high-speed storage systems to reduce latency.
Typical use cases include:

• Storing intermediate ETL results

• Reusing computation results during pipeline reruns

• Accelerating dashboards and applications through cached aggregates

Hot vs. Cold Data Storage

Data tiering strategies classify data based on access frequency.

• Hot Storage: Frequently accessed data stored on high-performance storage.

• Warm Storage: Moderately accessed data stored on medium-cost storage.

• Cold/Archive Storage: Rarely accessed data stored on low-cost archival storage.

This strategy helps organizations manage storage costs while maintaining data availability.

3. Example: Partitioning Data for Efficient Retrieval (PySpark)

The following example demonstrates how large datasets can be written to storage in a partitioned format using PySpark.
Partitioning improves query performance by enabling engines to scan only relevant data segments.
1 from pyspark . sql import SparkSession
2
3 # Create Spark session
4 spark = SparkSession . builder \
5 . appName ( " PartitionExample " ) \
6 . getOrCreate ()
7
8 # Load dataset
9 df = spark . read . csv ( " sales_data . csv " , header = True , inferSchema = True )
10
11 # Convert date column to proper format
12 from pyspark . sql . functions import year , month
13
14 df = df . withColumn ( " year " , year ( " date " ) ) \
15 . withColumn ( " month " , month ( " date " ) )

18
 0.1. SCHEDULE LIBRARY FOR WORKFLOW AUTOMATION 19

16
17 # Write data partitioned by year and month
18 df . write \
19 . partitionBy ( " year " , " month " ) \
20 . mode ( " overwrite " ) \
21 . parquet ( " data_lake / sa les_pa rtitio ned " )
22
23 print ( " Data successfully written in partitioned format " )

Example 1: Partitioning Data for Efficient Retrieval (PySpark)

Partitioning data in a data lake enables efficient retrieval. Spark can automatically read only relevant partitions based on filter
conditions, which reduces I/O and improves query performance.
1 from pyspark . sql import SparkSession
2
3 # Initialize Spark Session
4 spark = SparkSession . builder . appName ( " P ar t i ti o n in g E xa m p le " ) . getOrCreate ()
5
6 # Assume ’ sales_df ’ is a Spark DataFrame loaded earlier in the pipeline
7 # sales_df = spark . read . parquet (" landing_zone / sales_data /")
8
9 # Write the data to a data lake partitioned by ’ year ’ and ’ month ’
10 output_path = " data_lake / sales / "
11
12 ( sales_df . write
13 . partitionBy ( " year " , " month " ) # Organizes data into year = YYYY / month = MM / folders
14 . mode ( " overwrite " ) # Overwrite existing data
15 . format ( " parquet " ) # Efficient columnar storage format
16 . save ( output_path ) )
17
18 print ( f " Data successfully written to { output_path } with year / month partitions . " )
19

20 # Retrieval : Read only sales for a specific month

21 jan_sales_df = spark . read . parquet ( output_path ) \
22 . where ( " year = ’2024 ’ AND month = ’01 ’ " ) # Only scans relevant partitions

Example 2: Config-Driven ETL Pipeline with Multiple Storage Destinations

A common pattern in modern pipelines is to support multiple storage destinations, such as a raw data lake (ELT) and a curated
warehouse (ETL). The following Python class demonstrates a simple configurable ETL pipeline.
1 import pandas as pd
2 import logging
3
4 logger = logging . getLogger ( __name__ )
5
6 class ETLPipeline :
7 " " " A simple ETL pipeline with configurable storage destinations . " " "
8
9 def __init__ ( self , config : dict ) :
10 self . config = config
11 self . data = None
12

13 def extract ( self ) -> pd . DataFrame :

14 logger . info ( f " Extracting data from { self . config [ ’ source_path ’]} " )
15 self . data = pd . read_csv ( self . config [ ’ source_path ’ ])
16 logger . info ( f " Extracted { len ( self . data ) } rows . " )
17 return self . data
18
19 def transform ( self ) -> pd . DataFrame :
20 logger . info ( " Starting transformation ... " )
21 if self . data is not None :
22 if ’ date ’ in self . data . columns :
23 self . data [ ’ date ’] = pd . to_datetime ( self . data [ ’ date ’ ])
24 logger . info ( " Transformation complete . " )
25 return self . data
26
27 def load ( self ) :
28 dest_type = self . config [ ’ destination_type ’]
29 dest_path = self . config [ ’ destination_path ’]
30 logger . info ( f " Loading data to { dest_type } at { dest_path } " )
31
32 if dest_type == " data_lake " :
33 self . data . to_parquet ( f " { dest_path }/ raw_data . parquet " , index = False )
34 logger . info ( " Data written to data lake in Parquet format . " )
35 elif dest_type == " data_warehouse " :
36 logger . info ( f " Loading transformed data into warehouse table : { dest_path } " )
37 # Example : self . data . to_sql ( ’ sales_table ’, connection , if_exists = ’ append ’)
38 else :
39 raise ValueError ( f " Unknown destination type : { dest_type } " )
40
41 logger . info ( " Load phase completed . " )
42
43 # --- Usage ---
44 elt_config = {
45 ’ source_path ’: ’ sales_data_raw . csv ’ ,
46 ’ destination_type ’: ’ data_lake ’ ,
47 ’ destination_path ’: ’ s3 :// my - data - lake - bronze / sales / ’
48 }
49
50 etl_config = {
51 ’ source_path ’: ’ sales_data_raw . csv ’ ,
52 ’ destination_type ’: ’ data_warehouse ’ ,
53 ’ destination_path ’: ’ analytics . sales_fact ’

19
 20

54 }
55
56 pipeline = ETLPipeline ( etl_config ) # or elt_config
57 pipeline . extract ()
58 pipeline . transform ()
59 pipeline . load ()

6.6 Automated Report Generation (Excel, HTML, PDF)

Introduction to Automated Reporting
Automated report generation is the process of programmatically creating structured documents that present data analysis
results without manual intervention. This ensures consistency, saves time, and enables regular distribution of insights.
Definition: Automated reporting involves pulling data from sources, processing it, formatting results into human-readable
formats (Excel, HTML, PDF), and distributing them automatically on a schedule or trigger.
Core Benefits:
Benefit Description
Time Efficiency Eliminates repetitive manual reporting work
Accuracy Reduces human error in data compilation
and calculations
Consistency Same format and quality every time reports
are generated
Timeliness Can be scheduled to run at optimal times or
triggered by events
Scalability Handle growing amounts of data and increas-
ing report frequency easily
Common Use Cases:
Use Case Application
Sales Reporting Pull daily sales data, generate revenue
trends, email PDF to managers
Website Analytics Fetch traffic data from APIs, show visitor
trends, publish weekly reports
Finance/Accounting Collect expense/revenue data, create balance
sheets, distribute automatically
IT Monitoring Track server uptime, CPU usage, generate
dashboards, notify on critical events

Key Python Libraries for Report Generation

Library Purpose Use Case
pandas Data handling and manipulation Core data processing for all re-
ports
matplotlib / seaborn Static visualizations Charts and graphs for
PDF/HTML reports
plotly Interactive visualizations Web-based dashboards and
HTML reports
fpdf / ReportLab PDF generation Printable reports, formal docu-
mentation
openpyxl / xlsxwriter Excel file creation Spreadsheet reports with formu-
las and formatting
Jinja2 HTML templating Dynamic web-based reports

Automated Report Generation Workflow

Extract → Transform → Visualize → Export → Deliver

(Gather data) (Process/analyze) (Charts/tables) (PDF/Excel/HTML) (Email/upload)

1. Excel Report Generation

Excel reports are ideal for stakeholders who need to perform additional analysis or work with raw numbers.
Key Concepts:

• DataFrames as foundation: pandas DataFrames provide the structure for tabular data.

• Multiple sheets: Organize related information across worksheet tabs.

• Formatting: Apply styles, colors, and fonts for readability.

• Formulas: Embed Excel formulas for dynamic calculations.

1 import pandas as pd
2 from datetime import datetime
3
4 def g e n e r a t e _ e x c e l _ r e p o r t ( data_path , output_path ) :
5 """
6 Generate formatted Excel report with multiple sheets
7 """
8 # Extract data
9 data = pd . read_csv ( data_path )
10
11 # Transform - create summary statistics
12 summary = pd . DataFrame ({
13 ’ Metric ’: [ ’ Total ’ , ’ Average ’ , ’ Maximum ’ , ’ Minimum ’] ,
14 ’ Value ’: [

20
 0.1. SCHEDULE LIBRARY FOR WORKFLOW AUTOMATION 21

15 data [ ’ amount ’ ]. sum () ,

16 data [ ’ amount ’ ]. mean () ,
17 data [ ’ amount ’ ]. max () ,
18 data [ ’ amount ’ ]. min ()
19 ]
20 })
21
22 # Export to Excel with multiple sheets
23 with pd . ExcelWriter ( output_path , engine = ’ openpyxl ’) as writer :
24 data . to_excel ( writer , sheet_name = ’ Raw Data ’ , index = False )
25 summary . to_excel ( writer , sheet_name = ’ Summary ’ , index = False )
26

27 # Add metadata
28 workbook = writer . book
29 summary_sheet = writer . sheets [ ’ Summary ’]
30 summary_sheet [ ’ A1 ’] = f " Report Generated : { datetime . now () } "

2. HTML Report Generation

HTML reports enable web-based viewing with interactive elements and are easily shareable via email or web servers.
Key Concepts:

• Templating: Use Jinja2 to separate content from presentation.

• Interactivity: Embed JavaScript libraries like Plotly for interactive charts.

• Responsive design: Create reports that work on different devices.

• Linking: Connect related sections with hyperlinks.

HTML Template Example ([Link]):

1 <! DOCTYPE html >
2 < html >
3 < head >
4 < title > {{ report_title }} </ title >
5 < style >
6 body { font - family : Arial , sans - serif ; margin : 40 px ; }
7 h1 { color : #333; }
8 table { border - collapse : collapse ; width : 100%; }
9 th , td { border : 1 px solid # ddd ; padding : 8 px ; text - align : left ; }
10 th { background - color : #4 CAF50 ; color : white ; }
11 </ style >
12 </ head >
13 < body >
14 < h1 > {{ report_title }} </ h1 >
15 <p > Generated : {{ generation_date }} </ p >
16
17 < h2 > Summary Statistics </ h2 >
18 < table >
19 < tr >
20 < th > Metric </ th >
21 < th > Value </ th >
22 </ tr >
23 {% for item in summary_data %}
24 < tr >
25 < td > {{ item . Metric }} </ td >
26 < td > {{ item . Value }} </ td >
27 </ tr >
28 {% endfor %}
29 </ table >
30 </ body >
31 </ html >

6.6 Automated Report Generation (Excel, HTML, PDF)

3. PDF Report Generation
PDF reports are suitable for formal documentation, printing, and sharing with external stakeholders.
Key Concepts for PDF Generation:

• Page layout: Control margins, page breaks, and orientation

• Font management: Use different fonts for headings and body text

• Image embedding: Include charts and graphs as images

• Headers and footers: Add consistent page numbering and document information

1 from fpdf import FPDF

2

3 pdf = FPDF ()
4 pdf . add_page ()
5
6 # Title
7 pdf . set_font ( ’ Arial ’ , ’B ’ , 16)
8 pdf . cell (0 , 10 , " Daily Sales Report " , ln = True , align = ’C ’)
9 pdf . cell (0 , 10 , f " Date : { datetime . now () . strftime ( ’%Y -% m -% d ’) } " , ln = True , align = ’C ’)
10 pdf . ln (10)
11
12 # KPIs
13 pdf . set_font ( ’ Arial ’ , ’B ’ , 12)
14 pdf . cell (0 , 10 , " Key Performance Indicators " , ln = True )

21
 22

15 pdf . set_font ( ’ Arial ’ , ’ ’ , 10)

16
17 for kpi , value in kpis . items () :
18 pdf . cell (0 , 8 , f " { kpi }: { value : ,.2 f } " , ln = True )
19
20 pdf . output ( " dai ly _s al es _r ep or t . pdf " )

Best Practices for Automated Reporting

• Parameterize Everything: Make report elements configurable (dates, thresholds, formats)

• Include Metadata: Always add generation timestamp, data sources, and version information

• Handle Missing Data Gracefully: Implement checks for empty datasets

• Log Generation Process: Track when reports are generated and any errors encountered

• Validate Outputs: Verify file sizes and content before distribution

• Archive Previous Versions: Maintain history for audit purposes

6.7 Case Study: End-to-End Automated Analytics Pipeline

Overview
This case study synthesizes concepts from Chapter 6 into a complete, real-world example of an automated analytics pipeline.
It demonstrates how data engineering components work together in production.
Business Scenario:

• Company: E-commerce Retailer ”ShopSmart”

• Problem: Manual reporting on daily sales, inventory, and customer metrics takes 3 hours each morning

• Goal: Fully automate data collection, processing, analysis, and reporting with zero manual intervention

Pipeline Architecture
The pipeline follows a modern ELT (Extract, Load, Transform) pattern with multiple storage layers:

[Source Systems] → [Landing Zone] → [Processing] → [Serving Layer] → [Reports]

• Source Systems: Database, API, CSV

• Landing Zone: Data Lake (raw storage)

• Processing: Spark or Python ETL jobs

• Serving Layer: Data Warehouse, Excel, PDF

Pipeline Components
Component 1: Data Extraction (Scheduled) Automatically pull data from multiple source systems.
1 class DataExtractor :
2 def ext ra ct _s al es _d at a ( self ) :
3 df = pd . read_csv ( self . config [ ’ sales_source ’ ])
4 return df
5

6 def e x t r a c t _ i n v e n t o r y _ d a t a ( self ) :
7 response = requests . get ( self . config [ ’ inventory_api ’ ])
8 data = response . json ()
9 return pd . DataFrame ( data )
10
11 def e x t r a c t _ c u s t o m e r _ d a t a ( self ) :
12 return pd . read_csv ( self . config [ ’ customer_source ’ ])

Component 2: Data Lake Storage (Landing Zone) Store raw data in its original format for auditability and reprocessing.
1 class DataLake :
2 def store_raw_data ( self , data , source_name ) :
3 date_str = datetime . now () . strftime ( " %Y -% m -% d " )
4 partition_path = f " { self . base_path }/{ source_name }/ date ={ date_str } "
5 os . makedirs ( partition_path , exist_ok = True )
6 data . to_parquet ( f " { partition_path }/ raw_data . parquet " )
7 return f " { partition_path }/ raw_data . parquet "

Component 3: Data Transformation (Processing Layer) Clean, validate, and transform raw data into analytical
structures.
1 class DataTransformer :
2 def clean_sales_data ( self , raw_sales ) :
3 cleaned = raw_sales . drop_duplicates ( subset =[ ’ order_id ’ ])
4 cleaned [ ’ customer_id ’] = cleaned [ ’ customer_id ’ ]. fillna ( ’ UNKNOWN ’)
5 cleaned [ ’ amount ’] = cleaned [ ’ amount ’ ]. fillna (0) . astype ( float )
6 cleaned [ ’ order_date ’] = pd . to_datetime ( cleaned [ ’ order_date ’ ])
7 return cleaned
8
9 def calculate_kpis ( self , sales_df , inventory_df , customer_df ) :
10 kpis = {}
11 kpis [ ’ total_sales ’] = sales_df [ ’ amount ’ ]. sum ()
12 kpis [ ’ avg_order_value ’] = sales_df [ ’ amount ’ ]. mean ()

22
 0.1. SCHEDULE LIBRARY FOR WORKFLOW AUTOMATION 23

13 kpis [ ’ order_count ’] = len ( sales_df )

14 kpis [ ’ unique_customers ’] = sales_df [ ’ customer_id ’ ]. nunique ()
15 kpis [ ’ low_stock_items ’] = len ( inventory_df [ inventory_df [ ’ quantity ’] < 10])
16 return kpis

1 class ReportGenerator :
2 def c r e a t e _ e x c e l _ d a s h b o a r d ( self , sales_df , kpis ) :
3 with pd . ExcelWriter ( f " reports / daily_dashboard . xlsx " ) as writer :
4 pd . DataFrame ( list ( kpis . items () ) , columns =[ ’ KPI ’ , ’ Value ’ ]) . to_excel ( writer , sheet_name = ’
Dashboard ’)
5 sales_df . to_excel ( writer , sheet_name = ’ Sales Details ’)
6

7 def cr eate_p df_rep ort ( self , sales_df , kpis ) :

8 from fpdf import FPDF
9 pdf = FPDF ()
10 pdf . add_page ()
11 pdf . set_font ( ’ Arial ’ , ’B ’ , 16)
12 pdf . cell (0 , 10 , " Daily Sales Report " , ln = True , align = ’C ’)
13 for kpi , value in kpis . items () :
14 pdf . cell (0 , 8 , f " { kpi }: { value : ,.2 f } " , ln = True )
15 pdf . output ( " reports / daily_report . pdf " )

Component 5: Orchestration (Pipeline Controller) Coordinate extraction, storage, transformation, and report gener-
ation.
1 class An alytic sPipel ine :
2 def run ( self ) :
3 raw_sales = self . extractor . ex tra ct _s al es _d at a ()
4 raw_inventory = self . extractor . e x t r a c t _ i n v e n t o r y _ d a t a ()
5 raw_customers = self . extractor . e x t r a c t _ c u s t o m e r _ d a t a ()
6 self . data_lake . store_raw_data ( raw_sales , ’ sales ’)
7 cleaned_sales = self . transformer . clean_sales_data ( raw_sales )
8 kpis = self . transformer . calculate_kpis ( cleaned_sales , raw_inventory , raw_customers )
9 excel_report = self . reporter . c r e a t e _ e x c e l _ d a s h b o a r d ( cleaned_sales , kpis )
10 pdf_report = self . reporter . cr eate_p df_rep ort ( cleaned_sales , kpis )

Pipeline Monitoring and Alerting

Include metrics, logs, and error tracking to ensure reliability.
1 class PipelineMonitor :
2 def g e n e r a t e _ h e a l t h _ r e p o r t ( self ) :
3 report = {
4 ’ total_executions ’: len ( self . metrics [ ’ execution_time ’ ]) ,
5 ’ avg _e xe cu ti on _t im e ’: np . mean ([ m [ ’ duration ’] for m in self . metrics [ ’ execution_time ’ ]]) ,
6 ’ error_rate ’: self . metrics [ ’ error_count ’] / max ( len ( self . metrics [ ’ execution_time ’ ]) , 1)
7 }
8 return report

Key Lessons and Best Practices

• Separation of Concerns: Each component has a single responsibility

• Idempotency: Pipeline can be safely re-run

• Observability: Logging and metrics at every step

• Error Handling: Graceful failure with detailed logs

• Configuration-Driven: Externalize all settings

• Data Lineage: Preserve raw data for audit trail

• Automated Validation: Prevent bad data from propagating

Performance Optimization Techniques

Technique Application
Vectorized Operations Use pandas/numpy instead of loops
Partitioning Organize data by date for faster queries
Caching Store intermediate results for frequent re-
ports
Parallel Processing Extract from multiple sources concurrently
Incremental Loading Process only new/changed data

23