ADVANCED PYTHON PROGRAMMING

CSI3007

[Link]-21842
Assistant Professor Sr
SCOPE-VIT
 Course Objectives:

1. To be able to apply advanced python programming concepts for

industry standard problems.
2. To perform advanced Data Preprocessing tasks like Data Merging
and Munging
3. To be able to develop powerful Web-Apps using Python

Sivaranjani.A, AP/SCOPE VIT

 Course Outcomes
1. Understand the nuances of Data Structures
2. Derive an understanding of a classes and objects and their
potential
3. Gain knowledge of multithreading concepts and implementing the
same
4. Appreciate the difference between different data processing
techniques
5. Learn to apply Python features for Data Science
6. Get an insight into Metrics Analysis
7. Develop web-apps and build models for IoT
Sivaranjani.A, AP/SCOPE VIT
 Module:1
Data Structures 4 Hours
• Problem solving using Python Data Structures : LIST, DICT, TUPLES and
SET- Functions and Exceptions Lamda Functions and Parallel
processing MAPS Filtering - Itertools Generators

Sivaranjani.A, AP/SCOPE VIT

 Module:2
Classes and Objects 4 Hours
• Classes as User Defined Data Type ,Objects as Instances of Classes, Creating
Class and Objects, Creating Objects By Passing Values, Variables & Methods
in a Class Data Abstraction, Data Hiding, Encapsulation, Modularity,
Inheritance, Polymorphism

Sivaranjani.A, AP/SCOPE VIT

 Module:3
Multithreading in Python 4 Hours
• Python Multithreading and Multiprocessing Multithreading and
multiprocessing Basics Threading module and example Pytho
multithreading - Multithreaded Priority Queue

Sivaranjani.A, AP/SCOPE VIT

 Module:4
Data Processing 4Hours
• Handling CSV, Excel and JSON data - Creating NumPy arrays, Indexing and
slicing in NumPy, Downloading and parsing data, Creating
multidimensional arrays, NumPy Data types, Array Attribute, Indexing and
Slicing, Creating array views copies, Manipulating array shapes I/O-
MATPLOT LIB

Sivaranjani.A, AP/SCOPE VIT

 Module:5
Data Science Perspectives 5 Hours
• Using multilevel series, Series and Data Frames, Grouping, aggregating,
Merge DataFrames, Generate summary tables, Group data into logical
pieces, Manipulate dates, Creating metrics for analysis

Sivaranjani.A, AP/SCOPE VIT

 Module:6
Data Handling Techniques 3 Hours
• Data wrangling ,Merging and joining,- Loan Prediction Problem, Data
Mugging using Pandas

Sivaranjani.A, AP/SCOPE VIT

 Module:7
Web Applications 4 Hours
• Web Applications With Python Django / Flask / Web2Py Database
Programming NoSQL databases - Embedded Application using IOT Devices
- Building a Predictive Model for IOT and Web programming

Sivaranjani.A, AP/SCOPE VIT

 Multithreading & Multiprocessing
● Multithreading refers to the ability of a processor to execute multiple threads concurrently.
● Multiprocessing refers to the ability of a system to run multiple processors in parallel,
where each processor can run one or more threads.

Sivaranjani.A, AP/SCOPE VIT

Multithreading & Multiprocessing

Multithreading Multiprocessing

Sivaranjani.A, AP/SCOPE VIT

 Multithreading & Multiprocessing
●

Sivaranjani.A, AP/SCOPE VIT

 Multithreading & Multiprocessing
● Multithreading and multiprocessing are two ways to achieve multitasking in
Python.
● Multitasking is useful for running functions and code concurrently or in parallel.
● There is a difference between concurrency and parallelism.
● Parallelism allows multiple tasks to execute at the same time, whereas
concurrency make progress on multiple task and execute one at a time in an
interleaving manner.
● Python supports both through built-in modules.

Sivaranjani.A, AP/SCOPE VIT

 Multithreading & Multiprocessing
● A process is an independent instance executed in a processor core.

● Threads are components of a process and run concurrently (inside that process).
● Processes do not share the same memory space, while threads do.
● Threads are lighter and cause less overhead. Also, because they share the same memory
inside a process, it is easier, faster, and safer to share data.
● Multithreading best for I/O-Bound Tasks and Multiprocessing best for CPU-bound tasks
● True parallelism can ONLY be achieved using multiprocessing. That is because only
one thread can be executed at a given time inside a process time-space. This is assured by
Python’s global interpreter lock (GIL).
● Processes execution is scheduled by the operating system, while threads are scheduled
by the GIL.
● Multithreading implements concurrency, multiprocessing implements parallelism.
Processes run on separate processing nodes.
● EX-MT: A web browser downloading a file while you type in another tab.
● EX-MP: Training a machine learning model using 8 CPU cores in parallel.
Sivaranjani.A, AP/SCOPE VIT
 What is Thread ?
● A thread is the smallest unit of execution within a process.
● Python allows creating and managing threads using the built-in
_thread, and threading, module.
● Threads run concurrently within the same program and share the same memory
space.
● Useful especially for I/O-bound tasks like file handling and network
operations, where waiting time can be utilized efficiently.

Sivaranjani.A, AP/SCOPE VIT

Life cycle of Thread

Sivaranjani.A, AP/SCOPE VIT

 Life cycle of Thread
A thread in Python (or generally in any programming language) goes through various states during its life
cycle:
1. New (Created)
○ Thread object is created but not started yet.
○ Example: t1 = [Link](target=func)
2. Runnable (Ready)
○ Thread is ready to run, waiting for CPU scheduling.
○ Happens after calling [Link](), but before execution actually begins.
3. Running
○ Thread is executing its run() method.
○ Only one Python thread can execute bytecode at a time (GIL limitation).
4. Waiting / Blocked
○ Thread is waiting for a resource, like I/O operation, [Link](), or join() on another thread.
5. Terminated (Dead)
○ Thread completes execution or is stopped.
○ Thread cannot be restarted once terminated ([Link]() again will raise an error).
Sivaranjani.A, AP/SCOPE VIT
 _Thread Module
_thread Module (Low-Level Threading):
● Old, low-level module (introduced in Python 1.5).

● Provides basic primitives for working with threads.

● Functions like start_new_thread(), allocate_lock(), exit().

● Very minimal control → no direct support for managing thread lifecycle, joining, or
object-oriented usage.

● Mostly not recommended for new code (kept for backward compatibility).

Sivaranjani.A, AP/SCOPE VIT

 _Thread Module
import _thread
import time
def worker(name):
for i in range(3):
print(f"Thread {name} running {i}")
[Link](1)
# Start two threads
_thread.start_new_thread(worker, ("A",))
_thread.start_new_thread(worker, ("B",))
[Link](5) # wait for threads to finish

Sivaranjani.A, AP/SCOPE VIT

 Threading module
The threading module in Python provides a high-level interface for creating and managing
threads.
It builds on top of the lower-level _thread module but is easier and safer to use.
It allows you to:
● Create threads easily

● Control thread execution

● Synchronize threads

● Share resources safely

Sivaranjani.A, AP/SCOPE VIT

import threading
Threading module
import time

def worker():

print("Thread started")

[Link](2) # Simulate some work

print("Thread finished")

t = [Link](target=worker)

[Link]()

[Link]() # Main program waits here until thread completes

print("Main thread continues")

Sivaranjani.A, AP/SCOPE VIT
 Threading module
import threading
# Start threads
import time
[Link]()
def worker(name):
[Link]()
for i in range(3):
print(f"Thread {name} running {i}")
# Wait until both finish
[Link](1)
[Link]()
# Create threads
[Link]()
t1 = [Link](target=worker, args=("A",))
t2 = [Link](target=worker, args=("B",))

Sivaranjani.A, AP/SCOPE VIT

 Threading module
import threading
def add(x, y):
print(x + y)
t = [Link](target=add, args=(5, 3))
[Link]()
[Link]()

Sivaranjani.A, AP/SCOPE VIT

 Threading module-START ()
import threading

import time
OUTPUT:
from time import ctime Creating thread 0 at Fri Sep 18 16:24:25 2020
def myThread(num): Starting thread 0 at Fri Sep 18 16:24:25 2020
Thread 0: started at Fri Sep 18 16:24:25 2020
print("Thread %d: started at %s" % (num, ctime([Link]())))
Creating thread 1 at Fri Sep 18 16:24:25 2020
[Link](2) Starting thread 1 at Fri Sep 18 16:24:25 2020
print("Thread %d: finished at %s" % (num, ctime([Link]())))
Thread 1: started at Fri Sep 18 16:24:25 2020
Creating thread 2 at Fri Sep 18 16:24:25 2020
for i in range(0, 3): Starting thread 2 at Fri Sep 18 16:24:25 2020
print("Creating thread %d at %s" % (i, ctime([Link]()))) Thread 2: started at Fri Sep 18 16:24:25 2020
Thread 0: finished at Fri Sep 18 16:24:27 2020
thread = [Link](target=myThread, args=(i,))
Thread 2: finished at Fri Sep 18 16:24:27 2020
print("Starting thread %d at %s" % (i, ctime([Link]()))) Thread 1: finished at Fri Sep 18 16:24:27 2020
[Link]()

Sivaranjani.A, AP/SCOPE VIT

 Threading module-JOIN ()
import threading

import time
Creating thread 0 at Fri Sep 18 16:37:26 2020
from time import ctime Starting thread 0 at Fri Sep 18 16:37:26 2020
Thread 0: started at Fri Sep 18 16:37:26 2020
Thread 0: finished at Fri Sep 18 16:37:28 2020
def myThread(num):
Creating thread 1 at Fri Sep 18 16:37:28 2020
print("Thread %d: started at %s" % (num, ctime([Link]()))) Starting thread 1 at Fri Sep 18 16:37:28 2020
[Link](2)
Thread 1: started at Fri Sep 18 16:37:28 2020
Thread 1: finished at Fri Sep 18 16:37:30 2020
print("Thread %d: finished at %s" % (num, ctime([Link]()))) Creating thread 2 at Fri Sep 18 16:37:30 2020
Starting thread 2 at Fri Sep 18 16:37:30 2020
Thread 2: started at Fri Sep 18 16:37:30 2020
for i in range(0, 3):
Thread 2: finished at Fri Sep 18 16:37:32 2020
print("Creating thread %d at %s" % (i, ctime([Link]())))

thread = [Link](target=myThread, args=(i,))

print("Starting thread %d at %s" % (i, ctime([Link]())))

[Link]()

[Link]()

Sivaranjani.A, AP/SCOPE VIT

 Threading module-isalive()
import threading

import time
Creating thread 0 at Fri Sep 18 16:51:42 2020
from time import ctime Starting thread 0 at Fri Sep 18 16:51:42 2020
Thread 0: started at Fri Sep 18 16:51:42 2020
Thread alive: True
def myThread(num):
Creating thread 1 at Fri Sep 18 16:51:43 2020
print("Thread %d: started at %s" % (num, ctime([Link]()))) Starting thread 1 at Fri Sep 18 16:51:43 2020
[Link](2)
Thread 1: started at Fri Sep 18 16:51:43 2020
Thread 0: finished at Fri Sep 18 16:51:44 2020
print("Thread %d: finished at %s" % (num, ctime([Link]()))) Thread alive: True
Creating thread 2 at Fri Sep 18 16:51:44 2020
Starting thread 2 at Fri Sep 18 16:51:44 2020
for i in range(0, 3):
Thread 2: started at Fri Sep 18 16:51:44 2020
print("Creating thread %d at %s" % (i, ctime([Link]()))) Thread 1: finished at Fri Sep 18 16:51:45 2020
thread = [Link](target=myThread, args=(i,)) Thread alive: True
Thread 2: finished at Fri Sep 18 16:51:46 2020
print("Starting thread %d at %s" % (i, ctime([Link]())))

[Link]()

[Link](1)

print("Thread alive: ",thread.is_alive())

Sivaranjani.A, AP/SCOPE VIT

Threading module-other methods
● threading.active_Count(): It returns the number of active threads.
● threading.current_Thread(): It returns the current thread being executed.
● [Link](): It returns a list of active threads.

Sivaranjani.A, AP/SCOPE VIT

 What will happen?
● Threads share the same memory space.
● Multiple threads may access or modify data at the same time.
● Two or more threads wait forever because each is holding a resource the
other needs.
● A thread never gets CPU time or resources because others always take
priority.
● Too many threads trying to use limited resources
● Too many threads → frequent switching between threads.

Sivaranjani.A, AP/SCOPE VIT

 Threading issues and Challenges
● A race condition occurs when multiple threads try to access and modify the same
shared data at the same time, which often leads to unpredictable and incorrect results.
● A deadlock happens when two or more threads are waiting for each other’s resources
indefinitely, causing the program to freeze.
● Starvation takes place when some threads are repeatedly denied CPU or resource access
because higher-priority threads keep getting served first.
● Resource contention arises when several threads compete for limited system resources
such as database connections, files, or network sockets.
● Too many threads lead to context switching overhead, where the CPU spends more time
switching between threads than doing actual work, reducing efficiency.
● Debugging multithreaded programs is difficult because thread execution is
non-deterministic, meaning the same program may behave differently each time it runs.

Sivaranjani.A, AP/SCOPE VIT

 To over come these challenges
● A lock (mutex) ensures that only one thread can access a shared resource at a time,
preventing race conditions.
● Semaphores allow a fixed number of threads to access a shared resource, which is useful
for managing limited resources.
● Using lock ordering or timeout locks can prevent deadlocks by making sure threads
acquire resources in a consistent order or release them after a time limit.
● Thread pools reduce overhead by limiting the number of active threads and reusing
existing ones instead of creating new threads each time.
● Fair scheduling mechanisms make sure that every thread eventually gets a chance to run,
preventing starvation.
● Logging and high-level concurrency libraries such as threading and [Link]
make debugging easier and help in writing safer concurrent code.

Sivaranjani.A, AP/SCOPE VIT

 Multithread Priority Queue
● Even after applying synchronization techniques like locks and
semaphores, we still face the question: which task should run first when
multiple tasks are waiting.
● A Multithreaded Priority Queue in Python is a data structure that
enables efficient, thread-safe management of prioritized tasks in
concurrent or parallel programs.
● A priority queue is a type of queue where each element has a priority;
elements with a higher priority are processed first.
● Python's [Link] class offers built-in thread safety using
internal locks, making it suitable for multithreaded environments.

Sivaranjani.A, AP/SCOPE VIT

 Multithread Priority Queue

Basic Operations and Methods

● put(item): Adds an item to the queue with associated priority (tuples like
(priority, data) are commonly used).
● get(): Removes and returns the item with the highest priority (lowest
priority number).
● qsize(): Returns the number of items in the queue.
● empty(): Returns True if the queue is empty; otherwise, False.
● full(): Returns True if the queue reaches its maximum size.

Sivaranjani.A, AP/SCOPE VIT

 Multithread Priority Queue
Thread Safety
● The PriorityQueue class uses locks to prevent race conditions, so multiple threads
can safely add and remove items concurrently.
● It supports multi-producer (several threads adding items) and multi-consumer
(several threads removing items) scenarios.
Structure of Priority Queue Items
● Common format: a tuple (priority, data), where lower integer values of priority
correspond to higher task priority.
● Custom classes can be used for more complex data structures, but standard usage
revolves around tuples.

Sivaranjani.A, AP/SCOPE VIT

 Multithread Priority Queue
import threading
import queue
import time
# Create a thread-safe priority queue
pq = [Link]()
# Worker function
def worker():
while not [Link]():
priority, task = [Link]() # Get task with highest priority (lowest number)
print(f"{threading.current_thread().name} is working on: {task} (priority {priority})")
[Link](1) # Simulate task execution
pq.task_done()

Sivaranjani.A, AP/SCOPE VIT

 Multithread Priority Queue
# Add tasks to the priority queue
# Lower number = higher priority
[Link]((2, "Task A"))
[Link]((1, "Task B"))
[Link]((3, "Task C"))
[Link]((1, "Task D"))
# Create multiple threads
threads = [ ]
for i in range(2):
t = [Link](target=worker, name=f"Thread-{i+1}")
[Link]( )
[Link](t)

Sivaranjani.A, AP/SCOPE VIT

 Multithread Priority Queue
# Wait for all threads to finish
for t in threads: Output: Thread-1 is working on: Task B (priority
[Link]() 1)Thread-2 is working on: Task D (priority 1)

print("All tasks completed.") Thread-2 is working on: Task A (priority 2)Thread-1 is

working on: Task C (priority 3)

All tasks completed.

Sivaranjani.A, AP/SCOPE VIT

 Multithread Priority Queue- Producer consumer problem

Definition:
The Producer-Consumer problem is a classic multithreading and synchronization
problem, where two types of threads interact with a shared resource (buffer):
● Producer → produces data/items and puts them into the buffer.
● Consumer → takes data/items from the buffer and processes them.
Constraints:
● The buffer has a limited size.
○ If full → producer must wait.
○ If empty → consumer must wait.
Avoid race conditions and ensure all items are produced and consumed correctly.

Sivaranjani.A, AP/SCOPE VIT

 Multithread Priority Queue- Producer consumer problem
How It Works With Threads
1. Producer thread creates an item → puts it into the buffer.
2. Consumer thread takes an item → processes it.
3. Threads use synchronization mechanisms like:
○ Locks
○ Semaphores
○ Thread-safe queues ([Link] or [Link])
Key Concept: Prevents race conditions and ensures proper coordination.
Difference with a normal queue:
● In a normal queue ([Link]), consumers process items in the order they were
produced (FIFO).
● In a priority queue ([Link]), each item has a priority, and consumers
always take the highest priority item (lowest number first).

Sivaranjani.A, AP/SCOPE VIT

 Multithread Priority Queue- Producer consumer problem

import threading
import queue
import time
import random

buffer = [Link](maxsize=5) # shared priority queue

def producer():
for i in range(10):
priority = [Link](1, 10)
item = f"item-{i}"
[Link]((priority, item)) # produce item with priority
print(f"Produced {item} with priority {priority}")
[Link]([Link]())

def consumer():
for i in range(10):
priority, item = [Link]() # consumes highest priority item (lowest number)
print(f"Consumed {item} with priority {priority}")
buffer.task_done()
[Link]([Link]())

t1 = [Link](target=producer)
t2 = [Link](target=consumer)

[Link]()
[Link]()

[Link]()
[Link]()

print("All items produced and consumed.")

Sivaranjani.A, AP/SCOPE VIT