I.M.C.
A Semester – 5
MASTERING DATA SCIENCE A
GUIDE TO PYTHON KEY LIBRARY
Unit – 2
1|P a g e
Unit -2: Genrative AI
Contents
Broadcasting ......................................................................................................................................................................... 6
Aggregation Functions.......................................................................................................................................................... 7
NumPy Array Sorting and Comparison ................................................................................................................................. 9
Random Module in NumPy ................................................................................................................................................. 11
Data Pre-processing ............................................................................................................................................................ 13
Pandas Integration with Mysql ........................................................................................................................................... 21
Web data collection using beautifulshop, requestes, scrappy ............................................................................................ 23
2|P a g e
• NumPy stands for Numerical Python.
• It is a powerful library used for numerical operations, especially
with arrays.
• It is the foundation for most data preprocessing in AI/ML.
Why use NumPy in AI/ML?
• Fast and efficient with large datasets.
• Easily integrates with TensorFlow and other AI tools.
• Supports matrix operations, reshaping, and more.
• 1D Array → a single row (like a line of items):
• [Link]([1, 2, 3]) → shape = (3,)
• 2D Array → rows and columns (like a table):
• [Link]([[1, 2], [3, 4]]) → shape = (2, 2)
• 3D Array → multiple 2D arrays stacked (like a cube):
• shape = (layers, rows, columns)
Feature Description
ndarray A fast and efficient multidimensional array object
Broadcasting Perform operations on arrays of different shapes
Eliminate loops by operating on arrays as a
Vectorization
whole
3|P a g e
Mathematical
Includes trigonometry, statistics, algebra, etc.
functions
Indexing/Slicing Powerful tools for manipulating arrays
Integration Works seamlessly with C/C++ and other libraries
• 1. ndarray (N-dimensional array)
• The heart of NumPy is the ndarray class.
• It is a homogeneous(All elements are of the same type, like all
integers or all floats) multidimensional array.
Ex:
import numpy as np
arr_2d = [Link]([[1, 2, 3], [4, 5, 6]])
print(arr_2d)
• Array Creation function
Function Description
[Link]() Creates an array from a list/tuple
[Link](shape) Creates array filled with zeros
[Link](shape) Creates array filled with ones
4|P a g e
[Link](start, stop, step) Creates array with evenly spaced values
[Link](start, stop, Creates array with specified number of
num) values between start and stop
import numpy as np
print("[Link]():", [Link]([1, 2, 3]))
print("[Link]():\n", [Link]((2, 3)))
print("[Link]():\n", [Link]((2, 2)))
print("[Link]():\n", [Link](3))
print("[Link]():", [Link](1, 10, 2))
print("[Link]():", [Link](0, 1, 5))
• Reshaping Arrays(Changing the shape of an array without
changing its data):
arr = [Link](6) # [0 1 2 3 4 5]
[Link](2, 3) # [[0 1 2], [3 4 5]]
• Indexing and Slicing:
arr = [Link]([[1, 2, 3], [4, 5, 6]])
arr[0, 1] # 2
arr[:, 1] # [2 5] => All rows, second column
• Vectorized Operations; Vectorized operations mean performing
arithmetic operations directly on arrays (like addition, subtraction,
5|P a g e
multiplication) without using loops. It is faster and simpler than
traditional loops.
a = [Link]([1, 2, 3])
b = [Link]([4, 5, 6])
print(a + b ) # [5 7 9]
Print(a * 2) # [2 4 6]
Print(a **2)#power/squre
Broadcasting
• Allows operations between arrays of different shapes:
a = [Link]([[1], [2], [3]]) # Shape: (3,1)
b = [Link]([10, 20, 30]) # Shape: (3,)
result = a + b
# [[11 21 31], [12 22 32], [13 23 33]]
6|P a g e
Aggregation Functions
Function Description
[Link]() Sum of elements
[Link]() Mean (average)
[Link](), [Link]() Minimum / Maximum
[Link](), [Link]() Index of min/max
[Link](arr,axis=1) Give sum of all columns
[Link](arr,axis=0) Give sum of all the rows
Transpose the array(changes row
arr.T
into column)
import numpy as np
# Create a 2D array
arr = [Link]([[10, 20, 30],[40, 50, 60]])
print("Original Array:\n", arr)
# Sum of all elements
print("\nSum:", [Link](arr))
# mean (average) of elements
print("Mean:", [Link](arr))
#Minimum and Maximum
7|P a g e
print("Minimum:", [Link](arr))
print("Maximum:", [Link](arr))
#Index of Min/Max
print("Index of Min :", [Link](arr))
print("Index of Max :", [Link](arr))
#Sum along axis
print("\nSum along rows (axis=1):", [Link](arr, axis=1))
print("Sum along columns (axis=0):", [Link](arr, axis=0))
#Transpose the arr
Print(arr.T)
8|P a g e
NumPy Array Sorting and Comparison
Function Description
[Link](arr) Sorts elements
[Link](arr) Returns unique elements
[Link](condition) Returns indices where condition is True
[Link]() Boolean checks
import numpy as np
# Create an unsorted array with some duplicate elements
arr = [Link]([40, 10, 30, 20, 10, 50])
print("Original Array:\n", arr)
# Sorting the array
sorted_arr = [Link](arr)
print("\nSorted Array:\n", sorted_arr)
#Indices that would sort the array
indices = [Link](arr)
print("Indices to sort the array (argsort):\n", indices)
# Unique elements
unique_values = [Link](arr)
print("Unique Elements:\n", unique_values)
9|P a g e
# Conditional Comparison: Elements > 25
greater_than_25 = arr[arr > 25]
print("Elements greater than 25:\n", greater_than_25)
#Where condition is True
where_result = [Link](arr == 10)
print("Indices where value == 10:\n", where_result)
# Check if any elements > 45
print("Any element > 45?:", [Link](arr > 45)
#Check if all elements > 5
print("All elements > 5?:", [Link](arr > 5))
# Boolean mask: Multiple conditions (10 < arr < 40)
mask = (arr > 10) & (arr < 40)
print("Elements between 10 and 40:\n", arr[mask])
10 | P a g e
Random Module in NumPy
Function Description
random() Random floats in [0, 1)
integers(low, high) Random integers in [low, high)
choice(a, size) Randomly pick elements from an array
shuffle(x) Shuffle the sequence in place
permutation(x) Return a shuffled version of input
1. Random Numbers(gives a random number between 0 and 1)
import numpy as np
print([Link]())
[Link] Integers
Import numpy as np
print([Link](1, 10)) # one number between 1 and 9
• 3. Random Choice: Pick a random item from a list or array:
Import numpy as np
print([Link]([3, 5, 7, 9])) # randomly picks one
[Link]() – Shuffle an existing array (changes the original)
arr = [Link]([1, 2, 3, 4, 5])
[Link](arr)
11 | P a g e
print(arr) # Randomly shuffled array
5. Random Numbers in Arrays
import numpy as np
print([Link](3)) # array with 3 random numbers
print([Link](2, 3)) # 2 rows, 3 columns of random numbers
print([Link](1, 100, size=(3, 2))) # 3x2 array of random
integers
12 | P a g e
Data Pre-processing
• Data preprocessing is the process of cleaning, organizing, and
transforming raw data into a form that can be used effectively by AI
or machine learning models.
• Think of it like this:
• If you're building a house, you need clean and strong bricks.
• Similarly, AI models need clean and well-structured data to learn
and give good results.
Raw data often contains:
• Missing values
• Duplicate records
• Wrong formats
• Extra symbols or noise
• If not cleaned, the model may give wrong or poor results.
• Imagine building a language translator using a dataset of English
and Hindi sentences:
• Some sentences might be incomplete.
• Some may contain strange characters.
• Some may have spelling errors.
• You must fix all this before training the AI model.
What Tasks are Done in Preprocessing?
Task Purpose
• Remove null values • Missing info is useless for the model
• Remove duplicates • Avoid repeating the same data
• Fix text formatting • Make all text lowercase, remove symbols
13 | P a g e
• Break long text into smaller units
• Tokenization
(words)
• Normalize numbers • Keep numbers within a range
• Label encoding • Convert words to numbers
• What is Pandas?
• Pandas is a Python library used for working with data sets.
• It has functions for analyzing, cleaning, exploring, and manipulating
data.
• The name "Pandas" has a reference to both "Panel Data", and "Python
Data Analysis" and was created by Wes McKinney in 2008.
Pandas Series
• A Pandas Series is like a column in a table.
• It is a one-dimensional array holding data of any type.
Ex:
import pandas as pd
a = [1, 7, 2]
myvar = [Link](a)
print(myvar)
• Key/Value Objects as Series
• import pandas as pd
calories = {"day1": 420, "day2": 380, "day3": 390}
myvar = [Link](calories)
print(myvar)
14 | P a g e
• Pandas Data Frames
• A Pandas Data Frame is a 2 dimensional data structure, like a 2
dimensional array, or a table with rows and columns.
• A Data Frame is like a spreadsheet or table in Python.
• Each key in the dictionary becomes a column name.
• Each list becomes the column data.
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
#load data into a DataFrame object:
df = [Link](data)
print(df)
• Read CSV Files
• A simple way to store big data sets is to use CSV files (comma
separated files).
• CSV files contains plain text and is a well know format that can be
read by everyone including Pandas.
• Reading a CSV File:
• import pandas as pd
• df = pd.read_csv("[Link]") # Read the file into a DataFrame
• print(df) # Show first 5 rows
15 | P a g e
Writing .csv file.
• import pandas as pd
• data = {
• "Name": ["Ravi", "Sneha"],
• "Age": [25, 27]
• }
• df = [Link](data)
• df.to_csv("C:/Users/ritur/OneDrive/Desktop/python/[Link]",
index=False) # Saves without row numbers
• Dataset Overview in Pandas:
Command Purpose
[Link]() Shows first 5 rows of data
[Link]() Shows last 5 rows of data
[Link]() Summary: data types, nulls, memory
[Link]() Stats of numerical columns
import pandas as pd
# Load the CSV file
df = pd.read_csv("F:/Other/[Link]")
# 1. Display the shape of the DataFrame (rows, columns)
print("Shape of the DataFrame:")
print([Link])
print("-" * 50)
# 2. Display the first 5 rows
16 | P a g e
print(" First 5 Rows:")
print([Link]())
print("-" * 50)
# 3. Display the last 5 rows
print(" Last 5 Rows:")
print([Link]())
print("-" * 50)
# 4. Summary info: data types, null values, memory
print(" Info Summary:")
print([Link]())
print("-" * 50)
# 5. Statistical description of numeric columns
print(" Descriptive Statistics:")
print([Link]())
print("-" * 50)
• Handling Missing Values
• Missing values are blank or null entries in your dataset. They can
occur due to:
• Human error
• Incomplete data collection
• Data transfer issues
• If not handled properly, AI models can give incorrect results.
• Check for Missing Values in [Link]
• Remove Rows with Missing Values”
17 | P a g e
• Pandas automatically fills blank cells with NaN when
reading a CSV file using pd.read_csv().
• df_cleaned = [Link]()
• print(df_cleaned)
• [Link]() is a Pandas function used to remove all rows that contain
missing (NaN/null) values from a DataFrame.
import pandas as pd
df=pd.read_csv("[Link]")
print([Link]())
print("-"*50)
print([Link]().sum())
print("-"*50)
df_cleaned=[Link]()
print("-"*50)
df_cleaned=[Link](how=‘all’)
print("-"*50)
df_cleaned=[Link](axis=1)
print("-"*50)
df_cleaned=[Link](subset=['Age’])
print(df_cleaned)
• Fill Missing Values with Default Value
df['Age'] = df['Age'].fillna(0) # Fill Age with 0
df['Marks'] = df['Marks'].fillna(60) # Fill Marks with
average/min/safe value
Print(df)
18 | P a g e
• Use when:
• You have domain knowledge or a safe default value
• Cleaning Data
• Data cleaning means fixing or removing incorrect, missing, or
poorly formatted data.
• Common Data Cleaning Tasks in Python:
• Check & Handle Missing Values
• print([Link]().sum()) # Count missing values
• # Fill missing values
df['Age'] = df['Age'].fillna(0) # Fill Age with 0
df['Marks'] = df['Marks'].fillna(60) # Fill Marks with default
# Or drop rows with missing values
# df = [Link]()
print("Cleaned DataFrame:")
print(df.to_string(index=False))
• Data of Wrong Format
• Cells with data of wrong format can make it difficult, or even
impossible, to analyze data.
• To fix it, you have two options: remove the rows, or convert all cells in
the columns into the same format.
19 | P a g e
import pandas as pd
# Read the CSV file
df = pd.read_csv('[Link]’)
# Convert the 'Date' column to datetime, handle mixed/invalid formats
df['Date'] = pd.to_datetime(df['Date'], errors='coerce’)
# Print full DataFrame
print(df)
20 | P a g e
Pandas Integration with Mysql
• To integrate Pandas with MySQL, you can use the sqlalchemy and
pymysql libraries.
• This lets you read from and write to MySQL databases using Pandas
DataFrames.
• Pip install sqlalchemy
• pip install mysql-connector-python
import pandas as pd
from sqlalchemy import create_engine
# Step 1: MySQL connection details
# Replace with your actual credentials
user = "root"
password = "root123"
host = "localhost"
port = 3306
database = "students"
table_name = "stu"
# Step 2: Create SQLAlchemy engine
engine =
create_engine(f"mysql+mysqlconnector://{user}:{password}@{host}:{port}
/{database}")
# Step 3: Read table into DataFrame
df = pd.read_sql(f"SELECT * FROM {table_name}", con=engine)
# Step 4: Display the data
21 | P a g e
print(" Data from table:")
print(df)
Inserting row using pandas
Step 3: INSERT new data into the table
# Create a DataFrame with new row(s) to insert
new_data = [Link]({
'name': ['Kiran', 'Meena'],
'city': ['Surat', 'Ahmedabad'],
'age': [28, 32]
})
# Insert into MySQL table
new_data.to_sql(name=table_name, con=engine, if_exists='append',
index=False)
df = pd.read_sql(f"SELECT * FROM {table_name}", con=engine)
print(" Data from table:")
print(df)
22 | P a g e
Web data collection using beautifulshop, requestes, scrappy
• Web scraping or web data collection is the process of automatically
extracting information from websites.
• Examples:
• Getting product prices from Amazon
• Collecting news headlines
• Fetching weather updates
• Extracting movie names
Scrapy
• Nowadays data is everything and if someone wants to get data from
webpages then one way to use an API or implement Web Scraping
techniques.
• In Python, Web scraping can be done easily by using scraping tools
like BeautifulSoup., REQUESTS, scrapy
• 1. Using requests and BeautifulSoup
Simple idea:
• requests is used to get the webpage
• BeautifulSoup is used to extract data from the webpage
• BeautifulSoup helps to parse and extract information from
HTML or XML content easily.
• bs4 = Beautiful Soup 4 → a Python library used for parsing HTML &
[Link] = the main class from that library.
import requests
from bs4 import BeautifulSoup
# Step 1: Send a request to the website
23 | P a g e
url = "[Link]
response = [Link](url)
# Step 2: Parse HTML content
soup = BeautifulSoup([Link], '[Link]')
# Step 3: Extract specific data (e.g., <h1> tags)
headings = soup.find_all('h1')
# Step 4: Print results
for h in headings:
print([Link])
• Why not use random websites?
• Some websites block bots.
• Some have legal restrictions.
• Your IP could get banned.
Another example continue with different web site
url = "[Link]
response = [Link](url)
soup = BeautifulSoup([Link], "[Link]")
quotes = soup.find_all("span", class_="text")
for q in quotes:
print([Link])
• 2. Using Scrapy
• Simple idea:
• Scrapy is a framework used to build spiders (web crawlers) to visit
pages and extract data.
24 | P a g e
• Scrapy is a Python tool that helps you:
• Visit websites
• Grab data (like text, prices, links, etc.)
• Save it (in files like JSON, CSV, etc.)
• It's faster and more powerful than tools like requests + BeautifulSoup,
especially for large websites with many pages.
• In order to save the time one use Scrapy.
• With the help of Scrapy one can :
1. Fetch millions of data efficiently
2. Run it on server
3. Fetching data
4. Run spider in multiple processes
• Basic Idea
• Spider: A Python class that tells Scrapy what website to visit and
what data to collect.
• Selectors: CSS or XPath code used to find elements on a webpage.
• Output: Save scraped data as JSON, CSV, etc.
• 1. Create a New Scrapy Project
• Open your terminal or command prompt and run:
scrapy startproject myproject
cd myproject
• 2. Create a Spider
• scrapy genspider example [Link]
• This creates a spider file: myproject/spiders/[Link]
25 | P a g e
• Now edit that file like this:
import scrapy
class ExampleSpider([Link]):
name = "example"
start_urls = ['[Link]
def parse(self, response):
title = [Link]('title::text').get()
yield {'title': title}
• 3. Run the Spider
• In your terminal, run:
• scrapy crawl example
• Or to save to a file:
• scrapy crawl example -o [Link]
Another example of different web site change in [Link] file
import scrapy
class ExampleSpider([Link]):
name = "example"
start_urls = ["[Link]
def parse(self, response):
# Loop through all quotes on the page
for quote in [Link]("[Link]"):
yield {
"text": [Link]("[Link]::text").get(),
"author": [Link]("[Link]::text").get(),
26 | P a g e
"tags": [Link]("[Link] [Link]::text").getall()
# Follow the next page link if available
next_page = [Link]("[Link] a::attr(href)").get()
if next_page is not None:
yield [Link](next_page, callback=[Link])
27 | P a g e
Thank You
:: Any Query ::
Contact: Ruparel Education Pvt. Ltd.
Mobile No: 7600044051
28 | P a g e