UNIT-2
EXPLAIN PANDA DATA STRUCTUTRE
Pandas is a powerful Python library used for data manipulation and data analysis. It
provides flexible and efficient data structures to handle and analyze structured data.
The two main Pandas data structures are:
1. Series (1-Dimensional)
2. DataFrame (2-Dimensional)
1. Series
A Series is a one-dimensional labeled array that can hold different types of data such as
integers, floats, or strings.
Features of Series
One-dimensional data structure
Has index and values
Similar to a column in a table
Example
import pandas as pd
data = [Link]([10, 20, 30, 40])
print(data)
Output
0 10
1 20
2 30
3 40
Here:
Index → 0,1,2,3
Values → 10,20,30,40
Example with Custom Index
data = [Link]([100, 200, 300], index=['a','b','c'])
print(data)
2. DataFrame
A DataFrame is a two-dimensional labeled data structure with rows and columns, similar
to a table or spreadsheet.
Features of DataFrame
Two-dimensional structure
Can store different types of data
Columns have names
Most commonly used structure in Pandas
Example
import pandas as pd
data = {
"Name": ["Ram", "Ravi", "Sita"],
"Age": [21, 22, 20],
"Marks": [85, 90, 88]
}
df = [Link](data)
print(df)
Output
Name Age Marks
0 Ram 21 85
1 Ravi 22 90
2 Sita 20 88
Difference Between Series and DataFrame
Feature Series DataFrame
Dimension One-dimensional Two-dimensional
Structure Single column Multiple columns
Data Type Same or mixed Different types
Example List-like Table-like
Simple Diagram
Series (1D)
Index Value
0 10
1 20
2 30
DataFrame (2D)
Index Name Age Marks
0 Ram 21 85
1 Ravi 22 90
2 Sita 20 88
2. Essential Functionality, Summarizing and Computing Descriptive Statistics
Pandas provides many built-in functions that help in data manipulation, summarizing
data, and computing descriptive statistics. These functions help analysts quickly
understand the dataset.
1. Essential Functionality of Pandas
Essential functionality refers to the basic operations used to inspect and manipulate data
in a Pandas DataFrame.
Important Functions
Function Description
head() Displays first few rows of the dataset
tail() Displays last few rows
info() Shows information about dataset
shape Returns number of rows and columns
columns Shows column names
dtypes Displays data types of columns
Example
import pandas as pd
data = pd.read_csv("[Link]")
print([Link]())
print([Link]())
print([Link])
These functions help understand structure and content of the dataset.
2. Summarizing Data
Summarizing means getting a quick overview of the dataset.
Common Summarizing Functions
Function Purpose
describe() Provides summary statistics
value_counts() Counts unique values
unique() Shows unique values
nunique() Counts number of unique values
Example
print([Link]())
print(data['Age'].value_counts())
This helps identify:
Data distribution
Most frequent values
Range of data
3. Computing Descriptive Statistics
Descriptive statistics summarize the main characteristics of data using numerical values.
Common Descriptive Statistics
Function Meaning
mean() Average value
median() Middle value
mode() Most frequent value
sum() Total sum
min() Minimum value
max() Maximum value
std() Standard deviation
var() Variance
Example
print(data['Marks'].mean())
print(data['Marks'].median())
print(data['Marks'].std())
print(data['Marks'].min())
print(data['Marks'].max())
Variance Formula (used in descriptive statistics)
s2=∑(xi−xˉ)2n−1s^2 = \frac{\sum (x_i - \bar{x})^2}{n-1}s2=n−1∑(xi−xˉ)2
spreadspreadspread
2.0
nnn
7.0
xˉ≈−0.01,;s2≈1.934\bar{x}\approx -0.01,; s^2\approx 1.934xˉ≈−0.01,;s2≈1.934
xbar=-0.01s^2 ~ 1.934
Where:
xix_ixi = each value
xˉ\bar{x}xˉ = mean
nnn = number of observations
Variance measures how far the data values spread from the mean.
Example Output of describe()
Statistic Value
Count 100
Mean 75.4
Std 8.2
Min 50
Max 95
[Link] intracting with web applicatiopns API
Web APIs (Application Programming Interfaces) allow applications to communicate
with each other over the internet. In data science and Python programming, APIs are often
used to collect data from websites, services, or online databases.
For example, APIs are used to get:
Weather information
Social media data
Financial data
Maps and location data
[Link] is a Web API?
A Web API is an interface that allows a program to request data from a web server and
receive a response.
The communication usually happens using the HTTP protocol.
Common HTTP Methods
Method Purpose
GET Retrieve data from server
POST Send data to server
PUT Update existing data
DELETE Remove data
Interacting with Web APIs Using Python
Python provides libraries such as requests to interact with web APIs easily.
Step 1: Install Requests Library
pip install requests
Step 2: Sending a Request to an API
import requests
url = "[Link]
response = [Link](url)
print(response.status_code)
print([Link])
Explanation:
[Link]() sends a request to the API
The server returns a response containing the data
Step 3: Working with JSON Data
Most APIs return data in JSON (JavaScript Object Notation) format.
Example:
import requests
url = "[Link]
response = [Link](url)
data = [Link]()
print(data)
Example: Getting Weather Data from API
import requests
url = "[Link]
response = [Link](url)
weather_data = [Link]()
print(weather_data)
Advantages of Using Web APIs
Access real-time data
Automate data collection
Integrate different systems
Useful for data science and machine learning projects
[Link] and Writing Data in Text Format
In data analysis, data is often stored in text files such as .txt, .csv, or .json. Python provides
simple methods and libraries like Pandas to read and write data in text format.
1. Reading Data from Text Files
Reading data means loading data from a file into a Python program for analysis.
Reading a Text File
file = open("[Link]", "r")
content = [Link]()
print(content)
[Link]()
Explanation:
"r" means read mode
read() reads the entire file content
Reading a CSV File using Pandas
CSV (Comma Separated Values) files are common text files used in data analysis.
import pandas as pd
data = pd.read_csv("[Link]")
print([Link]())
Explanation:
read_csv() reads the CSV file
head() shows the first few rows
2. Writing Data to Text Files
Writing data means saving data from a Python program into a file.
Writing to a Text File
file = open("[Link]", "w")
[Link]("Hello Data Science")
[Link]()
Explanation:
"w" means write mode
write() stores data in the file
Writing Data to CSV File using Pandas
import pandas as pd
data = {"Name": ["Ram", "Sita"], "Marks": [85, 90]}
df = [Link](data)
df.to_csv("[Link]")
Explanation:
to_csv() writes DataFrame data into a CSV file.
3. File Modes in Python
Mode Description
r Read file
w Write file
a Append data
r+ Read and write
Simple Diagram
Text File (.txt / .csv)
│
▼
Python Program
│
Read / Write Data
│
▼
Processed Data
[Link] Scraping, Binary Data Formats, Data Loading, Storage and File Formats
These concepts are important in data science and Python for collecting, storing, and
managing data efficiently.
1. Web Scraping
Web Scraping is the process of extracting data from websites automatically using
programs.
It helps collect large amounts of data from the internet for analysis.
Tools Used in Python
BeautifulSoup
Requests
Scrapy
Example
import requests
from bs4 import BeautifulSoup
url = "[Link]
response = [Link](url)
soup = BeautifulSoup([Link], "[Link]")
print([Link])
Applications
Price comparison websites
News data collection
Social media analysis
Market research
2. Binary Data Formats
Binary data formats store data in binary form (0s and 1s) instead of text.
They are used for efficient storage and faster processing.
Examples
Pickle
HDF5
Feather
Parquet
Advantages
Faster reading and writing
Smaller file size
Efficient for large datasets
3. Data Loading
Data loading refers to the process of importing data from different sources into a
program for analysis.
Sources of Data
CSV files
Excel files
Databases
Web APIs
JSON files
Example in Python
import pandas as pd
data = pd.read_csv("[Link]")
print([Link]())
4. Data Storage
Data storage means saving data so it can be used later.
Common Storage Methods
Text files (CSV, TXT)
Binary files
Databases (SQL, NoSQL)
Cloud storage
Example:
data.to_csv("[Link]")
5. File Formats Used in Data Science
Different file formats are used depending on the type and size of data.
File Format Description
CSV Simple text format for tabular data
JSON Data exchange format used in web APIs
Excel Spreadsheet format
HDF5 Large hierarchical data storage
Parquet Column-based storage used in big data
Pickle Python object storage
Simple Data Flow Diagram
Data Sources (Web, Files, APIs)
│
▼
Data Loading
│
▼
Data Processing
│
▼
Storage in File Formats
(CSV, JSON, Binary etc.)
6. Handling Missing Data,
Handling Missing Data ��
Handling Missing Data is an important step in data preprocessing in data science. Missing
data occurs when some values in a dataset are not recorded, empty, or unknown.
In Python, especially when using Pandas, missing values are represented as NaN (Not a
Number).
Causes of Missing Data
Missing data can occur due to:
Errors during data collection
Data entry mistakes
Equipment or sensor failure
Incomplete surveys
Data corruption
Identifying Missing Data
Pandas provides functions to detect missing values.
Function Description
isnull() Checks for missing values
notnull() Checks for non-missing values
Example
import pandas as pd
data = pd.read_csv("[Link]")
print([Link]())
This will display True where values are missing.
Methods to Handle Missing Data
1. Removing Missing Values
The dropna() function removes rows or columns containing missing data.
Example:
data = [Link]()
This removes rows that contain NaN values.
2. Filling Missing Values
The fillna() function replaces missing values with a specific value.
Example:
data = [Link](0)
This replaces missing values with 0.
3. Replacing with Mean or Median
Sometimes missing values are replaced with average values.
Example:
data['Age'] = data['Age'].fillna(data['Age'].mean())
4. Forward Fill and Backward Fill
Method Description
ffill() Fill using previous value
bfill() Fill using next value
Example:
[Link](method='ffill')
Data Transformation,:
Data Transformation ��
Data Transformation is the process of converting data from one format or structure into
another to make it suitable for analysis and processing. It is an important step in data
preprocessing in data science.
Data transformation helps improve data quality, consistency, and usability.
Objectives of Data Transformation
Convert data into a useful format
Improve data quality
Prepare data for analysis and machine learning
Standardize data values
Common Types of Data Transformation
1. Normalization
Normalization scales data values to a common range, usually between 0 and 1.
Example:
normalized_value = (x - min) / (max - min)
It is useful when data values vary widely.
2. Standardization
Standardization converts data so that it has mean = 0 and standard deviation = 1.
This is commonly used in machine learning algorithms.
3. Aggregation
Aggregation means combining multiple values into a single value.
Example:
Total sales per month
Average marks of students
Example in Python:
import pandas as pd
data['Total'] = data['Price'] * data['Quantity']
4. Data Type Conversion
Changing the data type of a variable.
Example:
data['Age'] = data['Age'].astype(int)
5. Creating New Variables (Feature Engineering)
New columns can be created from existing data.
Example:
data['Profit'] = data['Revenue'] - data['Cost']
Example Table
Before Transformation:
Name Price Quantity
Ram 50 2
Ravi 40 3
After Transformation:
Name Price Quantity Total
Ram 50 2 100
Ravi 40 3 120
String Manipulation:
String Manipulation in Python (Pandas) ��
String Manipulation refers to the process of modifying, cleaning, and analyzing text data.
In data science, datasets often contain text such as names, addresses, emails, or product
descriptions, which need to be processed before analysis.
In Python, especially with Pandas, string operations are performed using the .str accessor.
Common String Manipulation Functions
Function Description
[Link]() Converts text to lowercase
[Link]() Converts text to uppercase
[Link]() Removes extra spaces
[Link]() Replaces characters or words
[Link]() Checks if a string contains specific text
[Link]() Splits a string into parts
Examples in Python
Convert Text to Lowercase
import pandas as pd
data['Name'] = data['Name'].[Link]()
Example:
Name
RAM
RAVI
After conversion:
Name
ram
ravi
Convert Text to Uppercase
data['Name'] = data['Name'].[Link]()
Remove Extra Spaces
data['City'] = data['City'].[Link]()
Replace Words
data['City'] = data['City'].[Link]("Hyd", "Hyderabad")