Scribd
Upload
14 views15 pages

Python Pandas EDA Guide for BTech Students

Uploaded by

patel.b.jaydeep24
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
14 views15 pages

Python Pandas EDA Guide for BTech Students

Uploaded by

patel.b.jaydeep24
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25

(A Constituent College of Somaiya Vidyavihar University)

KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25


Batch:A2 Roll No:16010423058 Experiment No.:3 Aim: To perform

exploratory data analysis using python Pandas

Resources needed: Python IDE

Theory:
Pandas is a Python library that provides extensive means for data analysis. Data scientists often
work with data stored in table formats like .csv, .tsv, or .xlsx. Pandas makes it very convenient to
load, process, and analyze such tabular data using SQL-like queries. Python has long been great
for data munging and preparation, but less so for data analysis and modeling. pandas helps fill
this gap, enabling you to carry out
your entire data analysis workflow in
Python. In conjunction with
Matplotlib and Seaborn, Pandas
provides a wide range of
opportunities for visual analysis of
tabular data.

Installing pandas library:

Conda install pandas

Or

pip install pandas

The main data structures in Pandas are implemented with Series and DataFrame classes. The
former is a one-dimensional indexed array of some fixed data type. The latter is a two-
dimensional data structure - a table - where each column contains data of the same type. You can
see it as a dictionary of Series instances. DataFrames are great for representing real data: rows
correspond to instances (examples, observations, etc.), and columns correspond to features of
these instances.

A series can be created using list ,dictionary etc. with index(implicit indexing) or without
index(explicit indexing).

import pandas as pd
data1=[Link]({2:'a', 1:'b', 3:'c'}) #implicit indexing
data2=[Link]({2:'a', 1:'b', 3:'c'}, index=[1,2,3]) # explicit indexing
#loc attribute allows indexing and slicing that always references the explicit index:
[Link][2]
#iloc attribute allows indexing and slicing that always references the implicit (A Constituent

College of Somaiya Vidyavihar University)

KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25

#Python-style index
[Link][1]

Following are the various series related operations


•Append(): [Link](s1) # Stitch s1 to s3
•Drop: [Link]('e') #Delete the value whose index is e
•Addition: [Link](s3)#addition according to the index, and it would be filled with NaN (null
value) if the indexes are different.
•Subtraction: [Link](s3) #substraction according to the index, and it would be filled with
NaN (null value) if the indexes are different.
•Multiplication: [Link](s3) #multiplication according to the index, and it would be filled
with NaN (null value) if the indexes are different.
•Division: [Link](s3)
•Median: [Link]()
•Sum: [Link]()
•Maximum & Minimum :
[Link]() [Link]()

A data frame object keeps track


of both data (numerical as well
as text), and column and row
headers. It can have multiple
columns of data.
convert a numpy array to a
pandas data frame with [Link]
frame().
import numpy as np
h = [[1,2],[3,4]]
df_h = [Link](h)
print('Data Frame:', df_h)
data frame can read data from dictionary and files as well.

Reading and writing data from files:


CSVs don't have indexes like our DataFrames, so all we need to do is just designate the
index_col when reading.
import pandas as pd
df=pd.read_csv("C:/Users/Admin/Desktop/ADVANCED
PYTHON/DATA/[Link]",index_col=0)
#Reading the dataset in a dataframe using Pandas
print(df)
To write data to a new csv file use to_csv()
df3.to_csv('[Link]')
df3.to_excel('[Link]', sheet_name='Sheet1')

Following functions of dataframe can be used to explore dataset to get summary of it.

(A Constituent College of Somaiya Vidyavihar University)

KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25

• info() provides the essential details about your dataset, such as the number of rows and
columns, the number of non-null values, what type of data is in each column, and how
much memory your DataFrame is using.
[Link]()
• describe() is used to get a summary of numeric values in your dataset. It calculates the
mean, standard deviation, minimum value, maximum value, 1st percentile, 2nd percentile,
3rd percentile of the columns with numeric values. It also counts the number of variables
in the dataset.
[Link]()
describe() can also be used on a categorical variable to get the count of rows, unique
count of categories, top category, and freq of top category
temp_df['product'].describe()
•head() outputs the first five rows of your DataFrame by default, but we could also pass a
number as well
print([Link])
•tail() outputs last five rows
by default.
print([Link])
•shape() outputs just a tuple
of (rows, columns):
[Link]
Row and column selection:
Each row and column of
dataframe is a series . following
functions can be used for row
selection and column selection
Extracting a column using square brackets will return a Series.
prod_col=temp_df['product']
#selecting multiple columns
subset=temp_df[['product','price']]
accessing rows:
.loc - locates by name
prom = movies_df.loc["Prometheus"]
prom
.iloc- locates by numerical index
prod = [Link][1]
prod

Further analysis using pandas dataframe:

value_counts() can tell us the frequency of all values in a column.


temp_df['product'].value_counts().head(10)
nunique() to count number of unique values that occur in dataset or in a column
[Link]() #to see the counts of unique numbers in each column

(A Constituent College of Somaiya Vidyavihar University)


KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25

df["Embarked"].nunique() #to get the unique count of a column


corr() generate the relationship between each continuous variable:
temp_df.corr()
Correlation tables are a numerical representation of the bivariate relationships in the
dataset.
astype() can be used to change the datatype of that column
df["Embarked"] = df["Embarked"].astype("category")
df["Embarked"].dtype

column clean up funtions:


append() will return a copy after appending without affecting the original DataFrame(if
inplace attribute is used).
temp_df = [Link](df)
temp_df.shape
drop_duplicates() method will return a copy of
DataFramewith duplicates removed. temp_df =
temp_df.drop_duplicates()
#inorder to avoid reasignment it can be done
temp_df.drop_duplicates(inplace=True)
temp_df.shape
.columns print the column names of dataset also can be used for renaming
temp_df.columns
drop() can be used to delete
columns
[Link](columns=['A', 'C'])
rename() method is used to rename certain or all columns via a
dict. temp_df.rename(columns={
'Account_Created': 'Acc_Created',
'Last_Login': 'Lst_Login'
}, inplace=True)
temp_df.columns

Handling null values using pandas:


Mostly Python's None or NumPy's [Link] indicates missing or null values. • isnull() checks
which cells in our DataFrame are null. It returns a DataFrame where each cell is either True
or False depending on that cell's null status.
temp_df.isnull()
To count the number of nulls in each column we use an aggregate function .sum() for
summing: temp_df.isnull().sum()
To get rid of rows or columns with nulls. Removing null data is only suggested if you have a
small amount of missing data. .dropna() will delete any row with at least a single null value, but

(A Constituent College of Somaiya Vidyavihar University)


KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25

it will return a new DataFrame without altering the original one.


temp_df.dropna()
– Or drop columns with null values by setting axis=1.
temp_df.dropna(axis=1)
– Replace nulls with non-null values, a technique known as imputation. Normally null
value is replaced with mean or the median of that column.

Conditional selections/ Filtering


Comparison operators are used for filtering
Take a column from the DataFrame and apply a Boolean condition to
it. condition = (movies_df['Director'] == "Ridley Scott")
It returns a Series of True and False [Link] more examples on
conditionals Select movies_df where movies_df Director equals Ridley Scott.
movies_df[movies_df['Director'] == "Ridley Scott"]
movies_df[movies_df['Rating'] >= 8.6].head(3)
movies_df[(movies_df['Director'] ==
'Christopher Nolan') | (movies_df['Director'] == 'Ridley
Scott')].head()
movies_df[movies_df['Director'].isin(['Christopher Nolan', 'Ridley Scott'])].head()

Summary statistics Functions/ Aggregate Functions


Aggregating functions are the ones that reduce the dimension of the returned objects.
DataFrames include some aggregate functions to understand the overall properties of a dataset
[Link](): Count the number of rows /items
[Link](): To find mean average of data frame
Synatx: [Link]()#where Population is column name
[Link](): To find median of data frame
[Link]():
[Link](): Do a summation operation on any column in the DataFrame
[Link](): To find Product of all items
[Link]():To find standard deviation of a data
frame [Link]():To find varianceof data frame
[Link](), [Link](): To find Minimum and
maximum [Link](), [Link](): First and last item

GROUP BY and aggregation


“group by” process can involve one or more of the following steps:
–Splitting the data into groups based on some criteria.
–Applying a function to each group independently.
–Combining the results into a data structure
Apply step involves following
–Aggregation: compute a summary statistic (or statistics) for each group. Some examples:

(A Constituent College of Somaiya Vidyavihar University)

KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25

•Compute group sums or means.


•Compute group sizes / counts.
–Transformation: perform some group-specific computations and return a like-indexed
object. Some examples:
•Standardize data (zscore) within a group.
•Filling NAs within groups with a value derived from each group.
–Filtration: discard some groups, according to a group-wise computation that evaluates True
or False. Some examples:
•Discard data that belongs to groups with only a few members.
•Filter out data based on the group sum or mean.

# group the data on team value.


gk = [Link]('Team')
# Finding the values contained in the "Boston Celtics" group
gk.get_group('Boston
Celtics')

Applying functions
To iterate over a DataFrame
or Series we can use list, but
doing so — especially on
large datasets — is very
slow. An efficient alternative
is to apply() a function to the
dataset. Using apply() will be much faster than iterating manually over rows because pandas is
utilizing vectorization. Combining datasets:

Combining Datasets
Concat: s to append either columns or rows from one DataFrame to another.
Joining two dataframe on the index
merge two dataframes on key
attribute

Activities:
1. Download data set with atleast 1500 rows and 10-20 columns(numeric and non numeric)
from valid data sources
2. Read same in pandas DataFrame
3. Perform in detail Exploratory data analysis of this dataset
●Get information and description of dataset.
●See if any null values are present. Display count of null values.

(A Constituent College of Somaiya Vidyavihar University)


KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25

●Choose appropriate technique to handle missing values.(imputation with use of inplace)


●Use sorting of data in dataframe to display topmost 5 or 8 records based on one or more
column values(conditional filtering)
●Get frequency listing of any one relavant column(2 cases)
●Sorting of rows and columns,( implicit and explicit indexing)
●Accessing particular row based on certain condition and displaying only one or few
columns from it.(3 cases with compound conditions)
●Minimum and maximum values related analysis
●Use of group by on one or more columns(2 cases)
●Add new column to existing dataframe and populate same using existing columns data.
●Use of appropriate aggregate functions with groupby.(2 cases)
●Selection on particular groups based on name or condition
●Find correlation between any two columns values.
●Try transformation(normalization using any technique) on data set
●Joining , merging and concatenation of data in dataframe.

Write down observation for your dataset for each of

above listed task of analysis. Result: (script and

output)

(A Constituent College of Somaiya Vidyavihar University)


KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25
(A Constituent College of Somaiya Vidyavihar University)

KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25


(A Constituent College of Somaiya Vidyavihar University)

KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25


Outcomes:
CO3: Demonstrate use of pandas, numpy, matplotlib and Seaborn libraries for data analysis
and data visualization

(A Constituent College of Somaiya Vidyavihar University)


KJSCE/IT/AIDS/SY BTech/SEM III/PL1- PYTHON/2024-25

Conclusion: (Conclusion to be based on the objectives and outcomes achieved)

We understood the application of Pandas library in python.


References:

1. 1. Daniel Arbuckle, Learning Python Testing, Packt Publishing, 1st Edition, 2014
2. Wesly J Chun, Core Python Applications Programming, O’Reilly, 3rd Edition,
2015 3. Wes McKinney, Python for Data Analysis, O’Reilly, 1st Edition, 2017
4. Albert Lukaszewsk, MySQL for Python, Packt Publishing, 1st Edition, 2010 5.
Eric Chou, Mastering Python Networking, Packt Publishing, 2nd Edition, 2017

(A Constituent College of Somaiya Vidyavihar University)

Common questions

Powered by AI

Pandas simplifies handling and cleaning large datasets with several built-in functions. Missing data can be identified using isnull(), which checks for null values. One can remove rows with dropna(), or fill missing values using methods like fillna() for imputation. This supports replacing null values with statistical measures like mean or median. These tools ensure efficient cleanup without altering the original dataset unless explicitly required with inplace=True, thus making it very efficient for managing large datasets .

The groupby function in Pandas splits the data into separate groups based on some criteria, allowing for aggregate functions to compute summary statistics such as mean, sum, and count within these groups. This process involves splitting, applying, and combining data through aggregation, transformation, and filtration steps. Aggregation can involve computing statistics across groups, transformation standardizes data within groups, and filtration discards groups based on computation results. These capabilities enable detailed and nuanced data exploration and analysis .

The describe() function in Pandas is instrumental in initial data exploration as it provides a statistical summary of numeric columns, including count, mean, standard deviation, minimum, and maximum values, and percentiles. This summary offers insights into data distribution, central tendency, and variability, enabling a quick assessment of data characteristics and identification of potential anomalies or points of interest early in the data analysis process .

The primary data structures provided by the Pandas library are Series and DataFrame. A Series is a one-dimensional indexed array that can consist of any data type, while a DataFrame is a two-dimensional data structure, essentially a table, where each column contains data of the same type. You can visualize a DataFrame as a dictionary of Series instances, making it suitable for representing real-world tabular data where rows correspond to instances and columns correspond to features of these instances .

Pandas facilitates merging and concatenating datasets using functions like concat, which allows appending columns or rows from one DataFrame to another, and merge, which combines DataFrames on shared keys. These operations are beneficial for integrating multiple datasets into a cohesive framework, enabling complex data analyses that leverage multiple data sources. Merging specifically helps in aligning data on common fields, aiding in the creation of comprehensive datasets for analysis .

Sorting and indexing in Pandas enhance data accessibility by enabling efficient arrangement and retrieval of data. Sorting can arrange data based on column values or indexes, optimizing data organization and analysis workflows. Indexing, through methods like loc and iloc, provides efficient data access based on label or integer positions, making data filtering and selection both intuitive and rapid, which is critical for efficient data manipulation especially in complex datasets .

The apply() function in Pandas is more efficient than using a for-loop for row iteration because it leverages vectorization, making operations faster by performing them over entire arrays at once, rather than row-by-row. This efficiency is particularly noticeable with large datasets where the overhead of Python loops becomes significant. Apply() processes data using lower-level optimizations that improve performance significantly compared to manual iteration .

The drop_duplicates() function aids in maintaining data integrity by removing duplicate records from a DataFrame, ensuring that analyses are conducted on unique data instances. This is crucial when data redundancy could skew results and lead to incorrect conclusions. By enforcing uniqueness, these functions help maintain the quality and accuracy of datasets, particularly in large-scale data analysis where duplicate data is more prevalent .

Matplotlib and Seaborn play a crucial role in visual data analysis when used alongside Pandas. They provide extensive means for visualizing tabular data, which is vital for exploring datasets, identifying patterns, trends, and outliers, and effectively communicating data insights. These libraries offer functionalities to create various types of plots, facilitating easier comprehension and interpretation of complex datasets and enhancing the overall data analysis process .

Pandas correlation tables quantify relationships between continuous variables in a dataset, providing coefficients that indicate the strength and direction of linear relationships. Positive values imply direct relationships, while negative values show inverse relationships. By analyzing these coefficients, one can identify which variables are likely influencing each other, aiding in hypothesis formulation and deeper statistical analysis, thereby enhancing decision-making processes .

You might also like