Python for
Explorative Data Analysis
(EDA)
Python Libraries for Data Science
Many popular Python toolboxes/libraries:
• NumPy
• SciPy
• Pandas
• SciKit-Learn
Visualization libraries
• matplotlib
• Seaborn
and many more …
2
Python Libraries for Data Analysis
NumPy:
▪ introduces objects for multidimensional arrays and matrices, as well as
functions that allow to easily perform advanced mathematical and statistical
operations on those objects
▪ provides vectorization of mathematical operations on arrays and matrices
which significantly improves the performance
▪ many other python libraries are built on NumPy
Link: [Link]
3
Python Libraries for Data Science
Pandas:
▪ adds data structures and tools designed to work with table-like data (similar
to Series and Data Frames in R)
▪ provides tools for data manipulation: reshaping, merging, sorting, slicing,
aggregation etc.
▪ allows handling missing data
Link: [Link]
4
Python Libraries for Data Science
matplotlib:
▪ python 2D plotting library which produces publication quality figures in a
variety of hardcopy formats
▪ a set of functionalities similar to those of MATLAB
▪ line plots, scatter plots, barcharts, histograms, pie charts etc.
▪ relatively low-level; some effort needed to create advanced visualization
Link: [Link]
5
Python Libraries for Data Science
Seaborn:
▪ based on matplotlib
▪ provides high level interface for drawing attractive statistical graphics
▪ Similar (in style) to the popular ggplot2 library in R
Link: [Link]
6
Data Frame data types
Pandas Type Native Python Type Description
object string The most general dtype. Will be
assigned to your column if column
has mixed types (numbers and
strings).
int64 int Numeric characters. 64 refers to
the memory allocated to hold this
character.
float64 float Numeric characters with decimals.
If a column contains numbers and
NaNs(see below), pandas will
default to float64, in case your
missing value has a decimal.
datetime64, timedelta[ns] N/A (but see the datetime module Values meant to hold time data.
in Python’s standard library) Look into these for time series
experiments.
7
Data Frames attributes
Python objects have attributes and methods.
[Link] description
dtypes list the types of the columns
columns list the column names
axes list the row labels and column names
ndim number of dimensions
size number of elements
shape return a tuple representing the dimensionality
values numpy representation of the data
8
Data Frames methods
Unlike attributes, python methods have parenthesis.
All attributes and methods can be listed with a dir() function: dir(df)
[Link]() description
head( [n] ), tail( [n] ) first/last n rows
describe() generate descriptive statistics (for numeric columns only)
max(), min() return max/min values for all numeric columns
mean(), median() return mean/median values for all numeric columns
std() standard deviation
sample([n]) returns a random sample of the data frame
dropna() drop all the records with missing values
9
Aggregation Functions in Pandas
Aggregation - computing a summary statistic about each group, i.e.
• compute group sums or means
• compute group sizes/counts
Common aggregation functions:
min, max
count, sum, prod
mean, median, mode, mad
std, var
10
Basic Descriptive Statistics
[Link]() description
describe Basic statistics (count, mean, std, min, quantiles, max)
min, max Minimum and maximum values
mean, median, mode Arithmetic average, median and mode
var, std Variance and standard deviation
sem Standard error of mean
skew Sample skewness
kurt kurtosis
11
Graphics
description
distplot histogram
barplot estimate of central tendency for a numeric variable
violinplot similar to boxplot, also shows the probability density of the
data
jointplot Scatterplot
regplot Regression plot
pairplot Pairplot
boxplot boxplot
swarmplot categorical scatterplot
factorplot General categorical plot
12
To store all Python code files, data, outputs, plots
etc.. in one place…
[Link] the following folder
“C:\iFEEL\MADM\Python”
2. Copy paste downloaded “[Link]” file in
this “Python” folder
13
Launch : Python Spyder IDE Scientific Python Development IDE
2. Click Anaconda ➔
3. Click Spyder ➔
1. Click windows ➔
14
Click “View“➔
Click
“Windows
layouts“➔
Click “Rstudio layout“➔
15
Set “C:\iFEEL\MADM\Python” as current working directory
Click
Box-3 : browse
Variables icon
Box-1 : Enter code Plots
History
Box-2 : Console ➔outputs Box-4 : Files & Help
16
Congratulations !!.... Your Python coding starts now ….
# is used for Text remarks , it will NOT considered as Code for running
17
Before proceed further….
Save file ;
File-> Save as ➔ ‘credit-risk’ in your folder
18
Now, File name shown ➔ ‘[Link]’
Before proceed further….
Save file ;
File-> Save as ➔ ‘credit-risk’ in your folder
19
Write at line 2 …Remark as what you are doing in next line/lines
Pandas provides NumPy stands for ‘Numerical
high-performance Python’ or ‘Numeric Python’. It is
data manipulation an open source module of Python
in Python which provides fast mathematical
computation on arrays and
matrices
20
Pandas imported and short-named as ‘pd’ OR
Run this line no. 3 …. by pressing “F9” Click this icon
21
From Pandas library ..import these two for data manipulation
Run this line no. 4 …. by pressing “F9”
22
Import numpy and short-named as ‘np’
Run this line no. 5 …. by pressing “F9”
23
Import matlpot and short-named as ‘plt’
Run this line …. by pressing “F9”
24
Import Seaborn graphics and short-named as ‘sns’
###Seaborn is a Python data visualization library based on matplotlib.
Run this line …. by pressing “F9” 25
To display graphics on screen.. Type code line 9
Run this line …. by pressing “F9”
26
To display OUTPUTS in below Console…. Type code lines 12-15
Run these lines …. ONE-by-ONE …by pressing “F9”
27
Now….copy paste the downloaded data file in this ‘Working
Directory’… ( Do it from
28
Load Excel data ( CSV format saved) as ‘df’ in Python
29
In Box-3…the following
‘df’ should appear..
30
To understand data types ….use line 21 code..
Output Analysis:
RangeIndex ➔[Link] Rows
Object ➔ Nominal/Ordinal
( Categorical data)
int64/float64 ➔ Interval/Ratio
( Numeric data ) 31
To get Descriptive Stats for ‘Numeric data’...use line 23 code..
32
To get Descriptive Stats for ‘Numeric data’...use line 23 code..
33
To get Descriptive Stats for ‘Categorical data’...use line 25 code..
Unique ➔ [Link] levels / categories
top & freq ➔ Max category & its frequency ( Males are Max with 499 freq.)
34
agg ➔ aggregate function to find any
Square brackets for stats.. Round brackets() for
selecting a column ..agg..function & within square brackets
[] for stats.
To find ‘LoanAmount’ data distribution …type code line 27
Output …in Console.
Comment on the distribution.. Class discuss..
35
To find ‘LoanAmount’ data distribution …type code line 27
Output …in Console.
Comment on the distribution.. Class discuss..
36
Plot Histogram of ‘LoanAmount’ data ..type code line 33-34
( To run press F9 by selecting both lines)
Output …in
separate window
Comment on the
distribution.. Class
discuss.. 37
Plot curve of ‘LoanAmount’ data ..type code line 36-37
( To run press F9 by selecting both lines) [ kde=True will plot curve ]
Output …in
separate window
Comment on the
distribution.. Class
discuss..
38
groupby ➔ group data as per object ( categorical data )
To find Gender wise LoanAmount distribution. Use line code 39
Output
Comment on the distribution..
Class discuss..
39
hue➔shows groupwise fill➔fills bars with colors
Kde True➔draws curve
[Link] ➔Gender wise LoanAmount . Use line code 41-42
Output on separate window..
Comment on the distribution..
Class discuss..
40
41
kind kde➔draws only curve
Only [Link] (No histogram) ➔Education wise LoanAmount . Use line code 45
Output on separate window..
Comment on the distribution..
Class discuss..
42
43
Practice : Do on your own 46-47 line EDA and share your screen
44
Practice : Do on your own 46-47 line EDA and share your screen
45
Practice : Do on your own 49 line EDA and share your screen
46
Practice : Do on your own 49 line EDA and share your screen
47
48
EDA-2 : Genderwise comparison of LoanAmount & ApplicantIncome
Create new dataframe dfG of Gender groupwise
49
EDA-2 : Genderwise comparison of LoanAmount & ApplicantIncome
Find Stats as per line 53 code.
50
EDA-2 : Genderwise comparison of LoanAmount & ApplicantIncome
Scatter plot as per line 55 code
51
52
EDA-2 : Genderwise comparison of LoanAmount & ApplicantIncome
Joint plot with kde as per line 57-58 code
53
54
EDA-3 : Correlation Genderwise for ALL numeric data types
Pearson correlation as per line 60 code
55
56
EDA-3 : Scatter plot Genderwise for ALL numeric data types
Plot as per line 62 code
57
58
EDA-3 : Practice Assignment
59
EDA-3 : Practice Assignment
60
EDA-3 : Practice Assignment
61
EDA-3 : Practice Assignment
62
63
EDA-3 : Practice Assignment
64
65
EDA-3 : Practice Assignment
66
67
EDA-3 : Practice Assignment
68
69
EDA-4 : Categorical Data Analysis using Crosstables and catplots
Click ‘df’ in Box-3
70
Rows
Pandas Two columns (Edu & Gender )
cross table Values for Aggregate
Aggregate function :
fn. mean mean
EDA-4 : Categorical Data Analysis using Crosstables and catplots
71
72
X-axis
Seaborn Y-axis
Categorical Genderwise Columns in
plot plot
EDA-4 : Categorical Data Analysis using Crosstables and catplots
73
74
EDA-4 :
Practice : As per line 80-81 ( Hint : Line code 74-79 )
75
EDA-4 :
Practice : As per line 80-81 ( Hint : Line code 74-79 )
76
77
EDA-4 :
Practice : As per line 80-81 ( Hint : Line code 74-79 )
78
79
EDA-4 :
Creating crosstable and saving it to Excel ( Read 87 )
80
Print both Row-Totals Custom heading for
and Column-Totals Totals
Creating crosstable ( code as per 88-89 )
81
Custom Excel file name
Dataframe to be saved…
saving it to Excel…( code as per 91 )
82
Now..on your own…Practice as per 92
83
Now..on your own…Practice as per 92
84
Now..on your own…Practice as per 94
85
Now..on your
own…Practice
as per 94
86
EDA-5 : Box-plot Analysis
87
EDA-5 : Box-plot Analysis
88
89
Better advanced Box-plot …as per code 103-104
90
91
Some advanced Plots and tables…
92
lm➔linear model
x ➔ x=axis, y➔ y-axis, col ➔Columns in plot, hue➔ Genderwise
Note : Select both lines 107-108 , then Run F9
93
94
95
Counts ‘Object’ variables
96
Index (Row-names) are used as labels in Pie plot
97
Index (Row-names) are used as labels in Pie plot
Autopct ➔ Automatic % figures in pie plot
%0.2f ➔ Syntax for printing “Two decimal” numbers
% ➔ Extra % sign is just to show after no. (ie:25.42%)
98
Index (Row-names) are used as labels in Pie plot
Autopct ➔ Automatic % figures in pie plot
%0.2f ➔ Syntax for printing “Two decimal” numbers
% ➔ Extra % sign is just to show after no. (ie:25.42%)
99
100
Hint : Use code lines 110-114
Like the Output in next slide…
101
102
103
Creating Frequency Pivot table as below ….
104
User defined table name
Index ➔ Rows of Table
105
106
Any unique column to count
107
Aggregate function counts
and gives frequency
108
True ➔ Prints both Row and
Column totals
109
Gives names for total
margins
110
Note : Select ALL lines 122-127 …… and then only Run F9
111
Displays the table ➔
112
We can directly plot ‘bar plot’ by using code line 129..
Plot in next slide….
113
To adjust x-axis labels properly display ..click “Configure subplot”
114
Click…Tight layout
115
116
Any Output/Table can be
export to Excel giving your
own file name
117
Practice :
a) Get Mean Applicant Income as per below Pivot table format
b) Draw ..Direct plot from Pivot table and format it
c) Write table to Excel CSV file
( Hint : Use aggfunc=[Link]) & code lines 122-132
118
Practice :
a) Get Mean Applicant Income as per below Pivot table format
b) Draw ..Direct plot from Pivot table and format it
c) Write table to Excel CSV file
( Hint : Use aggfunc=[Link]) & code lines 122-132
119