Python and Basic Statistics
Contents
•What is Python & History
•Installing Python & Python Environment
•Basic commands in Python
•Data importing
•Basic details of the data frames
•Basic statistics
•Measures of dispersion
•Data exploration
•Data cleaning
Introduction to Python &
History
What is python
•It’s a language
•Human-readable syntax and well Documented
•Open Source (Free)
•Powerful scripting language with simple Syntax
•[Link]
•Used by many data scientists and developers
Installing Python & Python
IDEs
Writing and executing python programs
•Python has many options to write and execute a program
•You can use Text Editors or Command line interfaces or Notebook or an
IDE
•We will use Spyder IDE in our course
•Anaconda distribution has all the required software's inbuilt. We just
need to download and install it.
Installing Python, Anaconda
•Download and install Anaconda3
•It automatically installs
• Ipython
• Jupyter notebook
• Spyder IDE
• Google Colab
Python on Cloud – Google Colab
Basic Commands in Python
Before you code
• Python is case sensitive
• Be careful while using the Variable names and Function names
• Sales_data is not same as sales_data
• Print() is not same as print()
Basic Commands
571+95
19*17
print(57+39)
print(19*17)
print(“Statinfer")
# use hash(#) for comments
#Division example
34/56
Important Packages
Packages
•To be a good data scientist on python, on need to be very comfortable
with below packages
• numpy
• pandas
• scikit-Learn
• matplotlib
• nltk
Important Packages- NumPy
▪NumPy is for fast operations on vectors and matrices, including
mathematical, logical, shape manipulation, sorting, selecting.
▪It is the foundation on which all higher level tools for scientific Python
packages are built
import numpy as np
income = [Link]([9000, 8500, 9800, 12000, 7900, 6700, 10000])
expenses=income*0.6525
print(expenses)
savings=income-expenses
print(savings)
Important Packages- Pandas
•Data frames and data handling
•Pandas has Data structures and operations for manipulating numerical tables
and time series.
import pandas as pd
#bank= pd.read_csv('D:\\Datasets\\Bank Tele Marketing\\bank_market.csv’)
bank= pd.read_csv('[Link]
tasets/master/Bank%20Tele%20Marketing/bank_market.csv')
print(bank)
Important Packages- Matplotlib
Plotting library similar to MATLAB plots
X = [Link](50)
Y = [Link](50)
print("X Array \n" ,X )
print("Y Array \n", Y)
import matplotlib as mp
[Link](X,Y)
Important Packages- Scikit-Learn
• Machine learning algorithms made easy
import sklearn as sk
import pandas as pd
air = pd.read_csv("D:\\Google
Drive\\Training\\Datasets\\AirPassengers\\[Link]")
air
from sklearn.linear_model import LinearRegression
lr = LinearRegression()
[Link](air[["Promotion_Budget"]], air[["Passengers"]])
#Coefficients
print(lr.coef_)
print(lr.intercept_)
Basic Commands on Datasets
•Is the data imported correctly? Are the variables imported in right
format? Did we import all the rows?
•Once the dataset is inside Python, we would like to do some basic
checks to get an idea on the dataset.
•Just printing the data is not a good option, always.
•Is a good practice to check the number of rows, columns, quick look at
the variable structures, a summary and data snapshot
18
Check list after Import
Dataset - Sales_by_country_v1.csv
Code Description
[Link] To check the number of rows and
columns
[Link]() All important information
[Link](10) First few observations of data
[Link](10) Last few observations of the data
19
Descriptive statistics
•The basic descriptive statistics to give us an idea on the variables and
their distributions
•Permit the analyst to describe many pieces of data with a few indices
•Central tendencies
• Mean
• Median
•Dispersion
• Range
• Variance
• Standard deviation
20
[Link]
Central tendencies
•Mean
•The arithmetic mean
•Sum of values/ Count of values
•Gives a quick idea on average of a variable
21
[Link]
Mean in Python
Import “Census Income Data/Income_data.csv”
gain_mean=Income["capital-gain"].mean()
print(gain_mean)
22
[Link]
Guess the mean
1.5,1.7,1.9,0.8,0.8,1.2,1.9,1.4, 9 , 0.7 , 1.1
Median
•Mean is not a good measure in presence of outliers
•For example Consider below data vector
• 1.5,1.7,1.9,0.8,0.8,1.2,1.9,1.4, 9 , 0.7 , 1.1
•90% of the above values are less than 2, but the mean of above vector
is 2
•There is an unusual value in the above data vector i.e 9
•It is also known as outlier.
•Mean is not the true middle value in presence of outliers. Mean is very
much effected by the outliers.
•We use median, the true middle value in such cases
24
•Sort the data either in ascending or descending order
[Link]
Median
1.5 0.7 • Mean of the data is 2
1.7 0.8 • Median of the data is 1.4
1.9 0.8 • Even if we have the outlier as 90, we will have the
0.8 1.1
same median
0.8 1.2
• Median is a positional measure, it doesn’t really
1.2 1.4 depend on outliers
1.9 1.5
• When there are no outliers then mean and median will
1.4 1.7
9
be nearly equal
1.9
0.7 1.9
• When mean is not equal to median it gives us an idea
1.1 9
on presence of outliers in the data
25
[Link]
Mean and Median
Import “Census Income Data/Income_data.csv”
gain_median=Income["capital-gain"].median()
print(gain_median)
26
[Link]
Dispersion Measures : Variance
and Standard Deviation
Dispersion
• Just knowing the central tendency is not enough.
• Two variables might have same mean, but they might be very different.
• Look at these two variables. Profit details of two companies A & B for last 14
Quarters in MMs
Mean
Company A 43 44 0 25 20 35 -8 13 -10 -8 32 11 -8 21 15
Company B 17 15 12 17 15 18 12 15 12 13 18 18 14 14 15
• Though the average profit is 15 in both the cases
• Company B has performed consistently than company A.
• There was even loses for company A 28
• Measures of dispersion become very vital in such cases
[Link]
Variance and Standard deviation
• Dispersion is the quantification of deviation of each point from the mean value.
• Variance is average of squared distances of each point from the mean
• Variance is a fairly good measure of dispersion.
• Variance in profit for company A is 352 and Company B is 4.9
Value Value-Mean (Value-Mean)^2
Value Value-Mean (Value-Mean)^2
43 28 784
17 2 4
44 29 841
15 0 0
0 -15 225 n
(
i )
12 -3 9
25 10 100
−
2
20 5 25 x x 17 2 4
15 0 0
35
-8
20
-23
400
529 2 = i =1
18 3 9
13 -2 4 n 12
15
-3
0
9
0
-10 -25 625
12 -3 9
-8 -23 529
13 -2 4
32 17 289
18 3 9 29
11 -4 16
-8 -23 529 18 3 9
21 6 36 14 -1 1
14 -1 1
15.0 352 [Link]
15.0 4.9
Standard Deviation
•Standard deviation is just the square root of variance
•Variance gives a good idea on dispersion, but it is of the order of
squares.
•Its very clear from the formula, variance unites are squared than that
of original data.
•Standard deviation is the variance measure that is in the same units as
the original data n
(x − x)
2
i
s= i =1
30
[Link]
LAB: Variance and Standard deviation
•Dataset: "./Online Retail Sales Data/Online [Link]”
• Use option pd.read_csv(“C:/Online Retail Sales Data/Online
[Link]”, encoding = “ISO-8859-1”)
•What is the variance and s.d of “UnitPrice”
•What is the variance and s.d of “Quantity”
31
[Link]
LAB: Variance and Standard deviation
#var and sd UnitPrice
var_UnitPrice=Online_Retail['UnitPrice'].var()
print("Variance of UnitPrice", var_UnitPrice)
std_UnitPrice=Online_Retail['UnitPrice'].std()
print("S.D of UnitPrice", std_UnitPrice)
#var and sd Quantity
var_Quantity=Online_Retail['Quantity'].var()
print("Variance of Quantity", var_Quantity)
std_Quantity=Online_Retail['Quantity'].std() 32
print("S.D of Quantity", std_Quantity)
[Link]
Percentiles
Percentiles
•A student attended an exam along with 1000 others.
• He got 68% marks? How good or bad he performed in the exam?
• What will be his rank overall?
• What will be his rank if there were 100 students overall?
•For example, with 68 marks, he stood at 90th position. There are 910
students who got less than 68, only 89 students got more marks than
him
•He is standing at 91 percentile.
•Instead of stating 68 marks, 91% gives a good idea on his performance
•Percentiles make the data easy to read
34
[Link]
Percentiles
•pth percentile: p percent of observations below it, (100 - p)% above it.
•Marks are 40 but percentile is 80%, what does this mean?
•80% of CAT exam percentile means
• 20% are above & 80% are below
•Percentiles help us in getting an idea on outliers.
•For example the highest income value is 400,000 but 95th percentile is
20,000 only. That means 95% of the values are less than 20,000. So the
values near 400,000 are clearly outliers
35
[Link]
Percentiles
Income['capital-gain'].quantile([0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1])
36
[Link]
Lab: Outlier detection
• Import “Give me some Credit\[Link]”
• Look at the percentiles of the variable monthly_utilization
• Are there any outliers?
38
[Link]
Code: Outlier detection
loans['monthly_utilization'].quantile([0, 0.1, 0.2, 0.3, 0
.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1])
Data Cleaning and Imputation
X1
11.0
11.1
11.9
10.9
10.8
.
11.5
11.6
11.6
11.4
11
12
11.8
11.4
11.9
41
[Link]
Missing Value Imputation
X1
•Standalone imputation 11.0
• Mean, median, other point estimates 11.1
• Convenient, easy to implement 11.9
• Assume: Distribution of the missing values is the same as the non- 10.9
missing values. 10.8
• Does not take into account inter-relationships .
11.5
11.6
• Eg: The average of available values is 11.4. Can we replace the
missing value in this table by 11.4 ? 11.6
11.4
11
12
11.8
11.4 42
11.9
[Link]
LAB: Outlier Treatment
•Perform imputation on monthly_utilization
Code: Outlier Treatment
loans['util_new']=loans['monthly_utilization']
loans['util_new'][loans['util_new']>1]=median_util
Model Building Life Cycle
Background Preparing
Data Building the Validating
and data for Deployment
Exploration model the model
Objective analysis
Business Select the right
Collect data Validate data Intime validation Deploy model
Objective model
Set Goals Explore data Outlier Variable Out of time Maintenance of
treatment selection validation model
Project Plan Basic Summary Missing value Model building
Model finetuning Model monitoring
treatment and finetuning
45
Budget & Identify outliers
Resources and missing Clean the data Model iterations
values 45
Prepare data for
Analysis
Conclusion
•In this session we discussed basics of python
•Basic statistics
•Basics of data exploration and cleaning