User Interface
Important Roles in ES Development
Numpy
Python Libraries for Data
Science Pandas:
▪ adds data structures and tools designed to work with table-like data (similar to Series and
Data Frames in R)
▪ provides tools for data manipulation: reshaping, merging, sorting, slicing, aggregation etc.
▪ allows handling missing data
34
Loading Python Libraries
In [ ]:
#Import Python Libraries
import numpy as np
import scipy as sp
import pandas as pd
import matplotlib as mpl
import seaborn as sns
Press Shift+Enter to execute the jupyter cell
35
Reading data using pandas
In [ ]:
#Read csv file
df = pd.read_csv("[Link]
36
Exploring data frames
In [3]:
#List first 5 records
[Link]()
Out[3]:
37
Hands-on exercises
✔ Try to read the first 10, 20, 50 records; ✔ Can
you guess how to view the last few records;
38
Data Frame data types
Pandas Type Native Python Type Description
object string The most general dtype. Will be assigned to your column if
column has mixed types
(numbers and strings).
int64 int Numeric characters. 64 refers to the memory allocated to hold
this character.
float64 float Numeric characters with decimals. If a column contains
numbers and NaNs(see below),
pandas will default to float64, in
case your missing value has a
decimal.
39
Data Frame data types
In [4]:
#Check a particular column type
df['salary'].dtype
Out[4]: dtype('int64')
In [5]: #Check types for all the columns
[Link]
sex
Out[4]: salary
rank dtype: object
discipline object object int64
phd int64 object int64
service
40
Data Frames attributes
Python objects have attributes and methods.
[Link] description
dtypes list the types of the columns
columns list the column names
axes list the row labels and column names ndim number of
dimensions
size number of elements
shape return a tuple representing the dimensionality values numpy
representation of the data
41
Hands-on exercises
✔ Find how many records this data frame has;
✔ How many elements are there?
✔ What are the column names?
✔ What types of columns we have in this data frame?
42
Data Frames methods
Unlike attributes, python methods have parenthesis.
All attributes and methods can be listed with a dir() function:
dir(df)[Link]() description
head( [n] ), tail( [n] ) first/last n rows
describe() generate descriptive statistics (for numeric columns only) max(), min()
return max/min values for all numeric columns mean(), median() return
mean/median values for all numeric columns std() standard deviation
sample([n]) returns a random sample of the data frame dropna() drop
all the records with missing values
43
Hands-on exercises
✔ Give the summary for the numeric columns in the dataset
✔ Calculate standard deviation for all numeric columns; ✔ What
are the mean values of the first 50 records in the dataset?
44
Selecting a column in a Data Frame
Method 1: Subset the data frame using column name:
df['sex']
Method 2: Use the column name as an attribute:
[Link]
45
Hands-on exercises
✔ Calculate the basic statistics for the salary column;
✔ Find how many values in the salary column (use count method);
✔ Calculate the average salary;
46
Data Frames groupby method
Using "group by" method we can:
• Split the data into groups based on some criteria
• Calculate statistics (or apply a function) to each group
• Similar to dplyr() function in R
In [ ]: #Group data using rank
df_rank = [Link]( ['rank'])
In [ ]: #Calculate mean value for each numeric column per each group
df_rank.mean()
47
Data Frames groupby method
Once groupby object is create we can calculate various statistics for each group:
In [ ]: #Calculate mean salary for each professor rank:
[Link]('rank')[['salary']].mean()
Note: If single brackets are used to specify the column (e.g. salary), then the output is Pandas Series
object. When double brackets are used the output is a Data Frame
48
Data Frames groupby method
groupby performance notes:
- no grouping/splitting occurs until it's needed. Creating the groupby object
only verifies that you have passed a valid mapping
- by default the group keys are sorted during the groupby operation. You may
want to pass sort=False for potential speedup:
In [ ]: #Calculate mean salary for each professor rank:
[Link](['rank'],
sort=False)[['salary']].mean()
49
Data Frame: filtering
To subset the data we can apply Boolean indexing. This indexing is commonly
known as a filter. For example if we want to subset the rows in which the salary
value is greater than $120K:
In [ ]: #Calculate mean salary for each professor rank:
df_sub = df[ df[ 'salary'] > 120000 ]
Any Boolean operator can be used to subset the data:
> greater; >= greater or equal;
< less; <= less or equal;
== equal; != not equal;
In [ ]: #Select only those rows that contain female professors:
df_f = df[ df[ 'sex'] == 'Female' ]
50
Data Frames: Slicing
There are a number of ways to subset the Data Frame:
• one or more columns
• one or more rows
• a subset of rows and columns
Rows and columns can be selected by their position or label
51
Data Frames: Slicing
When selecting one column, it is possible to use single set of brackets, but the
resulting object will be a Series (not a DataFrame):
In [ ]: #Select column salary:
df['salary']
When we need to select more than one column and/or make the output to be a
DataFrame, we should use double brackets:
In [ ]: #Select column salary:
df[['rank','salary']]
52
Data Frames: Selecting rows
If we need to select a range of rows, we can specify the range using ":"
In [ ]: #Select rows by their position:
df[10:20]
Notice that the first row has a position 0, and the last value in the range is
omitted:
So for 0:10 range the first 10 rows are returned with the positions starting with 0
and ending with 9
53
Data Frames: method loc
If we need to select a range of rows, using their labels we can use method loc:
In [ ]: #Select rows by their labels:
df_sub.loc[10:20,['rank','sex','salary' ]]
Out[ ]:
54
Data Frames: method iloc
If we need to select a range of rows and/or columns, using their positions we
can use method iloc:
In [ ]: #Select rows by their labels:
df_sub.iloc[10:20,[0, 3, 4, 5]]
Out[ ]:
55
Data Frames: method iloc (summary)
[Link][0] # First row of a data frame
[Link][i] #(i+1)th row
[Link][-1] # Last row
[Link][:, 0] # First column
[Link][:, -1] # Last column
[Link][0:7] #First 7 rows
[Link][:, 0:2] #First 2 columns
[Link][1:3, 0:2] #Second through third rows and first 2 columns
[Link][[0,5], [1,3]] #1st and 6th rows and 2nd and 4th columns
56
Data Frames: Sorting
We can sort the data by a value in the column. By default the sorting will occur in
ascending order and a new data frame is return.
In [ ]: # Create a new data frame from the original sorted by the column
Salary
df_sorted = df .sort_values( by ='service')
df_sorted.head()
Out[ ]:
57
Data Frames: Sorting
We can sort the data using 2 or more columns:
In [ ]: df_sorted = df.sort_values( by =['service', 'salary'], ascending =
[True, False]) df_sorted.head( 10)
Out[ ]:
58
Missing Values
Missing values are marked as NaN
In [ ]: # Read a dataset with missing values
flights = pd.read_csv("[Link]
In [ ]: # Select the rows that have at least one missing value
flights[[Link]().any(axis=1)].head()
Out[ ]:
59
Missing Values
There are a number of methods to deal with missing values in the data frame:
[Link]() description
dropna() Drop missing observations
dropna(how='all') Drop observations where all cells is NA
missing
dropna(axis=1, how='all')
Drop column if all the values are
dropna(thresh = 5) Drop rows that contain less than 5 non-missing values
fillna(0) Replace missing values with zeros
isnull() returns True if the value is missing
notnull() Returns True for non-missing values
60
Missing Values
• When summing the data, missing values will be treated as zero
• If all values are missing, the sum will be equal to NaN
• cumsum() and cumprod() methods ignore missing values but preserve them in
the resulting arrays
• Missing values in GroupBy method are excluded (just like in R) • Many
descriptive statistics methods have skipna option to control if missing data
should be excluded . This value is set to True by default (unlike R)
61
Aggregation Functions in Pandas
Aggregation - computing a summary statistic about each group, i.e.
• compute group sums or means
• compute group sizes/counts
Common aggregation functions:
min, max
count, sum, prod
mean, median, mode, mad
std, var
62
Aggregation Functions in Pandas
agg() method are useful when multiple statistics are computed per column:
In [ ]: flights[['dep_delay','arr_delay']].agg(['min','mean','max'])
Out[ ]:
63
Basic Descriptive Statistics
[Link]() description
describe Basic statistics (count, mean, std, min, quantiles, max) min, max
Minimum and maximum values
mean, median, mode Arithmetic average, median and mode var,
std Variance and standard deviation
sem Standard error of mean
skew Sample skewness
kurt kurtosis
64
TENSORFLOW
What is TensorFlow?
● TensorFlow is an open-source end-to-end platform for creating Machine Learning
applications.
● It is a symbolic math library that uses dataflow and differentiable programming to
perform various tasks focused on training and inference of deep neural networks. ● It
allows developers to create machine learning applications using various tools,
libraries, and community resources.
● Currently, the most famous deep learning library in the world is Google’s TensorFlow.
● Google product uses machine learning in all of its products to improve the search
engine, translation, image captioning or recommendations.
Tensorflow Components
1. Tensor
2. Graphs
Tensor Flow Algorithms
● Linear regression: [Link]
● Classification:[Link]
● Deep learning classification: [Link]
● Deep learning wipe and deep:
[Link]
● Booster tree regression: [Link] ●
Boosted tree classification: [Link]
Tensorflow example
Tensorflow to multiply two numbers
we will multiply X_1 and X_2 together. Tensorflow will create a node to
connect the operation. In our example, it is called multiply. When the graph
is determined, Tensorflow computational engines will multiply together X_1
and X_2.