Python for Data Science

1
 Agenda
Modules Topics
Module 1 DataScience Overview
Data Analytics Overview
Basics of Statistics
▪ Aggregation
▪ Variation and SD
▪ Percentiles and movements
▪ Co-variance and Co-relation
Module 2 Data Processing
▪ Procuring data
▪ Data cleaning
▪ Missing data
▪ Hypotheses

2
 Agenda
Modules Topics
Module 3 Fast Data Processing with
NumPy
▪ Introduction to NumPy
▪ Working with Arrays
▪ Array Indexing Vectors and matrices
▪ Understanding Dot product
▪ Matrix Related Operations
▪ Linear Algebra using NumPy
▪ Examples and Lab Exercises
Module 4 Introduction to Pandas
▪ Understanding Data input and
output
▪ Working with DataFrames
▪ Apply, joins, groupBy, Merging
▪ Series Data handling using Pandas
▪ Other Operations
▪ Using pandas Data Frames to solve
complex tasks
▪ Use pandas to handle Excel Files
▪ Examples and Lab Exercises

3
 Agenda
Modules Topics
Module 5 Plotting libraries
▪ Pandas Built-in Data Visualisation
▪ Introduction to Matplotlib
▪ Line Chart, Scatter plot, histogram
▪ Images
▪ Sales Data Analysis
▪ Aggregate analysis
▪ Answering different hypothesis

4
 Data Science Overview

21 August 2015 5
1. Data Science Overview

• Key Objectives
– Understand what is Data Science
– Data Scientists
– Examples of Data Science
– Python for Data Science
Data Science
• Is the umbrella of techniques where you are trying to
extract information and insights from data. This includes
MIS reporting on the lowest level to building predictive
models on the higher level.
What do I need to know
Data Scientist

Data scientist is someone

who knows more statistics
than a computer scientist
and more computer
science than a
statistician.
Knowledge base of a Data Scientist
Data
DataScience Examples
Science Metro Map
Python for Data Science
Data Analytics Overview

15
Key Objectives
• Processes in Data Science
• Data Wrangling, Data Exploration, and Model Selection
• Exploratory Data Analysis or EDA
• Data Visualization
• Plotting
• Hypothesis Building and Testing
Process in Data Science
Data science process flowchart
Data munging or wrangling
• Data munging or
data wrangling is
loosely the process
of manually
converting or
mapping data from
one "raw" form into
another format
that allows for
more convenient
consumption of the
data with the help
of semi-automated
tools
Data exploration
• Data exploration is the first step in data analysis and
typically involves summarizing the main characteristics
of a dataset.
• It is commonly conducted using visual analytics tools,
but can also be done using python or if required using a
statistical software, such as R
Model selection
• Model selection is the task of
selecting a statistical model from
a set of candidate models, given
data.
• First collect the data which is well
suited to the problem you are
trying to address.
• Then select a candidate models of
similar predictive or explanatory
power, the simplest model is most
likely to be the best choice.
Choosing a candidate model
Methods for choosing the set of candidate models
1. Exploratory data analysis
2. Scientific method
Exploratory data analysis (EDA)
• In statistics, exploratory data analysis (EDA) is an approach to
analyzing data sets to summarize their main characteristics,
often with visual methods.
• EDA is for seeing what the data can tell us beyond the formal
modeling or hypothesis testing task.
• Exploratory data analysis was promoted by John Tukey to
encourage statisticians to explore the data, and possibly
formulate hypotheses that could lead to new data collection and
experiments.
• EDA is different from initial data analysis (IDA).
• IDA focuses more narrowly on checking assumptions required for
model fitting and hypothesis testing, and handling missing values
and making transformations of variables as needed.
• EDA encompasses IDA
EDA Techniques
• Box plot
• Histogram
• Multi-vari chart
• Run chart
• Pareto chart
• Scatter plot
• Stem-and-leaf plot
• Parallel coordinates
• Odds ratio
• Multidimensional scaling
• Targeted projection pursuit
• Principal component analysis
• Multilinear PCA
• Projection methods such as grand tour,
guided tour and manual tour
• Interactive versions of these plots

[Link]
Scientific method
Data Visualization - Why Visualize?
What is a hypothesis test?
• A hypothesis test is a statistical test that is used to determine
whether there is enough evidence in a sample of data to infer
that a certain condition is true for the entire population.
• A hypothesis test examines two opposing hypotheses about a
population: the null hypothesis and the alternative
hypothesis. The null hypothesis is the statement being
tested. Usually the null hypothesis is a statement of "no
effect" or "no difference". The alternative hypothesis is the
statement you want to be able to conclude is true.
• Based on the sample data, the test determines whether to
reject the null hypothesis. You use a p-value, to make the
determination. If the p-value is less than or equal to the level
of significance, which is a cut-off point that you define, then
you can reject the null hypothesis.
What is a hypothesis test? Cont.
• A common misconception is that statistical hypothesis
tests are designed to select the more likely of two
hypotheses. Instead, a test will remain with the null
hypothesis until there is enough evidence (data) to
support the alternative hypothesis.
– Examples of questions you can answer with a
hypothesis test include:
• Does the mean height of undergraduate women
differ from 66 inches?
• Is the standard deviation of their height equal less
than 5 inches?
• Do male and female undergraduates differ in
height?
Statistical Analysis and
Business Applications
Key Objectives
• Introduction to Statistics
• Statistical Analysis
• Some Common Terms Used in Statistics
• Data Distribution: Central Tendency, Percentiles,
Dispersion
• Histogram
• Bell Curve
• Hypothesis Testing
• Chi-Square Test
• Correlation Matrix
• Inferential Statistics
Statistics
• What is Stats?
• Statistics is a mathematical body of science that
pertains to the collection, analysis, interpretation or
explanation, and presentation of data.
• Statistics to be a distinct mathematical science rather
than a branch of mathematics.

• Two Key Terms

1. Descriptive statistics
2. Inferential Statistics
Descriptive Statistics
• Descriptive statistics is used to summarize the
population data.
– We use Numerical descriptors like mean and standard
deviation for continuous data types (like income)
– While frequency and percentage are more useful in terms of
describing categorical data (like product name).
Inferential Statistics
• Inferential statistics is used for drawing meaningful
conclusions about the entire population
• We use patterns in the sample data to draw inferences about
the population that the sample represents, accounting for
randomness.
• Inferences may take the form of:
– answering yes/no questions about the data (hypothesis
testing),
– estimating numerical characteristics of the data
(estimation), describing associations within the data
(correlation) and modeling relationships within the data
(for example, using regression analysis).
• Inference can extend to forecasting, prediction and
estimation of unobserved values either in or associated with
the population being studied.
• Inference can include extrapolation and interpolation of
time series or spatial data, and can also include data mining
Statistical Analysis
• Statistical analysis:
1. Describe the nature of the data to be analyzed.
2. Explore the relation of the data to the underlying
population.
3. Create a model to summarize understanding of how the
data relates to the underlying population.
4. Prove (or disprove) the validity of the model.
5. Employ predictive analytics to run scenarios that will
help guide future actions.

The goal of statistical analysis is to identify trends. A retail

business, for example, might use statistical analysis to find
patterns in unstructured and semi-structured customer
data that can be used to create a more positive customer
experience and increase sales.
Types of Data

• Continuous
– height, age, time
• Discrete
– # of days worked this week, # leaves on a tree
• Ordinal
– {Good, O.K., Bad}
• Nominal
– {Yes/No}, {Teacher/Chemist/Haberdasher}
Some Common Terms Used in Statistics
• Population: is a set of entities concerning which
statistical inferences are to be drawn.
• Sample: a number of independent observations from the
same probability distribution
• Parameter: the distribution of a random variable as
belonging to a family of probability distributions,
distinguished from each other by the values of a finite
number of parameters
• Bias: a factor that causes a statistical sample of a
population to have some examples of the population
less represented than others.
Outliers (and their treatment)
Outliers (and their treatment)

• An "outlier" is an observation that does not fit

the pattern in the rest of the data

– Check the data

– Check with the measurer
– If reason to believe it is NOT real, change it if possible,
otherwise leave it out (but note).
– If reason to believe it is real, leave it out and note.
The Mean
• The Mean (Arithmetic)
• The mean is defined as the sum of all the elements,
divided by the number of elements.
• The statistical mean of a set of observations is the
average of the measurements in a set of data
The Mode
• The mode is defined as the most frequently
element in a set of elements.
– For example [1, 3, 6, 6, 6, 6, 7, 7, 12, 12, 17] has a mode of 6.

• Given the list of data [1, 1, 2, 4, 4] the mode is

not unique - the dataset may be said to be
bimodal, while a set with more than two modes
may be described as multimodal.
The Median
• The median is defined as the middle element,
or the value separating the higher half of a
sample from the lower half.
• If there is an even number of elements, it is half
the sum of the middle two elements.
• Given the list of data [1, 1, 2, 4, 4] the median
is 2.
The Variance
• But there can be a lot of variance in individual
elements,
e.g. teacher salaries
Average = €22,000
Lowest = € 12,000
Difference = 12,000 - 22,000 = -10,000
The Variance
The Variance
The Variance
The Variance
• Sum of (Sample - Average) = 0, thus we need to
define variance.

• The variance of a set of data is a cumulative

measure of the squares of the difference of all
the data values from the mean divided by
sample size minus one.
Standard Deviation
• The standard deviation of a set of data is the
positive square root of the variance.

-1

-1
Question 2
• Find the mean and variance of the following sample
values :

36, 41, 43, 44, 46

Question 2

• Mean:

• =(36 + 41 + 43 + 44 + 46) / 5

• =210 / 5

• =42
Question 2

Variance

Difference Square Variance

36 – 42 = -6 36 = 58 / (5 -1)
41 – 42 = -1 1 = 58 / 4
43 – 42 = 1 1 = 14.5
44 – 42 = 2 4
46 – 42 = 4 16 Standard Deviation
------------------------------------ = SquareRoot(14.5)
58 = 3.8
Choosing a statistical test
Choosing a statistical test
CORRELATION ANALYSIS

It is a statistical measure which

shows relationship between two or
more variable moving in the same or
in opposite direction
Correlation Analysis

• The sample correlation coefficient (r) measures the

degree of linearity in the relationship between X and
Y.
-1 < r < +1

Strong negative relationship Strong positive relationship

• r = 0 indicates no linear relationship

• In Excel, use =CORREL(array1,array2),
where array1 is the range for X and array2 is the
range for Y.
Types of correlation

correlation

Simple ,
positive Linear
multiple
& negative & non-linear
& partial
Methods of correlation
• Scatter diagram
• Product moment or covariance
• Rank correlation
• Concurrent deviation
Scatter diagram
• Perfectly +ve
Less-degree +ve

Weak Positive
Correlation
High degree +ve

Strong Positive
Correlation
Perfectly -ve
High degree -ve
Strong Negative Correlation
Less degree -ve

Weak Negative
Correlation
Zero degree
 Mathematical
Computing with
Python (NumPy)
Key Objectives
• NumPy Overview
• Properties, Purpose, and Types of ndarray
• Class and Attributes of ndarray Object
• Basic Operations: Concept and Examples
• Accessing Array Elements: Indexing, Slicing, Iteration,
Indexing with Boolean Arrays • Copy and Views
• Universal Functions (ufunc)
• Shape Manipulation
• Broadcasting
• Linear Algebra
What is Numpy
• NumPy is a Python C extension library for array-oriented
computing:
1. Efficient
2. In-memory
3. Contiguous (or Strided)
4. Homogeneous (but types can be algebraic)
NumPy is suited to many applications
• Image processing
• Signal processing
• Linear algebra
• A plethora of others

NumPy is the foundation of the python

scientific stack
Numpy Ecosystem
Quick Start
Array Shape

• One dimensional arrays have a 1-tuple

for their shape
• ...Two dimensional arrays have a 2-tuple
• ...And so on
Array Element Type (dtype)
• NumPy arrays comprise elements of a single data type
• The type object is accessible through the .dtype
attribute

Few of the most important attributes of dtype objects

• [Link] — big or little endian
• [Link] — element size of this dtype
• [Link] — a name for this dtype object
• [Link] — type object used to create scalars

There are many others...

• Array dtypes are usually inferred automatically
• But can also be specified explicitly
NumPy Built-in dtype Hierarchy

np.datetime64 is a new addition in NumPy 1.7

Array Creation
• Explicitly from a list of values

• As a range of values

• By specifying the number of elements

• Zero-initialized

• One-initialized

• Uninitialized
• Constant diagonal value

• Multiple diagonal values

Array Memory Layout
Indexing and Slicing
NumPy array indices can also take an optional stride
Array Views
• Simple assigments do not make copies of arrays (same
semantics as Python). Slicing operations do not make
copies either; they return views on the original array.

• Array views contain a pointer to the original data, but

may have different shape or stride values. Views always
have [Link] equal to False.
Universal Functions (ufuncs)
• NumPy ufuncs are functions that operate element-wise
on one or more arrays

• ufuncs dispatch to optimized C inner-loops based on

array dtype
NumPy has many built-in ufuncs
• comparison: <=, ==, !=, >=, >
• arithmetic: +, -, *, /, reciprocal, square
• exponential: exp, expm1, exp2, log, log10, log1p, log2,
power, sqrt
• trigonometric: sin, cos, tan, acsin, arccos, atctan
• hyperbolic: sinh, cosh, tanh, acsinh, arccosh, atctanh
• bitwise operations: &, |, ~, ^, left_shift, right_shift
• logical operations: and, logical_xor, not, or
• predicates: isfinite, isinf, isnan, signbit
• other: abs, ceil, floor, mod, modf, round, sinc, sign,
trunc
Axis
• Array method reductions take an optional axis
parameter that specifies over which axes to reduce
axis=None reduces into a single scalar

axis =None is the default

• axis=0 reduces into the zeroth dimension
• axis=0 reduces into the first dimension
Broadcasting
• A key feature of NumPy is broadcasting, where arrays
with different, but compatible shapes can be used as
arguments to ufuncs

In this case an array scalar is broadcast to an array with shape

(5, )
• A slightly more involved broadcasting example in two
dimensions

Here an array of shape (3, 1) is broadcast to an array with shape (3,

2)
Broadcasting Rules
• n order for an operation to broadcast, the size of all the
trailing dimensions for both arrays must either:
• be equal OR be one
Square Peg in a Round Hole
• If the dimensions do not match up, [Link] may be
useful
Array Methods
• Predicates
– [Link](), [Link]()
• Reductions
– [Link](), [Link](), [Link](), [Link](), [Link](), [Link]()
• Manipulation
– [Link](), [Link](), [Link](...), [Link](), [Link](...),
[Link](...)
• Complex Numbers
– [Link], [Link], [Link]()
Fancy Indexing
• NumPy arrays may be used to index into other arrays
• Boolean arrays can also be used as indices into other
arrays
NumPy Functions
• Data I/O
– fromfile, genfromtxt, load, loadtxt, save, savetxt
• Mesh Creation
– mgrid, meshgrid, ogrid
• Manipulation
– einsum, hstack, take, vstack
Array Subclasses
• [Link] — Masked arrays
• [Link] — Matrix operators
• [Link] — Memory-mapped arrays
• [Link] — Record arrays
Other Subpackages
• [Link] — Fast Fourier transforms
• [Link] — Efficient polynomials
• [Link] — Linear algebra
– cholesky, det, eig, eigvals, inv, lstsq, norm, qr, svd
• [Link] — C standard library math functions
• [Link] — Random number generation
– beta, gamma, geometric, hypergeometric, lognormal, normal, poisson,
uniform, weibull
FFT
Data Manipulation with
Python (Pandas)

101
Key Objectives
• Introduction to Pandas
• Data Structures
• Series
• DataFrame
• Missing Values
• Data Operations
• Data Standardization
• Pandas File Read and Write Support
• SQL Operation
Structured Data
Structured Data
• A familiar data model
– Heterogeneous columns or hyperslabs
– Each column/hyperslab is homogeneously typed
– Relational databases (SQL, etc.) are just a special
case
• Need good performance in row- and column-oriented
operations
• Support for axis metadata
• Data alignment is critical
• Seamless integration with Python data structures and
NumPy
Structured Data Challenges
• Table modification: column insertion/deletion
• Axis indexing and data alignment
• Aggregation and transformation by group (“group by”)
• Missing data handling
• Pivoting and reshaping
• Merging and joining
• Time series-specific manipulations
• Fast IO: flat files, databases, HDF5, ...
Not all fun and games
• We care nearly equally about
– Performance
– Ease-of-use (syntax / API fits your mental model)
– Expressiveness
• Clean, consistent API design is hard and
underappreciated
The big picture
• Build a foundation for data analysis and statistical
computing
• Craft the most expressive / flexible in-memory data
manipulation tool in any language
– Preferably also one of the fastest, too
• Vastly simplify the data preparation, munging, and
integration process
• Comfortable abstractions: master data-fu without
needing to be a computer scientist
• Later: extend API with distributed computing backend
for larger-than-memory datasets
Pandas: a brief history
• Starting building April 2008 back at AQR
• Open-sourced (BSD license) mid-2009
• 29075 lines of Python/Cython code as of yesterday, and
growing fast
• Heavily tested, being used by many companies (inc. lots
of financial firms) in production
Series
Dataframe
Dataframe
Dataframe
Axis Indexing
• Key pandas feature
• The axis index is a data structure itself, which can be
customized to support things like:
– 1-1 O(1) indexing with hashable Python objects
– Datetime indexing for time series data
– Hierarchical (multi-level) indexing
• Use Python dict to support O(1) lookups and O(n)
realignment ops. Can specialize to get better
performance and memory usag
Axis Indexing
• Every axis has an index
• Automatic alignment between differently-indexed
objects: makes it nearly impossible to accidentally
combine misaligned data
• Hierarchical indexing provides an intuitive way of
structuring and working with higher-dimensional data
• Natural way of expressing “group by” and join-type
operations
• As good or in many cases much more integrated/flexible
than commercial or open-source alternatives to pandas/
Python
The trouble with Python dicts...
• Python dict memory footprint can be quite large
– 1MM key-value pairs: something like 70mb on a 64-bit
system
– Even though sizeof(PyObject*) == 8
• Python dict is great, but should use a faster, threadsafe
hash table for primitive C types (like 64-bit integer)
• BUT: using a hash table only necessary in the general
case. With monotonic indexes you don’t need one for
realignment ops
Some alignment numbers
• Hardware: Macbook Pro Core i7 laptop, Python 2.7.2
• Outer-join 500k-length indexes chosen from 1MM
elements
– Dict-based with random strings: 2.2 seconds
– Sorted strings: 400ms (5.5x faster)
– Sorted int64: 19ms (115x faster)
• Fortunately, time series data falls into this last category
• Alignment ops with C primitives could be fairly easily
parallelized with OpenMP in Cython
DataFrame, the pandas workhorse
• A 2D tabular data structure with row and column
indexes
• Hierarchical indexing one way to support higher-
dimensional data in a lower-dimensional structure
• Simplified NumPy type system: float, int, boolean,
object
• Rich indexing operations, SQL-like join/merges, etc.
• Support heterogeneous columns WITHOUT sacrificing
performance in the homogeneous (e.g. floating point
only) case
DataFrame, under the hood
Supporting size mutability
• In order to have good row-oriented performance, need
to store like-typed columns in a single ndarray
• “Column” insertion: accumulate 1 × N × . . .
homogeneous columns, later consolidate with other like-
typed into a single block
• I.e. avoid reallocate-copy or array concatenation steps
as long as possible
• Column deletions can be no-copy events (since ndarrays
support views)
Hierarchical Indexing
• Natural result of multi-key GroupBy
• An intuitive way to work with higher-dimensional data
• Much less ad hoc way of expressing reshaping operations
• Once you have it, things like Excel-style pivot tables
just “fall out”
Reshaping
[Link](’agefrom’).stack(’year’)
Reshaping implementation nuances
• Must deal with unbalanced group sizes / missing data
• Play vectorization tricks with the NumPy C-contiguous
memory layout: no Python for loops allowed
• Care must be taken to handle heterogeneous and
homogeneous data case
GroupBy
• High level process
– split data set into
groups
– apply function to
each group (an
aggregation or a
transformation)
– combine results
intelligently into a
result data structure
• Can be used to emulate
SQL GROUP BY
operations
GroupBy
• Grouping closely related to indexing
• Create correspondence between axis labels and group
labels using one of:
– Array of group labels (like a DataFrame column)
– Python function to be applied to each axis tick
• Can group by multiple keys
• For a hierarchically indexed axis, can select a level and
group by that (or some transformation thereof)
GroupBy implementation challenges
• Computing the group labels from arbitrary Python objects is very
expensive
– 77ms for 1MM strings with 1K groups
– 107ms for 1MM strings with 10K groups
– 350ms for 1MM strings with 100K groups
• To sort or not to sort (for iteration)?
– Once you have the labels, can reorder the data set in O(n) (with
a much smaller constant than computing the labels)
– Roughly 35ms to reorder 1MM float64 data points given the
labels
• (By contrast, computing the mean of 1MM elements takes 1.4ms)
• Python function call overhead is significant in cases with lots of
small groups; much better (orders of magnitude speedup) to write
specialized Cython routines
 Data Visualization in
Python using Matplotlib

126
Key Objectives
• Introduction to Data Visualization
• Python Libraries
• Plots
• Matplotlib Features:
– Line Properties Plot with (x, y)
– Controlling Line Patterns and Colors
– Set Axis, Labels, and Legend Properties
– Alpha and Annotation
– Multiple Plots
– Subplots
• Types of Plots and Seaborn
Why Visualize
Tiny dataset
Effective Visualization
Python Visualization Libraries
• Matplotlib
• Seaborn
• ggplot
• Bokeh
• pygal
• Plotly
• geoplotlib
• Gleam
• missingno
• Leather
Plotting with Matplotlib
Matplotlib:
• Python based plotting library offers matplotlib with a
complete 2D support along with limited 3D graphic
support. It is useful in producing publication quality
figures in interactive environment across platforms. It
can also be used for animations as well.

• [Link]
Simple Plot
Histogram
BoxPlot
Image Plot
WirePlot
• matplotlib toolkits extend funtionality for other kinds of
visualization
Scatter Plot
Seaborn makes plot pretty
Scatter Plot
• Adding multiple lines and a legend
Summary and Group Activities