Data Science & NumPy Basics

By Dr. P. Sita Rama Murty

 ● What is Data science?
● Datafication
● Exploratory Data Analysis
● The Data science process

UNIT IV ●
●
A data scientist role in this process
NumPy Basics:
○ The NumPy ndarray:
TOPICS ■ A Multidimensional Array
Object
■ Creating ndarrays
■ Data Types for ndarrays
 ● Operations between Arrays and Scalars
● Basic Indexing and Slicing
● Boolean Indexing

UNIT IV ●
●
●
Fancy Indexing
Data Processing Using Arrays
Expressing Conditional Logic as Array
Operations
TOPICS ● Methods for Boolean Arrays
● Sorting
● Unique
What is Data Science?
Data science is the domain of study that deals with vast volumes of
data using modern tools and techniques to find unseen patterns,
derive meaningful information, and make business decisions.
The data used for analysis can come from many different sources
and presented in various formats (noisy, structured, unstructured).
Datafication :
Datafication is defined as a process of “taking all aspects of life and
turning them into data.”
Once we datafy things, we can transform their purpose and turn
the information into new forms of value.
Example,
Google’s augmented-reality glasses datafy the gaze.
Twitter datafies stray thoughts.
LinkedIn datafies professional networks.
Exploratory Data Analysis
Exploratory Data Analysis (EDA) is the first step of building a
model.
EDA is used to understand the data, gain intuition, understand the
shape of it, and try to connect your understanding of the process
that generated the data to the data itself. EDA happens between
you and the data and isn’t about proving anything to anyone else
yet.
There are important reasons anyone working with data should do
EDA. Namely, to gain intuition about the data; to make
comparisons between distributions; for sanity checking (making
sure the data is on the scale you expect, in the format you thought
it should be); to find out where data is missing or if there are
outliers; and to summarize the data.
The basic tools of EDA are plots, graphs and summary statistics.
● It’s a method of systematically going through the data,
● plotting distributions of all variables (using box plots),
● plotting time series of data,
● transforming variables,
● looking at all pairwise relationships between variables using scatterplot
matrices,
● and generating summary statistics for all of them.
● At the very least that would mean computing their mean, minimum,
maximum, the upper and lower quartiles, and identifying outliers.
Although there’s lots of visualization involved in EDA, we
distinguish between EDA and data visualization in that EDA is
done toward the beginning of analysis, and data visualization is
done toward the end to communicate one’s findings. With EDA, the
graphics are solely done for you to understand what’s going on.
The Data Science Process
First we have the Real World. Inside the Real World are lots of
people busy at various activities. Say we have data on one of these
things.

Specifically, we’ll start with raw data—logs, Olympics records,

Enron employee emails, or recorded genetic material (note there
are lots of aspects to these activities already lost even when we
have that raw data).
We want to process this to make it clean for analysis. So we build
and use pipelines of data munging: joining, scraping, wrangling, or
whatever you want to call it. To do this we use tools such as
Python, shell scripts, R, or SQL, or all of the above.
Eventually we get the data down to a nice format, like something
with columns:
name | event | year | gender | event time

This is where you typically start in a standard statistics class, with

a clean, orderly dataset. But it’s not where you typically start in
the real world.
Once we have this clean dataset, we should be doing some kind of
EDA. In the course of doing EDA, we may realize that it isn’t
actually clean because of duplicates, missing values, absurd
outliers, and data that wasn’t actually logged or incorrectly logged.
If that’s the case, we may have to go back to collect more data, or
spend more time cleaning the dataset.
Next, we design our model to using some algorithm.

The model we choose depends on the type of problem we’re trying

to solve, of course, which could be a classification problem, a
prediction problem, or a basic description problem.

We then can interpret, visualize, report, or communicate our

results. This could take the form of reporting the results up to our
boss or coworkers, or publishing a paper in a journal and going out
and giving academic talks about it.
Alternatively, our goal may be to build or prototype a “data
product”; e.g., a spam classifier, or a search ranking algorithm, or a
recommendation system.

Now the key here that makes data science special and distinct from
statistics is that this data product then gets incorporated back into
the real world, and users interact with that product, and that
generates more data, which creates a feedback loop.
Data Scientists Role
Someone has to make the decisions about what data to collect, and
why.

That person needs to be formulating questions and hypotheses and

making a plan for how the problem will be attacked. And that
someone is the data scientist or our beloved data science team.

Data scientist needs to be involved in this process throughout,

meaning they are involved in the actual coding as well as in the
higher-level process, as shown in Figure.
We can think of the data science process as an extension of or
variation of the scientific method:
• Ask a question.
• Do background research.
• Construct a hypothesis.
• Test your hypothesis by doing an experiment.
• Analyze your data and draw a conclusion.
• Communicate your results.
In both the data science process and the scientific method, not
every problem requires one to go through all the steps, but almost
all problems can be solved with some combination of the stages.

For example, if your end goal is a data visualization (which itself

could be thought of as a data product), it’s possible you might not
do any machine learning or statistical modeling, but you’d want to
get all the way to a clean dataset, do some exploratory analysis,
and then create the visualization.
 ○ The NumPy ndarray:

NumPy ■ A Multidimensional Array

Object
■ Creating ndarrays
Basics ■ Data Types for ndarrays
NumPy
1. NumPy (Numerical Python), is one of the most important foundational
packages for numerical computing in Python.
2. One of the reasons NumPy is so important for numerical computations in
Python is because it is designed for efficiency on large arrays of data. There
are a number of reasons for this:
a. NumPy internally stores data in a contiguous block of memory,
independent of other built-in Python objects. NumPy’s library of
algorithms written in the C language can operate on this memory
without any type checking or other overhead.
b. NumPy arrays also use much less memory than built-in Python
sequences.
NumPy
c. NumPy operations perform complex computations on entire arrays
without the need for Python for loops.

Here are some of the things you’ll find in NumPy:

• ndarray, an efficient multidimensional array providing fast array-oriented

arithmetic operations and flexible broadcasting capabilities.

• Mathematical functions for fast operations on entire arrays of data without

having to write loops.
NumPy
• Tools for reading/writing array data to disk and working with
memory-mapped files.

• Linear algebra, random number generation, and Fourier transform

capabilities.

• A C API for connecting NumPy with libraries written in C, C++, or

FORTRAN.
● NumPy’s array class is called ndarray.
● ndarray is a fast, flexible container for large datasets in
Python.
● An ndarray is a generic multidimensional container for
homogeneous data; that is, all of the elements must be the same
type.
● Note: [Link] is not the same as the Standard Python
Library class [Link], which only handles one-dimensional
arrays and offers less functionality.
The more important attributes of an ndarray object are:

1. [Link]
● The number of axes (dimensions) of the array.

2. [Link]
● The dimensions of the array.
● This is a tuple of integers indicating the size of the array in
each dimension.
● For a matrix with n rows and m columns, shape will be (n,m).
3. [Link]
● the total number of elements of the array.
● This is equal to the product of the elements of shape.

4. [Link]
● an object describing the type of the elements in the array.
● One can create or specify dtype’s using standard Python types.
● Additionally NumPy provides types of its own. numpy.int32,
numpy.int16, and numpy.float64 are some examples.
5. [Link]
● the size in bytes of each element of the array.
● For example, an array of elements of type float64 has itemsize
8, while one of type complex32 has itemsize 4.
● It is equivalent to [Link].

6. [Link]
● the buffer containing the actual elements of the array.
● Normally, we won’t need to use this attribute because we will
access the elements in an array using indexing facilities.
 ○ The NumPy ndarray:

NumPy ■ A Multidimensional Array

Object
■ Creating ndarrays
Basics ■ Data Types for ndarrays
Creating ndarrays:

There are several ways to create arrays.

you can create an array from a regular Python list or tuple using
the array function. The type of the resulting array is deduced from
the type of the elements in the sequences.
A frequent error consists in calling array with multiple arguments,
rather than providing a single sequence as an argument.
array transforms sequences of sequences into two-dimensional
arrays, sequences of sequences of sequences into three-dimensional
arrays, and so on.
The type of the array can also be explicitly specified at creation
time:
Creating arrays with initial placeholders
Often, the elements of an array are The function zeros creates an array
originally unknown, but its size is full of zeros, the function ones
known. Hence, NumPy offers several creates an array full of ones, and the
functions to create arrays with initial function empty creates an array
placeholder content. whose initial content is random and
depends on the state of the memory.
By default, the dtype of the created
array is float64, but it can be
specified via the keyword argument
dtype.
Creating arrays with sequence of numbers
To create sequences of numbers, When arange is used with floating
NumPy provides the arange function point arguments, it is generally not
which is analogous to the Python possible to predict the number of
built-in range, but returns an array. elements obtained, due to the finite
floating point precision. For this
reason, it is usually better to use the
function linspace that receives as an
argument the number of elements
that we want, instead of the step.
 ○ The NumPy ndarray:

NumPy ■ A Multidimensional Array

Object
■ Creating ndarrays
Basics ■ Data Types for ndarrays
D
A
T
A

T
Y
P
E
S
D
A
T
A

T
Y
P
E
S
 ● Operations between Arrays and Scalars
● Basic Indexing and Slicing
● Boolean Indexing

UNIT IV ●
●
●
Fancy Indexing
Data Processing Using Arrays
Expressing Conditional Logic as Array
Operations
TOPICS ● Methods for Boolean Arrays
● Sorting
● Unique
Operations between arrays and scalars:
Arithmetic operations with scalars propagate the scalar argument
to each element in the array.
Operations between arrays and scalars:
Arrays are important because they enable you to express batch
operations on data without writing any for loops.

NumPy users call this vectorization.

Any arithmetic operations between equal-size arrays applies the

operation element-wise.
 ● Operations between Arrays and Scalars
● Basic Indexing and Slicing
● Boolean Indexing

UNIT IV ●
●
●
Fancy Indexing
Data Processing Using Arrays
Expressing Conditional Logic as Array
Operations
TOPICS ● Methods for Boolean Arrays
● Sorting
● Unique
Basic Indexing and Slicing:
NumPy provides many ways in which you may want to select a
subset of your data or individual elements.
One-dimensional arrays are simple; they act similarly to Python
lists.
As you can see, if you assign a scalar value to a slice, as in arr[5:8]
= 12, the value is propagated (or broadcasted henceforth) to the
entire selection.

An important first distinction from Python’s built-in lists is that

array slices are views on the original array.

This means that the data is not copied, and any modifications to the
view will be reflected in the source array.
As NumPy has been designed to be able to work with very large
arrays, you could imagine performance and memory problems if
NumPy insisted on always copying data.

If you want a copy of a slice of an ndarray instead of a view, you

will need to explicitly copy the array—for example, arr[5:8].copy().
In a two-dimensional array, the elements at each index are no
longer scalars but rather one-dimensional arrays.

The individual elements can be accessed by a comma-separated list

of indices.
In multidimensional arrays, if you omit later indices, the returned
object will be a lower dimensional ndarray consisting of all the data
along the higher dimensions. So in the 2 × 2 × 3 array arr3d:
Indexing with Slices:
A slice, therefore, selects a range of elements along an axis. It can
be helpful to read the expression arr2d[:2] as “select the first two
rows of arr2d.”

You can pass multiple slices just like you can pass multiple indexes.

When slicing like this, you always obtain array views of the same
number of dimensions. By mixing integer indexes and slices, you
get lower dimensional slices.
 ● Operations between Arrays and Scalars
● Basic Indexing and Slicing
● Boolean Indexing

UNIT IV ●
●
●
Fancy Indexing
Data Processing Using Arrays
Expressing Conditional Logic as Array
Operations
TOPICS ● Methods for Boolean Arrays
● Sorting
● Unique
Boolean Indexing:
● Boolean array can be passed when indexing the array.
● The boolean array must be of the same length as the array axis
it’s indexing.
● You can even mix and match boolean arrays with slices or
integers.
● The ~ operator can be useful when you want to invert a general
condition.
● Note : Boolean selection will not fail if the boolean array is not
the correct length, care is recommended when using this
feature.
● to combine multiple boolean conditions, use boolean arithmetic
operators like & (and) and | (or).
● Selecting data from an array by boolean indexing always
creates a copy of the data, even if the returned array is
unchanged.
● The Python keywords and and or do not work with boolean
arrays. Use & (and) and | (or) instead.
 ● Operations between Arrays and Scalars
● Basic Indexing and Slicing
● Boolean Indexing

UNIT IV ●
●
●
Fancy Indexing
Data Processing Using Arrays
Expressing Conditional Logic as Array
Operations
TOPICS ● Methods for Boolean Arrays
● Sorting
● Unique
Fancy Indexing:
● Fancy indexing is a term adopted by NumPy to describe
indexing using integer arrays.
● To select out a subset of the rows in a particular order, you can
simply pass a list or ndarray of integers specifying the desired
order.
● Using negative indices selects rows from the end.
● Passing multiple index arrays selects a one-dimensional array
of elements corresponding to each tuple of indices.
● Fancy indexing, unlike slicing, always copies the data into a
new array.
 ● Operations between Arrays and Scalars
● Basic Indexing and Slicing
● Boolean Indexing

UNIT IV ●
●
●
Fancy Indexing
Data Processing Using Arrays
Expressing Conditional Logic as Array
Operations
TOPICS ● Methods for Boolean Arrays
● Sorting
● Unique
Data Processing using Arrays:
● NumPy arrays enables us to express many kinds of data
processing tasks as concise array expressions that might
otherwise require writing loops.
● This practice of replacing explicit loops with array expressions
is commonly referred to as vectorization.
● In general, vectorized array operations will often be one or two
(or more) orders of magnitude faster than their pure Python
equivalents, with the biggest impact in any kind of numerical
computations.
 ● Operations between Arrays and Scalars
● Basic Indexing and Slicing
● Boolean Indexing

UNIT IV ●
●
●
Fancy Indexing
Data Processing Using Arrays
Expressing Conditional Logic as Array
Operations
TOPICS ● Methods for Boolean Arrays
● Sorting
● Unique
Expressing Conditional Logic as Array Operations:
● The [Link] function is a vectorized version of the
ternary expression x if condition else y.
● Example 1:
○ Suppose we had a boolean array and two arrays of values.
○ Suppose we wanted to take a value from xarr whenever the
corresponding value in cond is True, and otherwise take the
value from yarr.
○ With [Link] you can write this very concisely.
● The second and third arguments to [Link] don’t need to be
arrays; one or both of them can be scalars. A typical use of
[Link] in data analysis is to produce a new array of values
based on another array.
Expressing Conditional Logic as Array Operations:
● Example 2:
○ Suppose you had a matrix of randomly generated data and
you wanted to replace all positive values with 2 and all
negative values with –2.
 ● Operations between Arrays and Scalars
● Basic Indexing and Slicing
● Boolean Indexing

UNIT IV ●
●
●
Fancy Indexing
Data Processing Using Arrays
Expressing Conditional Logic as Array
Operations
TOPICS ● Methods for Boolean Arrays
● Sorting
● Unique
 ● Operations between Arrays and Scalars
● Basic Indexing and Slicing
● Boolean Indexing

UNIT IV ●
●
●
Fancy Indexing
Data Processing Using Arrays
Expressing Conditional Logic as Array
Operations
TOPICS ● Methods for Boolean Arrays
● Sorting
● Unique
Sorting:
● NumPy arrays can be sorted in-place with the sort method.
Sorting:
● You can sort each one-dimensional section of values in a
multidimensional array inplace along an axis by passing the
axis number to sort.
●
Sorting:
● The top-level method [Link] returns a sorted copy of an array
instead of modifying the array in-place.
●
 ● Operations between Arrays and Scalars
● Basic Indexing and Slicing
● Boolean Indexing

UNIT IV ●
●
●
Fancy Indexing
Data Processing Using Arrays
Expressing Conditional Logic as Array
Operations
TOPICS ● Methods for Boolean Arrays
● Sorting
● Unique
Unique:
● NumPy has some basic set operations for one-dimensional
ndarrays.
● A commonly used one is [Link], which returns the sorted
unique values in an array.
Exercises
Thank you