Concept of Data

Science
Application of Data
Science

LEARNING AI Project Cycle

Data Acquisition
OUTCOMES Visualization
Exploration
Python Libraries
NymPy, Pandas and
Matplotlib
Statistics and KNN
 INTRODUCTION
TO DATA SCIENCE
Introduction to Data Science

Data science is the study of data to extract meaningful

insights for business.

Artificial Intelligence completely depends on data.

Data is the core of Al, as it is the data that makes

machines intelligent.

D a t a c a n b e i n t h e f o r m o f n u m b e r s , t e x t , a u d i o , o r v i d e4o .
 THREE
DOMAINS OF AI
 Natural
Computer
Data Science Language
Vision
Processing

Works on numeric Works on textual

Works on images
and alpha- and speech-
and visual data
numeric data based data

6
Presentation Title 7
 Big Data is a collection of data
BIG DATA
that is large, complex, and

growing rapidly in volume,

v a r i e t y, a n d v e l o c i t y.

It is so large and complex that

traditional data management

systems are unable to store,

process, and analyze it.

8
 Data can be structured or
BIG DATA
unstructured.

Raw data is available in

massive amounts, and the job of

Data Scientist is to find

meaning and hidden patterns to

draw conclusions and

predictions based on the given

data. 9
10
 Fraud and Risk
Detection
Genetics and Genomics
Search engines
APPLICATIONS OF Website
Recommendation
DATA SCIENCE Advanced Image
Recognition
Speech Recognition
Gaming Platforms
Targeted Advertising
Airline Route Planning
AI PROJECT CYCLE The Scenario
Everyone enjoys socializing and eating out with friends and
f a m i l y. Restaurants offer buffets and a variety of meals for
c o n s u m e r s t o e n j o y. T h e y e s t i m a t e t h e n u m b e r o f c u s t o m e r s t h a t
would walk into their restaurant every day and cook food in bulk
to meet customer needs. It is difficult for restaurants to
accurately estimate the number of people who will visit the
restaurant each d a y. A s a result, they cook lots of food in
a n t i c i p a t i o n a l a r g e c r o w d . O f t e n , a t t h e e n d o f t h e d a y, a l a r g e
a m o u n t o f f o o d i s l e f t o v e r.
12
AI PROJECT CYCLE
This stale food cannot be served to customers the next day and, as
a result, gets thrown away or given sed pay for free. This results
in daily losses that add up a large amount of annual losses for the
restaurants.
A c c o r d i n g t o t h e U N E P F o o d Wa s t e I n d e x R e p o r t 2 0 2 1 , a r o u n d 9 3 1
million tonnes of food waste were generated in 2019. The report
covers three sectors: food retail, households, and food service.
Due to food waste, 690 million people had to go hungry in 2019.

13
 931 MILLION
TONNES FOOD WAS WASTED IN 2019

Of which

61% 26% 13%

came from households from food service from retail 14
15
STAGE 1: PROBLEM
SCOPING
Problem scoping will help find out more about the problem, find
out the factor the affect it, and define the goal of the project. Let
us fill the 4Ws Problem Canvas to help us in this process:

Who Canvas: Who is having the Problem?

Who are the stakeholders?  Restaurants offering buffets

 Restaurants chefs

What do you know about them?  Restaurants cook food in bulk every day for their
buffets to meet customer needs.
 They estimate the number of customers that would
16
walk into their restaurants every day.
 What Canvas: what is the nature of their Problem?

What is the problem?  Quite a large amount of food is left unconsumed every
day at the restaurants, which is either thrown away or
given for free to needy people.
 Restaurants have to bear everyday losses for the
unconsumed food.

How do you know it is a  Restaurants surverys have shown that restaurants

problem? face this problem of food waste.

Where Canvas: where does the problem arise?

What is the context or situation  Restaurants that serve buffet food.

in which the stakeholders  End of the day, when no further food consumption is
experience this problem? possible.

17
 Why Canvas: Why do you think it is a problem worth solving?

What would be of key value to  If the restaurants has a proper estimate of the
the stakeholders? quantity of food to be prepared every day, food
wastage can be reduced.

How would it improve their  Less or no food would be left unconsumed

situation?  Losses due to unconsumed food would reduce
considerably.

Our Restaurants owner Who

Have a problem of Losses due to food wastage What
While The food is left unconsumed due to improper estimate Where
An ideal solution To be able to predict the amount of food to be prepared for
would everyday consumption. Why

18
 In an AI project cycle, data

acquisition is the process of

gathering the data needed to

solve a problem. Data acquisition

DATA is important because it helps to

ACQUISITION understand the parameters

related to problem scoping.

19
STAGE 2: DATA
ACQUISITION
 In our scenario, the various factors that would affect the quantity of food to be
prepared for consumption in the next days buffet are:

 Total number of customers expected

 Quantity of dishes prepared per day

 Consumption of each dish

 Unconsumed quantity of dishes every day

 Price of dishes

 Quantity of dishes required for the next day

20
  Data exploration refers to the
initial step in data analysis in
which data analysts use data
visualization and statistical

STAGE 3: DATA
techniques to describe dataset
characterizations, such as size,

EXPLORATION quantity, and accuracy, in order

to better understand the nature
of the data.

 We extract the useful

information from the dataset
and make sure there are no
errors or missing elements in it.
21
  Data modeling is a critical step

STAGE 4: DATA
in simplifying complex data for
computers to process and

MODELLING make predictions. At the start,

data is usually presented in
charts or graphs to help people
spot patterns.

22
  The stage of testing the models
is known as EVALUATION. OR.

STAGE 5: Evaluation is a process

understanding the reliability of
of

EVALUATION any AI model, based on outputs

by feeding the test dataset into
the model and comparing it
with actual answers.

23
DATA
COLLECTION
 Data collection has been a part of the society even before we had advanced
technology and higher computational skills.

 Records have been maintained since older times to keep track of important
information.

 Data collection does not require technical skills, but analysis of data involves
numeric and alphanumeric data, which can be challenging for humans. This is where
data science can help.

 It not only helps us understand data better but also provides deeper and clearer
insights. When Al becomes part of the process, machines can provide even better
predictions and suggestions based on the data. 24
DATA
COLLECTION
some commonly known examples of datasets
are:

 Banks

 ATM machines

 Movies

25
SOURCE OF
DATA
OFFLINE DATA
ONLINE DATA COLLECTION
COLLECTION
Sensors Open-sourced government portals

Surveys Reliable Websites (Kaggle)

Interviews
World Organizations' open sourced
statistical websites
Observations

26
 Comma separated
Values (CSV)
TYPES OF DATA Spreadsheet
Structured Query
Language (SQL)
 CSV
(Comma Separated Values) It is a simple file format used to
store tabular data. Each line of this
file is a data record, and each
record consists of one or more
fields that separated by commas.
Hence, the name is CSV, i.e.,
Comma Separated Values. 28
  A spreadsheet is a table drawn
on paper or a digital table
created using a computer
program. It is used for
accounting and recording data

SPREADSHEET using rows and columns into

which information can be
entered. Microsoft Excel,
Apache OpenOffice Calc, and
Google Sheets are some
examples of programs that help
in creating spreadsheets.
29
SPREADSHEET

Calc 30
  SQL is a specialized
programming language
used fer designing,
programming, and
SQL managing data within
(Structured Query Language) Database Management
Systems (DBMS). It is
especially useful for
handling structured
data. 31
SQL

32
DATA ACCESS
 In order to use the collected data for programming purposes, we should know how
to access it in Python code.

 Python provides various packages, like NumPy, Pandas, and Matplotlib, that help us
access structured data (in tabular form) within the code.

 Let us take a look at some of these packages that are used for data analysis and
visualisation

33
NumPy
 NumPy stands for Numerical Python.

 It is the fundamental package for mathematical and logical operations on arrays in

Python.

 NumPy is a commonly used package that offers a wide range of arithmetic

operations that make it easy to work with numbers as well as arrays.

 An array is a homogeneous collection of data. set of multiple values of the same

data type.

 The values can be numbers, characters, Booleans but it is important to note that an
array can only have one data type at a time.
35
Array can be of one or more dimensions

A one-
dimensional
A two-
array is called
dimensional
vector
array is called
matrix

An array with
multiple
dimensions is
called an n-
dimensional array
 Difference between NumPy array and List
NumPy array List
 It is homogeneous collection of data.  It is hetrogeneous collection of data.
 It cannot be directly initialized. It can be  It can be directly initialized as it is part of
operated with the NumPy package only. Python syntax.
 Direct numerical operations can be
 Direct numerical operations cannot be
performed on arrays.
performed on list.
Ex: import numpy as np
Ex: a=lst=[10,20,30]/3
marks=[Link]([10,20,30])//3
print(a)
print(marks)
An error will generate this code
Output: [3 6 10]
 It is widely used for arithmetic operation.  It is widely used for data management.

 Array take less memory space.  List require more memory space.
 Functions like concatenation, append,
 Functions like concatenation, append,
reshape are not easily possible with
reshape are easily possible with arrays.
 NumPy can be imported in Jupyter Notebook in these
ways.

Python import statement

Function
for the NumPy Library

import numpy It import the entire NumPy package

It import the entire NumPy package but

import numpy as np
rename it as “np”

It imports only the arrays function from the

from numpy import array
NumPy package

from numpy import array as It imports only the arrays function from the
arr NumPy package but renames it as “arr”
Creating an one-dimensional Array using NumPy

import numpy as np
marks=[Link]([50,60,85])
print(marks)
Output: [50 60 85]

[Link](start, stop, step)

import numpy as np
a=[Link]([10,100,10])
print(a)
Output: [10 20 30 40 50 60 70 80 90 100]
Creating an one-dimensional Array using NumPy

[Link] (the size of random numbers required)

import numpy as np
a=[Link](5)
print(a)
Output: [0.84558274 0.08309987 0.94457852 0.30818991
0.9216126]
 Creating a two-dimensional Array using NumPy

randint() – This method returns an integer number selected element from the specified range.

import numpy as np
a=[Link](20,size=(2,3))
print(a)
Output: [[15 2 3]
[12 8 17]]
Creating a two-dimensional Array using NumPy

The [Link]() function returns a new array of given shape and type, with ones.

import numpy as np
a=[Link]((2,3))
print(a)
Output: [[1. 1. 1.]
[1. 1. 1.]]
Creating a two-dimensional Array using NumPy

The full() method creates a new array of given shape and type, filled with a given fill value.

import numpy as np
a=[Link]((3,4),5)
print(a)
Output: [[5 5 5 5]
[5 5 5 5]
[5 5 5 5]]
 Arithmetic Operations an Arrays
Operation Example

import numpy as np
a=[Link]([100,150])
Addition b=[Link]([150,208])
c=a+b
print(c) Output: [250 358]
import numpy as np
a=[Link]([100,150])
Subtraction b=[Link]([150,208])
c=a-b
print(c) Output: [-50 058]
import numpy as np
Subtraction Stock=[Link]([25,30,22,15,18])
(with single Item_sold=5
number) c=Stock-Item_sold
print(c) Output: [20 25 17 10 13]
 Arithmetic Operations an Arrays
Operation Example

import numpy as np
a=[Link]([2,3])
b=[Link]([4,5])
Multiplication
c=a*b
print(c) Output: [8 15]

import numpy as np
a=[Link]([100,150])
b=[Link]([5,3])
Division
c=a/b
print(c) Output: [20 50]

import numpy as np
Exponentiation arr=[Link]([5,3,2,5,8])
print(arr**2) Output: [25 9 4 25 64]
 Arithmetic Operations an Arrays
Operation Example

import numpy as np
a=[Link]([25,35,48,58])
b=[Link]([4,5,6,8])
Floor Division c=a//b
print(c) Output: [6 7 8 7]

import numpy as np
a=[Link]([25,35,48,58])
b=[Link]([4,5,6,8])
Modulus c=a%b
print(c) Output: [1 0 0 2]
 Additional Array Functions and Attributes
Operation Example

import numpy as np
arr=[Link]([25,35,48,58])
type arr_type=type(arr)
print(arr_type) Output: <class ‘[Link]’>
import numpy as np
arr=[Link]([25,35,48,58])
.ndim
dimensions=[Link]
print(dimensions) Output: 1
import numpy as np
arr=[Link]([25,35,48,58])
.shape
shape=[Link]
print(shape) Output: (4,)
import numpy as np
arr=[Link]([25,35,48,58])
.size
size=[Link]
print(size) Output: 4
 Additional Array Functions and Attributes
Operation Example

import numpy as np
arr=[Link]([25,35,48,58])
.dtype data_type=[Link]
print(data_type) Output: int32

import numpy as np
arr=[Link]([25,35,48,58])
[Link] max_value=[Link](arr)
print(max_value) Output: 58

import numpy as np
arr=[Link]([[25,35,48,58],[10,45,103,123]])
[Link] r_max_value=[Link](arr, axis=1)
(arr,axis=1) print("Row:",r_max_value)
[Link] c_max_value=[Link](arr, axis=0)
(arr,axis=0) print("Column:",c_max_value) Output: Row: [ 58 123]
Column: [ 25 45
 Additional Array Functions and Attributes

Operation Example

import numpy as np
arr=[Link]([58,35,25,48])
[Link](arr)
sorted_arr=[Link](arr)
print(sorted_arr) Output: [25 35 48 58]
import numpy as np
[Link](arr) arr=[Link]([88,45,55])
[::-1] sorted_arr_reverse=[Link](arr)[::-1]
print(sorted_arr_reverse) Output: [88 55 45]
import numpy as np
arr=[Link]([58,35,25,48])
sum_values=[Link](arr)
[Link](arr)
print(sum_values) Output: 166
Pandas
 Pandas is a Python library mainly used for data manipulation and analysis.

 It provides data structures and operations for handling numerical tables and time
series.

 The key data structures in Pandas are Series (1-dimensional), DataFrame (2-
dimensional), and Panel (3-dimensional)

 Which are used for data cleaning, transformation, and analysis.

 The name 'Pandas' is derived from the econometrics term "panel data“

 The two primary data structures of Pandas, i.e., Series and DataFrame, handle most
of the applications in the fields of finance, statistics, social science, and engineering.
51
 Pandas is built on top of NumPy and can integrate well with many other third-party
 Pandas is well suited for different kinds of data, like:
 Tabular data with heterogeneously-typed columns, as in an SQL table or Excel
spreadsheet.

 Ordered (data in a sequence) time series data. Time series data involves recording
observations at multiple time points. Ex: daily stock prices, monthly sales figures, or
hourly temperature readings.

 Arbitrary matrix data (homogeneously typed or heterogeneous) with row and

column labels. This means data arranged in a matrix-like format that can be of the
same data type or different data types across the matrix.
Homogeneous Data Heterogeneous Data
 File access using Pandas
 Create a CSV file. For example, enter the following details in Excel and save it as a
CSV extension.
 Write following code in Jupyter notebook and
run.
import pandas as pnd

marks=pnd.read_csv("C:\\Users\\levin\\Desktop\\[Link]")

print("First five rows")

print([Link](5))

print("Records after sorting")

Run your code in
sorted_info=marks.sort_values(by=['dob’])
Jupyter notebook by
print(sorted_info.head(5))
pressing Shift + Enter
 Features of Pandas
 Easy handling of missing data:

 Size mutability : Columns can be inserted and deleted from Pandas DataFrame.

 Explicit and automatic data alignment : Explicit alignment based on specified index
labels.
 Size mutability
 Inserting new column  Deleting the column ‘B’

import pandas as pd [Link](‘B’, axis=1, inplace=True)

data = {'A':[1,2,3], 'B': del df[‘C’]

[4,5,6]} print(df)
df=[Link](data)

df['C']=[7,8,9]print(df)
Explicit and automatic data alignment
 Creating a Series  Output

import pandas as pdseries1 = A 1

[Link]([1,2,3],
B 2
index=['A','B','C’])
C 3
print(series1)
dtype: int64
Matplotlib

 Matplotlib is a free and open source library.

 It is a useful tool in Python for creating two- dimensional plots of arrays.

 It is a visualisation library that works on multiple platforms and is based on NumPy

arrays.

 The major benefit of visualisation is that it helps us understand large amount of data

through easily readable plots, i.e., charts and graphs.

 Plots help us understand trends and patterns and make correlations.

58
Matplotlib

Bar Graph Scatter Plot Area Plot Pie Plot

59
INSTALLING
MATPLOTLIB

60
 Basic Statistics with Python
 In order to analyse numeric and alphanumeric data used in the data science domain,

basic statistical methods of mathematics can be used.

Operation Example

Mean is the average value of a sequence. It is calculated as:

Mean
mean = sum of all values / total number of values
Median is the 50th percentile value of a sequence. It is the middle value of
Median a
dataset when it is sorted in ascending order.
Mode is the value that appears most frequently in the sequence.
Mode

Standard deviation measures how spread out the values in a dataset are.
Standard A low Deviation standard deviation means the values are close to the
Deviation mean (average), while a high standard deviation means they are more
spread out
Variance is the average of the squared difference from the mean. It is a
Variance measure of how much the values in a dataset differ from the mean.
 Basic Statistics with Python
import statistics as st

ages=[25,30,35,40,45,50,55,60,65,70] Output
mean=[Link](ages)
Mean 47.5
median=[Link](ages)

mode=[Link](ages) Median 47.5

std_dev=[Link](ages)
Mode 25
variance=[Link](ages)

print("Mean",mean) Standard Deviation 15.14

print("Median",median)
Variance 229.17
print("Mode",mode)

print("Standard Deviation",round(std_dev,2))

print("Variance",round(variance,2))
Data
Visualisation
 Analysing the collected data can be difficult as it primarily consists of tables and

numbers. While machines work efficiently with numbers, humans need visual aids to

understand and comprehend information.

 Data visualisation is used to interpret the collected data and identify patterns and

trends within it.

64
 Basic Statistics with Python
The data collected during the data acquisition stage may have some errors. Some of the

issues that we can face with data are:

Error Reason
There are two ways in which the data can be erroneous:
Incorrect values: This happens when data contains values that do not
belong in that position. The values in the dataset can be incorrect. For
example, a phone number column contains a decimal value, or there
Erroneous
could be an address in the cost column.
Data
Invalid or null values: Sometimes, data gets corrupted, resulting in
invalid values or NaN (Not a Number) values. These values are
meaningless and cannot be processed. They must be removed from the
database.
Missing Data In some datasets, certain cells remain empty since the values are missing.

Outliers in data visualization are data points that are far from the majority
Outliers of the data in a set. They can be much higher or lower than other values
and can skew the results of data analysis.
Outliers
90

25
80

29 70

3 60

32 50

85 40

33 30

27 20

28 10

32.75
0
0 1 2 3 4 5 6 7 8 9

66
Scatter Plots
import [Link] as plt

vehicles=[10,20,25,40,50,60,75,80,85,90]

pollution=[30,20,90,96,100,300,375,400,405,500]

[Link](vehicles,pollution,color="blue",

marker='s’)

[Link]('Number of the vehical on the road’)

[Link]('Air pollution level (AQI)’)

[Link]('Number of vehicle vs. Air Pollution’)

[Link]()
67
Bar Chart
import [Link] as plt

cate=["Groceries","Utilities","Rent","Entertainm

ent"]

exp=[500,200,1000,300]

[Link](cate,exp,color="purple")

[Link]('Expense Categories’)

[Link]('Monthly expenses (in USD)’)

[Link]('Monthly expenses by category’)

[Link]()
68
69