DAI-101
Introduction to Data Science
Dr. Chetan Ralekar
([Link]@[Link])
Visiting Faculty, Mehta Family School for DS and AI, IITR
Research Affiliate, MIT, USA
Course Structure
Introduction to Data Science: Latest and greatest in data science, Real world applications
Python: Data types (bool, int, float, string, range, list, tuple, set/frozenset, dictionary, string/list operators), Loops (if, elif, else, for, while, for-else, while-else,
pass continue, break), Functions (def, lamda, map), Scoping, Modules, File Handling (Reading, Writing, Appending), Graph Plotting (Line/bar graphs, scatter
plots), Object Oriented Programming (Abstract data types, Object oriented design, Classes)
Data Analysis Foundation: Types of data (data matrix, numeric, categorical datasets), Data preparation (data cleaning, data reduction, and transformation)
Exploratory Data Analysis and Visualization: Univariate and bivariate analysis (Chi-square, t-test, ANOVA, Correlation coefficient), data visualization
(scatterplot, segmented bar plot, mosaic plot, contingency table, line chart, pie charts, histogram, box-whisker plot, violin plot, funnel chart, Venn plot, PCA,
volcano plot, heatmap)
Statistical Analysis: Measures of central tendency (arithmetic/geometric mean, median, mode, range, variance, SD, IQR) Confidence Intervals, Hypothesis
Testing, Parametric and non-parametric tests (one/two sample tests, Mann-Whitney), p-values, Bias and Variance trade-off
Machine Learning: Supervised, unsupervised and reinforcement learning, Linear algebra (transpose, vector-vector product, Euclidean plane/distance, norm,
determinant, transformation, Eigenvector, Concave/convex function, matrix calculus), Probability distribution (Binomial, Poisson, Gaussian), Conditional
probability (Bayes theorem), Model training, Overfitting and underfitting, Bias and variance, Supervised methods: Linear classification (PLA, pocket
algorithm), Linear regression, Gradient descent, Hyperparameter optimization, Logistic regression, Decision trees, SVM, Clustering, K-means, PCA
Deep learning and Big Data: Gradient Descent, Neural nets, Convolutional Neural Networks, Big Data technologies (MapReduce, HDFS)
What is Data Science?
• Data Science is an interdisciplinary field that focuses on extracting
meaningful insights and knowledge from data.
• It combines ideas and tools from statistics, mathematics, computer
science, and domain expertise to analyze large and complex datasets.
• In simple terms, data science is about:
➢Collecting data (from experiments, surveys, sensors, web, logs, etc.)
➢Cleaning and organizing it
➢Analyzing it using statistical and computational methods
➢Building models (often using machine learning) to find patterns or make
predictions
➢Communicating results to support decision-making
Applications of DS
Humanities
Data Machine/
Management Statistical
Learning
Law
Data Application
Domain
Science Expertise
Social
Visualization
Science
Mathematical
Optimization
Data ML/DL/
Analytics Artificial
Science Intelligence
“Data science produces insights.
Machine learning produces predictions”
Recommender Systems
• The ability to offer unique personalized
service
• Increase sales, click-through rates,
conversions, …
• Netflix recommender system valued
at $1B per year
• Amazon recommender system drives
a 20-35% lift in sales annually
• Collaborative filtering at scale
Predicting why patients are being readmitted
• Reduce costs
• Improve population health
• Find the “why” behind specific
populations being readmitted
• Data lakes of multiple data
sources
• Investigate ties between
readmission and socioeconomic
data points, patient history,
genetics,…
Real world Applications
• Search Engines: Google, Yahoo, Bing. Data sciences is used to get faster
searches.
• Driverless cars: With the help of Data Science techniques, the Data is
analyzed like what as the speed limit in highways, Busy Streets, Narrow
Roads, etc. And how to handle different situations while driving etc.
• Finance: Fraud detection, stock market. In the Stock Market, Data Science
is used to examine past behavior with past data and their goal is to examine
the future outcome.
• E-commerce: Amazon, Flipkart, Meesho
• Healthcare
Case Study
Predicting Student Performance in an Online Learning Platform
Problem Statement
An online education platform is facing high dropout rates in its courses.
The management wants to identify students who are at risk of dropping
out so that timely interventions (emails, mentoring, extra resources) can
be provided.
Continued…
1. Data collection
• Number of logins per week
• Average time spent on platform (minutes/day)
• Percentage of course videos completed
• Avg quiz score
• Assignments submitted
2. Data Cleaning and Preprocessing
• Some students have missing quiz scores
• Time spent has outliers (e.g. 0 or 12 hours/day)
• Operations include: Handling missing values, removing outliers, normalizing features etc.
Continued…
3. Exploratory Data Analysis
• Using plots and statistics, analysts explore patterns:
• Students who drop out log in 3× less on average
• Dropouts submit fewer than 40% of assignments
• Strong correlation between videos_watched and completion
• Techniques used: Histograms, box plots, Correlation matrices
4. Modeling (ML)
The goal is to predict drop_out
5. Deployment and Decision making
• The trained model is deployed in the platform:
• Every week, the model flags “at-risk” students
• Students with dropout probability > 0.7 are contacted
• Personalized emails and mentor calls are triggered
Python
• High-level, interpreted language
• Simple syntax, powerful capabilities
• Widely used in:
❖ Data Science & AI
❖ Scientific Computing
❖ Automation
❖ Web & Embedded Systems
Computing with Numbers
• To understand the concept of data types.
• To be familiar with the basic numeric data types in Python.
• To understand the fundamental principles of how numbers are
represented on a computer.
• To be able to use the Python math library.
• To be able to read and write programs that process numerical data.
Numerical Data Types
• The information that is stored and manipulated bu computers
programs is referred to as data.
• There are two different kinds of numbers!
• (5, 4, 3, 6) are whole numbers – they don’t have a fractional part
• (.25, .10, .05, .01) are decimal fractions
• Inside the computer, whole numbers and decimal fractions are
represented quite differently!
• We say that decimal fractions and whole numbers are two different
data types
Numerical Data Types
• The data type of an object determines what values it can have and
what operations can be performed on it.
• Whole numbers are represented using the integer (int) data type.
• These values can be positive or negative whole numbers.
• Numbers that can have fractional parts are represented as floating
point (or float) values.
Numeric Data Types
• Numbers that can have fractional parts are represented as floating
point (or float) values.
• How can we tell which is which?
• A numeric literal without a decimal point produces an int value
• A literal that has a decimal point is represented by a float (even if the fractional
part is 0)
19
Numeric Data Types
• Python has a special function to tell us the data type of any value.
>>> type(3)
<class 'int'>
>>> type(3.1)
<class 'float'>
>>> type(3.0)
<class 'float'>
>>> myInt = 32
>>> type(myInt)
<class 'int'>
>>>
20
Numeric Data Types
• Why do we need two number types?
• Values that represent counts can’t be fractional (you can’t have 3 ½ quarters)
• Most mathematical algorithms are very efficient with integers
• The float type stores only an approximation to the real number being
represented!
• Since floats aren’t exact, use an int whenever possible!
21
Numeric Data Types
• Operations on int produce int, operations on floats produce floats (except for /).
>>> 3.0+4.0
7.0
>>> 3+4
7
>>> 3.0*4.0
12.0
>>> 3*4
12
>>> 10.0/3.0
3.3333333333333335
>>> 10/3
3.3333333333333335
>>> 10 // 3
3
>>> 10.0 // 3.0
3.0
22
Numeric Data Types
• Integer division produces a whole number.
• That’s why 10//3 = 3!
• Think of it as ‘gozinta’, where 10//3 = 3 since 3 gozinta (goes
into) 10 3 times (with a remainder of 1)
• 10%3 = 1 is the remainder of the integer division of 10 by 3.
• a = (a/b)(b) + (a%b)
23
Using the Math Library
• Besides (+, -, *, /, //, **, %, abs), we have lots of other math functions
available in a math library.
• A library is a module with some useful definitions/functions.
24
Find answers to the following
• 5/2=
• 5 // 2 =
• int(3.9) =
• int(-3.9) =
• int(“007”) =
25
Using the Math Library
• Let’s write a program to compute the roots of a quadratic equation!
−b b 2 − 4ac
x=
2a
• The only part of this we don’t know how to do is find a square root…
but it’s in the math library!
26
Using the Math Library
• To use a library, we need to make sure this line is in our program:
import math
• Importing a library makes whatever functions are defined within it
available to the program.
27
Using the Math Library
• To access the sqrt library routine, we need to access it as [Link](x).
• Using this dot notation tells Python to use the sqrt function found in
the math library module.
• To calculate the root, you can do
discRoot = [Link](b*b – 4*a*c)
28
Using the Math Library
This program finds the real solutions to a quadratic
Please enter the coefficients (a, b, c): 3, 4, -1
The solutions are: 0.215250437022 -1.54858377035
• What do you suppose this means?
This program finds the real solutions to a quadratic
Please enter the coefficients (a, b, c): 1, 2, 3
Traceback (most recent call last):
File "<pyshell#26>", line 1, in -toplevel-
main()
File "C:\Documents and Settings\Terry\My Documents\Teaching\W04\CS 120\Textbook\code\chapter3\[Link]", line 14, in main
discRoot = [Link](b * b - 4 * a * c)
ValueError: math domain error
>>>
30
Math Library
• If a = 1, b = 2, c = 3, then we are trying to take the square root of a
negative number!
• Using the sqrt function is more efficient than using **. How could you
use ** to calculate a square root?
31
Accumulating Results: Factorial
• Say you are waiting in a line with five other people. How many ways
are there to arrange the six people?
• 720 -- 720 is the factorial of 6 (abbreviated 6!)
• Factorial is defined as:
n! = n(n-1)(n-2)…(1)
• So, 6! = 6*5*4*3*2*1 = 720
32
Accumulating Results: Factorial
• How we could we write a program to do this?
• Input number to take factorial of, n
Compute factorial of n, fact
Output fact
33
Accumulating Results: Factorial
• How did we calculate 6!?
• 6*5 = 30
• Take that 30, and 30 * 4 = 120
• Take that 120, and 120 * 3 = 360
• Take that 360, and 360 * 2 = 720
• Take that 720, and 720 * 1 = 720
34
Accumulating Results: Factorial
• What’s really going on?
• We’re doing repeated multiplications, and we’re keeping track of the
running product.
• This algorithm is known as an accumulator, because we’re building up
or accumulating the answer in a variable, known as the accumulator
variable.
35
Accumulating Results: Factorial
• The general form of an accumulator algorithm looks like this:
Initialize the accumulator variable
Loop until final result is reached
update the value of accumulator variable
36
Accumulating Results: Factorial
• It looks like we’ll need a loop!
fact = 1
for factor in [6, 5, 4, 3, 2, 1]:
fact = fact * factor
• Let’s trace through it to verify that this works!
37
Accumulating Results: Factorial
• Why did we need to initialize fact to 1? There are a couple reasons…
• Each time through the loop, the previous value of fact is used to calculate the
next value of fact. By doing the initialization, you know fact will have a value
the first time through.
• If you use fact without assigning it a value, what does Python do?
38
Accumulating Results: Factorial
• Since multiplication is associative and commutative, we can rewrite
our program as:
fact = 1
for factor in [2, 3, 4, 5, 6]:
fact = fact * factor
• Great! But what if we want to find the factorial of some other
number??
39
Accumulating Results: Factorial
• What does range(n) return?
0, 1, 2, 3, …, n-1
• range has another optional parameter! range(start, n) returns
start, start + 1, …, n-1
• But wait! There’s more!
range(start, n, step)
start, start+step, …, n-1
• list(<sequence>) to make a list
40
Accumulating Results: Factorial
• Let’s try some examples!
>>> list(range(10))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> list(range(5,10))
[5, 6, 7, 8, 9]
>>> list(range(5,10,2))
[5, 7, 9]
41
Accumulating Results: Factorial
• Our completed factorial program:
# [Link]
# Program to compute the factorial of a number
# Illustrates for loop with an accumulator
def main():
n = eval(input("Please enter a whole number: "))
fact = 1
for factor in range(n,1,-1):
fact = fact * factor
print("The factorial of", n, "is", fact)
main()
43
The Limits of Int
• What is 100!?
>>> main()
Please enter a whole number: 100
The factorial of 100 is
93326215443944152681699238856266700490715968264381621468592963895217599993
22991560894146397615651828625369792082722375825118521091686400000000000000
0000000000
• Wow! That’s a pretty big number!
44
The Limits of Int
• Newer versions of Python can handle it, but…
Python 1.5.2 (#0, Apr 13 1999, 10:51:12) [MSC 32 bit (Intel)] on win32
Copyright 1991-1995 Stichting Mathematisch Centrum , Amsterdam
>>> import fact
>>> [Link]()
Please enter a whole number: 13
13
12
11
10
Traceback (innermost last):
File " <pyshell #1>", line 1, in ?
fact.m ain()
File " C:\PROGRA~1\PYTHON~1.2\[Link]", line 5, in main
fact=fact*factor
Overfl owError: integer multipl ication
45
The Limits of Int
• What’s going on?
• While there are an infinite number of integers, there is a finite range of
integers that can be represented.
• This range depends on the number of bits a particular CPU uses to represent an
integer value. Typical PCs use 32 bits.
46
The Limits of Int
• Typical PCs use 32 bits
• That means there are 232 possible values, centered at 0.
• This range then is –231 to 231-1. We need to subtract one from the top
end to account for 0.
• But our 100! is much larger than this. How does it work?
47
Handling Large Numbers
• Does switching to float data types get us around the limitations of ints?
• If we initialize the accumulator to 1.0, we get
>>> main()
Please enter a whole number: 15
The factorial of 15 is 1.307674368e+012
• We no longer get an exact answer!
48
Handling Large Numbers: Long Int
• Very large and very small numbers are expressed in scientific or exponential
notation.
• 1.307674368e+012 means 1.307674368 * 1012
• Here the decimal needs to be moved right 12 decimal places to get the original
number, but there are only 9 digits, so 3 digits of precision have been lost.
49
Handling Large Numbers
• Floats are approximations
• Floats allow us to represent a larger range of values, but with lower
precision.
• Python has a solution, expanding ints!
• Python Ints are not a fixed size and expand to handle whatever value it
holds.
50
Handling Large Numbers
• Newer versions of Python automatically convert your ints to expanded
form when they grow so large as to overflow.
• We get indefinitely large values (e.g. 100!) at the cost of speed and
memory
51
Type Conversions
• We know that combining an int with an int produces an int, and
combining a float with a float produces a float.
• What happens when you mix an int and float in an expression?
x = 5.0 + 2
• What do you think should happen?
52
Type Conversions
• For Python to evaluate this expression, it must either convert 5.0 to 5
and do an integer addition, or convert 2 to 2.0 and do a floating point
addition.
• Converting a float to an int will lose information
• Ints can be converted to floats by adding “.0”
Python Programming, 2/e 53
Type Conversion
• In mixed-typed expressions Python will convert ints to floats.
• Sometimes we want to control the type conversion. This is called
explicit typing.
54
Type Conversions
>>> float(22//5)
4.0
>>> int(4.5)
4
>>> int(3.9)
3
>>> round(3.9)
4
>>> round(3)
3
55
The String Data Type
• Text is represented in programs by the string data type.
• A string is a sequence of characters enclosed within quotation
marks (") or apostrophes (').
56
The String Data Type
>>> str1="Hello"
>>> str2='spam'
>>> print(str1, str2)
Hello spam
>>> type(str1)
<class 'str'>
>>> type(str2)
<class 'str'>
57
The String Data Type
• Getting a string as input
>>> firstName = input("Please enter your name: ")
Please enter your name: John
>>> print("Hello", firstName)
Hello John
• Notice that the input is not evaluated. We want to store the typed
characters, not to evaluate them as a Python expression.
58
The String Data Type
• We can access the individual characters in a string through
indexing.
• The positions in a string are numbered from the left, starting with
0.
• The general form is <string>[<expr>], where the value of expr
determines which character is selected from the string.
59
The String Data Type
H e l l o B o b
0 1 2 3 4 5 6 7 8
>>> greet = "Hello Bob"
>>> greet[0]
'H'
>>> print(greet[0], greet[2], greet[4])
Hlo
>>> x = 8
>>> print(greet[x - 2])
B
60
The String Data Type
H e l l o B o b
0 1 2 3 4 5 6 7 8
• In a string of n characters, the last character is at
position n-1 since we start counting with 0.
• We can index from the right side using negative
indexes.
>>> greet[-1]
'b'
>>> greet[-3]
'B'
61
The String Data Type
• Indexing returns a string containing a single character from a
larger string.
• We can also access a contiguous sequence of characters, called
a substring, through a process called slicing.
62
The String Data Type
• Slicing:
<string>[<start>:<end>]
• start and end should both be ints
• The slice contains the substring beginning at position start and
runs up to but doesn’t include the position end.
63
The String Data Type
H e l l o B o b
0 1 2 3 4 5 6 7 8
>>> greet[0:3]
'Hel'
>>> greet[5:9]
' Bob'
>>> greet[:5]
'Hello'
>>> greet[5:]
' Bob'
>>> greet[:]
'Hello Bob'
64
The String Data Type
• If either expression is missing, then the start or the end of the
string are used.
• Can we put two strings together into a longer string?
• Concatenation “glues” two strings together (+)
• Repetition builds up a string by multiple concatenations of a string
with itself (*)
65
The String Data Type
• The function len will return the length of a string.
>>> "spam" + "eggs"
'spameggs'
>>> "Spam" + "And" + "Eggs"
'SpamAndEggs'
>>> 3 * "spam"
'spamspamspam'
>>> "spam" * 5
'spamspamspamspamspam'
>>> (3 * "spam") + ("eggs" * 5)
'spamspamspameggseggseggseggseggs'
66
The String Data Type
>>> len("spam")
4
>>> for ch in "Spam!":
print (ch, end=" ")
Spam!
67
The String Data Type
Operator Meaning
+ Concatenation
* Repetition
<string>[] Indexing
<string>[:] Slicing
len(<string>) Length
for <var> in <string> Iteration through characters
68