Different Classifiers (Algorithms)
• Logistic Regression
• Decision Tree Classifier
• Support Vector Machines
• K- Nearest Neighborhood
• Linear discriminant analysis
• Gaussian Naive Bayes
City Block Distance
Red one Manhattan Distance ≈ |x2-x1|+|y2-y1|
It simply look 2 symbols ,
if they are same, it
calculate the distance to
be 0 otherwise number of
distinct value.
Ex:3
Fins the nearest neighbor of C1
Is C1 eque
distance with
C2 and C3???
Here salary is important parameter, so distance of salary should be consider, but to
handle such situation, the counter action is……
Mostly Normalization is used as it preserves the distribution of Data
Will the Euclidian
or Manhattan will
work????
K=15 gives good performance
Regression
Use data to identify relationships among variables and
use these relationships to make predictions.
Prediction and Forecasting
• Predict home sales for December given the interest rate for this
month.
• Use time series data (e.g., sales vs. year) to forecast future
performance (next year sales).
• Predict the selling price of houses based on the size of house.
Collect data on several houses ( size of the house in [Link]) and their
selling price.
Can we use this data to predict the selling price of a specific house?
Some Regression Types
•Simple Linear Regression (y=Ax+C)
•Multiple linear regression (y=Ax1+Bx2+....+Fxn+ C)
•Nonlinear Regression (y= Ax12+Bx2+.....+C)
What is Linear Regression?
• It’s a Supervised Learning algorithm which goal is to predict continuous,
numerical values based on given data input.
• From the geometrical perspective, each data sample is a point.
• Linear Regression tries to find parameters of the linear function, so
the distance between the all the points and the line is as small as possible.
• Algorithm used for parameters update is called Gradient Descent.
Linear Regression
The Linear Regression model is a mathematical formula that
takes vector of numerical values (attributes of single data sample) as
an input and uses them to make a prediction.
Each vector of features is paired with target value (expected result
from formula).
Linear Regression
According to the algorithm, every feature has a weight parameter
assigned. It represents it’s importance to the model. The goal is to find
the values of weights so the following equation is met for every apartment
data.
The left side of the equation is a linear function. As manipulation of
weight values can change an angle of the line. Although, there is a
still one element missing. Current function is always going through (0,0)
point of the coordinate system.
The parameter is named bias and it gives the formula a freedom to
move on the y-axis up and down.
How we find the values of w0, w1, w2 and b?
1. Initialize with some random values
2. Find cost function (mean squared error (predicted output – actual output)
3. Use gradient descendant method and update values till error will be minimum.
General Hypothesis for linear regression
For better understanding we will use
linear regression with one variable
1. We are considering size of the house as independent variable (x)
2. Price of house as dependent variable (y)
3. We will be having data sets with one feature (size) and target (price)
4. We can draw these data as points in 2D space with size in x-axis and price in y-axis
5. We have to find values of parameters y and c such that y=mx+c which is the
hypothesis.
6. Initialize m and c with some random values
7. Find cost function (mean squared error (predicted output – actual output))
8. Use gradient descendant method and update the values of m and c till error will be
minimum
500
Housing Prices
(Opava) 400
300
Price 200
(in 1000s of
CZK) 100
0
0 500 1000 1500 2000 2500 3000
Size (feet2)
Supervised Learning Regression Problem
Given the “right answer” for each Predict real-valued output
example in the data.
Training Set How do we represent h ?
Learning Algorithm
Size of Estimated
h
house price
Linear regression with one variable.
Univariate linear regression.
Training set of Size in feet2 Price (CZK) in
housing prices (x) 1000's (y)
(Opava) 2104 460
1416 232
1534 315
852 178
… …
Notation:
m = Number of training examples
x’s = “input” variable / features
y’s = “output” variable / “target” variable
Training Set Size of the
Price ($) in
house in feet2
1000's (y)
(x)
2104 460
1416 232
1534 315
852 178
… …
Hypothesis:
‘s: Parameters
How to choose ‘s ?
3 3 3
2 2 2
1 1 1
0 0 0
0 1 2 3 0 1 2 3 0 1 2 3
Cost Function:
m is the number of training examples . J is called mean square error function
Idea: Choose so that is close to for our
training examples
Hypothesis: Simplified
Parameters:
Cost Function:
Goal:
Hypothesis:
Parameters:
Cost Function:
Goal:
(for fixed , this is a function of x) (function of the parameters )
500
400
Price ($)
in 1000’s 300
200
100
0
0 1000 2000 3000
Size in feet2 (x)
REGRESSION
Supervised Learning - Regression VS classification
Regression (linear and polynomial): relationship between the independent
variable and the dependent variable (e.g., predict continuous price based on size)
Classification: identifying to which of a set of categories a new
observation belongs
Inference Task of Linear Regression
There is unobserved variable between ice cream sale and shark attack ie
number of people coming to beach. It is more in summer hence more ice
cream sales and more shark attack. Such relationships are called spurious
Interaction Term: Radio X TV
K- NN