CS345:
Practical Machine Leaning and Data Mining
1/2565
Supervised Learning
Supervised Learning
• Supervised learning is used whenever we want to predict a certain
outcome from a given input, and we have examples of input/output pairs.
• We build a machine learning model from these input/output pairs, which
comprise our training set.
• Our goal is to make accurate predictions for new, never-before-seen data.
• Supervised learning often requires human effort to build the training set,
but afterward automates and often speeds up an otherwise laborious or
infeasible task.
Types of supervised machine learning problems
• Two major types :
1. Classification => predict a class label from a predefined list of possibilities. For
example,
• Classifying emails as either spam or not spam
• Classifying irises into one of three possible species
2. Regression => predict a continuous number, or a floating-point number in
programming terms. For example,
• Predicting a person’s annual income from their education, their age, and where they live
• predicting the yield of a corn farm given attributes such as previous yields, weather, and
number of employees working on the farm.
Generalization
Overfitting
Underfitting
Generalization of the model
• If a model is able to make accurate predictions on unseen data, we say it is
able to generalize from the training set to the test set.
• Assumptions:
• Future unseen data (test set) will have the same properties as the current training sets.
• Models that are accurate on the training set are expected to be accurate on the test set.
• But that may not happen if the trained model is tuned too specifically to the training set.
• We want to build a model that is able to generalize as accurately as possible.
Example data about boat selling
• Predict whether a customer will
buy a boat, given records of
previous boat buyers and
customers who we know are not
interested in buying a boat
• The goal is to send out
promotional emails to people
who are likely to actually make a
purchase, but not bother those
customers who won’t be
interested
Derived Rules from observation
• Many possible rules we could come up with that would explain
perfectly if someone in this dataset wants to buy a boat.
• “If the customer is older than 45 and has less than 3 children or is not
divorced, then they want to buy a boat.” On the data that is in the table, the
rule is perfectly accurate.
• People who are 66, 52, 53, or 58 years old want to buy a boat, while all others
don’t (No age appears twice in the data)
• Achieving 100 percent accuracy on the training set does not help us
there. We want to find a rule that will work well for new customers.
Overfitting and Underfitting
• Overfitting occurs when you fit a model too closely to the
particularities of the training set and obtain a model that works well
on the training set but is not able to generalize to new data.
• Underfitting occurs if your model is too simple—say, “Everybody who
owns a house buys a boat”—then you might not be able to capture all
the aspects of and variability in the data, and your model will do
badly even on the training set.
Overfit in regression
Training data Underfitting
Better fit
Overfitting
Overfit in classification
Training data Underfitting
Better fit
Overfitting
Overfit with k-NN classifiers
The trade-off between overfitting and underfitting
Relation of Model Complexity to Dataset Size
• The larger variety of data points your dataset contains, the more
complex a model you can use without overfitting.
• If we saw 10,000 more rows of customer data, and all of them complied with
the rule “If the customer is older than 45, and has less than 3 children or is not
divorced, then they want to buy a boat,”
• we would be much more likely to believe this to be a good rule than when it
was developed using only the 12 rows in previous Table.
• Having more data and building appropriately more complex models
can often work wonders for supervised learning tasks.
Some Sample Datasets
• Sythetic datasets from mglearn
• Real world dataset from sklearn
• Generate your own dataset - tools from sklearn
mglearn package
• mglearn is a package used for the following datasets
• It is a stand-alone package in case you really feel like you want to
install mglearn into your Python environment, for some reason or
another. You can install it by running
pip install mglearn
• in your terminal, or by running
!pip install mglearn
Forge Dataset
• A synthetic two-class classification dataset, which has two features.
• The following code creates a scatter plot visualizing all of the data points
in this dataset.
import mglearn
# generate dataset
X, y = [Link].make_forge()
# plot dataset
import matplotlib as plt
mglearn.discrete_scatter(X[:, 0], X[:, 1], y)
[Link](["Class 0", "Class 1"], loc=4)
[Link]("First feature")
[Link]("Second feature")
print("[Link]: {}".format([Link]))
[Link]: (26, 2)
Wave Dataset
• The synthetic wave dataset is used to illustrate regression algorithms,.
• It has a single input feature and a continuous target variable
• The plot shows the single feature on the x-axis and the regression
target (the output) on the y-axis:
X, y = [Link].make_wave(n_samples=40)
[Link](X, y, 'o')
[Link](-3, 3)
[Link]("Feature")
[Link]("Target")
The Wisconsin Breast Cancer dataset
• The real-world datasets that are included in scikit-learn.
• It records clinical measurements of breast cancer tumors.
• Each tumor is labeled as “benign” (for harmless tumors) or “malignant” (for
cancerous tumors).
• The task is to learn to predict whether a tumor is malignant based on the
measurements of the tissue.
from [Link] import load_breast_cancer cancer =
load_breast_cancer()
print("[Link](): \n{}".format([Link]()))
[Link]():
dict_keys(['feature_names', 'data', 'DESCR', 'target', 'target_names'])
• The dataset consists of 569 data points, with 30 features each:
print("Shape of cancer data: {}".format([Link]))
Shape of cancer data: (569, 30)
• Of these 569 data points, 212 are labeled as malignant and 357 as benign:
print("Sample counts per class:\n{}".format(
{n: v for n, v in zip(cancer.target_names, [Link]([Link]))}))
Sample counts per class: {'benign': 357, 'malignant': 212}
Notes:
• The zip() function takes iterables (can be zero or more), aggregates
them in a tuple, and return it.
•
Ref : [Link]
• To get a description of the semantic meaning of each feature, we can have a
look at the feature_names attribute:
print("Feature names:\n{}".format(cancer.feature_names))
Feature names:
['mean radius' 'mean texture' 'mean perimeter' 'mean area’
'mean smoothness' 'mean compactness' 'mean concavity’
'mean concave points' 'mean symmetry' 'mean fractal dimension’
'radius error' 'texture error' 'perimeter error' 'area error’
'smoothness error' 'compactness error' 'concavity error’
'concave points error' 'symmetry error' 'fractal dimension error’
'worst radius' 'worst texture' 'worst perimeter' 'worst area’
'worst smoothness' 'worst compactness' 'worst concavity’
'worst concave points' 'worst symmetry' 'worst fractal dimension']
The Boston Housing dataset
• The real-world regression datasets
• The task is to predict the median value of homes in several Boston
neighborhoods in the 1970s, using information such as crime rate, proximity
to the Charles River, highway accessibility, and so on.
• The dataset contains 506 data points, described by 13 features:
from [Link] import load_boston
boston = load_boston()
print("Data shape: {}".format([Link]))
Data shape: (506, 13)
Simple Regression
Dataset
Simple Binary
Classification
Dataset
Complex Binary
Classification Dataset