Preprocessing data
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
George Boorman
Core Curriculum Manager, DataCamp
scikit-learn requirements
Numeric data
No missing values
With real-world data:
This is rarely the case
We will often need to preprocess our data first
SUPERVISED LEARNING WITH SCIKIT-LEARN
Dealing with categorical features
scikit-learn will not accept categorical features by default
Need to convert categorical features into numeric values
Convert to binary features called dummy variables
0: Observation was NOT that category
1: Observation was that category
SUPERVISED LEARNING WITH SCIKIT-LEARN
Dummy variables
SUPERVISED LEARNING WITH SCIKIT-LEARN
Dummy variables
SUPERVISED LEARNING WITH SCIKIT-LEARN
Dummy variables
SUPERVISED LEARNING WITH SCIKIT-LEARN
Dealing with categorical features in Python
scikit-learn: OneHotEncoder()
pandas: get_dummies()
SUPERVISED LEARNING WITH SCIKIT-LEARN
Music dataset
popularity : Target variable
genre : Categorical feature
print([Link]())
popularity acousticness danceability ... tempo valence genre
0 41.0 0.6440 0.823 ... 102.619000 0.649 Jazz
1 62.0 0.0855 0.686 ... 173.915000 0.636 Rap
2 42.0 0.2390 0.669 ... 145.061000 0.494 Electronic
3 64.0 0.0125 0.522 ... 120.406497 0.595 Rock
4 60.0 0.1210 0.780 ... 96.056000 0.312 Rap
SUPERVISED LEARNING WITH SCIKIT-LEARN
EDA w/ categorical feature
SUPERVISED LEARNING WITH SCIKIT-LEARN
Encoding dummy variables
import pandas as pd
music_df = pd.read_csv('[Link]')
music_dummies = pd.get_dummies(music_df["genre"], drop_first=True)
print(music_dummies.head())
Anime Blues Classical Country Electronic Hip-Hop Jazz Rap Rock
0 0 0 0 0 0 0 1 0 0
1 0 0 0 0 0 0 0 1 0
2 0 0 0 0 1 0 0 0 0
3 0 0 0 0 0 0 0 0 1
4 0 0 0 0 0 0 0 1 0
music_dummies = [Link]([music_df, music_dummies], axis=1)
music_dummies = music_dummies.drop("genre", axis=1)
SUPERVISED LEARNING WITH SCIKIT-LEARN
Encoding dummy variables
music_dummies = pd.get_dummies(music_df, drop_first=True)
print(music_dummies.columns)
Index(['popularity', 'acousticness', 'danceability', 'duration_ms', 'energy',
'instrumentalness', 'liveness', 'loudness', 'speechiness', 'tempo',
'valence', 'genre_Anime', 'genre_Blues', 'genre_Classical',
'genre_Country', 'genre_Electronic', 'genre_Hip-Hop', 'genre_Jazz',
'genre_Rap', 'genre_Rock'],
dtype='object')
SUPERVISED LEARNING WITH SCIKIT-LEARN
Linear regression with dummy variables
from sklearn.model_selection import cross_val_score, KFold
from sklearn.linear_model import LinearRegression
X = music_dummies.drop("popularity", axis=1).values
y = music_dummies["popularity"].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
kf = KFold(n_splits=5, shuffle=True, random_state=42)
linreg = LinearRegression()
linreg_cv = cross_val_score(linreg, X_train, y_train, cv=kf,
scoring="neg_mean_squared_error")
print([Link](-linreg_cv))
[8.15792932, 8.63117538, 7.52275279, 8.6205778, 7.91329988]
SUPERVISED LEARNING WITH SCIKIT-LEARN
Let's practice!
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
Handling missing
data
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
George Boorman
Core Curriculum Manager, DataCamp
Missing data
No value for a feature in a particular row
This can occur because:
There may have been no observation
The data might be corrupt
We need to deal with missing data
SUPERVISED LEARNING WITH SCIKIT-LEARN
Music dataset
print(music_df.isna().sum().sort_values())
genre 8
popularity 31
loudness 44
liveness 46
tempo 46
speechiness 59
duration_ms 91
instrumentalness 91
danceability 143
valence 143
acousticness 200
energy 200
dtype: int64
SUPERVISED LEARNING WITH SCIKIT-LEARN
Dropping missing data
music_df = music_df.dropna(subset=["genre", "popularity", "loudness", "liveness", "tempo"])
print(music_df.isna().sum().sort_values())
popularity 0
liveness 0
loudness 0
tempo 0
genre 0
duration_ms 29
instrumentalness 29
speechiness 53
danceability 127
valence 127
acousticness 178
energy 178
dtype: int64
SUPERVISED LEARNING WITH SCIKIT-LEARN
Imputing values
Imputation - use subject-matter expertise to replace missing data with educated guesses
Common to use the mean
Can also use the median, or another value
For categorical values, we typically use the most frequent value - the mode
Must split our data first, to avoid data leakage
SUPERVISED LEARNING WITH SCIKIT-LEARN
Imputation with scikit-learn
from [Link] import SimpleImputer
X_cat = music_df["genre"].[Link](-1, 1)
X_num = music_df.drop(["genre", "popularity"], axis=1).values
y = music_df["popularity"].values
X_train_cat, X_test_cat, y_train, y_test = train_test_split(X_cat, y, test_size=0.2,
random_state=12)
X_train_num, X_test_num, y_train, y_test = train_test_split(X_num, y, test_size=0.2,
random_state=12)
imp_cat = SimpleImputer(strategy="most_frequent")
X_train_cat = imp_cat.fit_transform(X_train_cat)
X_test_cat = imp_cat.transform(X_test_cat)
SUPERVISED LEARNING WITH SCIKIT-LEARN
Imputation with scikit-learn
imp_num = SimpleImputer()
X_train_num = imp_num.fit_transform(X_train_num)
X_test_num = imp_num.transform(X_test_num)
X_train = [Link](X_train_num, X_train_cat, axis=1)
X_test = [Link](X_test_num, X_test_cat, axis=1)
Imputers are known as transformers
SUPERVISED LEARNING WITH SCIKIT-LEARN
Imputing within a pipeline
from [Link] import Pipeline
music_df = music_df.dropna(subset=["genre", "popularity", "loudness", "liveness", "tempo"])
music_df["genre"] = [Link](music_df["genre"] == "Rock", 1, 0)
X = music_df.drop("genre", axis=1).values
y = music_df["genre"].values
SUPERVISED LEARNING WITH SCIKIT-LEARN
Imputing within a pipeline
steps = [("imputation", SimpleImputer()),
("logistic_regression", LogisticRegression())]
pipeline = Pipeline(steps)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
random_state=42)
[Link](X_train, y_train)
[Link](X_test, y_test)
0.7593582887700535
SUPERVISED LEARNING WITH SCIKIT-LEARN
Let's practice!
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
Centering and
scaling
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
George Boorman
Core Curriculum Manager
Why scale our data?
print(music_df[["duration_ms", "loudness", "speechiness"]].describe())
duration_ms loudness speechiness
count 1.000000e+03 1000.000000 1000.000000
mean 2.176493e+05 -8.284354 0.078642
std 1.137703e+05 5.065447 0.088291
min -1.000000e+00 -38.718000 0.023400
25% 1.831070e+05 -9.658500 0.033700
50% 2.176493e+05 -7.033500 0.045000
75% 2.564468e+05 -5.034000 0.078642
max 1.617333e+06 -0.883000 0.710000
SUPERVISED LEARNING WITH SCIKIT-LEARN
Why scale our data?
Many models use some form of distance to inform them
Features on larger scales can disproportionately influence the model
Example: KNN uses distance explicitly when making predictions
We want features to be on a similar scale
Normalizing or standardizing (scaling and centering)
SUPERVISED LEARNING WITH SCIKIT-LEARN
How to scale our data
Subtract the mean and divide by variance
All features are centered around zero and have a variance of one
This is called standardization
Can also subtract the minimum and divide by the range
Minimum zero and maximum one
Can also normalize so the data ranges from -1 to +1
See scikit-learn docs for further details
SUPERVISED LEARNING WITH SCIKIT-LEARN
Scaling in scikit-learn
from [Link] import StandardScaler
X = music_df.drop("genre", axis=1).values
y = music_df["genre"].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = [Link](X_test)
print([Link](X), [Link](X))
print([Link](X_train_scaled), [Link](X_train_scaled))
19801.42536120538, 71343.52910125865
2.260817795600319e-17, 1.0
SUPERVISED LEARNING WITH SCIKIT-LEARN
Scaling in a pipeline
steps = [('scaler', StandardScaler()),
('knn', KNeighborsClassifier(n_neighbors=6))]
pipeline = Pipeline(steps)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=21)
knn_scaled = [Link](X_train, y_train)
y_pred = knn_scaled.predict(X_test)
print(knn_scaled.score(X_test, y_test))
0.81
SUPERVISED LEARNING WITH SCIKIT-LEARN
Comparing performance using unscaled data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=21)
knn_unscaled = KNeighborsClassifier(n_neighbors=6).fit(X_train, y_train)
print(knn_unscaled.score(X_test, y_test))
0.53
SUPERVISED LEARNING WITH SCIKIT-LEARN
CV and scaling in a pipeline
from sklearn.model_selection import GridSearchCV
steps = [('scaler', StandardScaler()),
('knn', KNeighborsClassifier())]
pipeline = Pipeline(steps)
parameters = {"knn__n_neighbors": [Link](1, 50)}
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=21)
cv = GridSearchCV(pipeline, param_grid=parameters)
[Link](X_train, y_train)
y_pred = [Link](X_test)
SUPERVISED LEARNING WITH SCIKIT-LEARN
Checking model parameters
print(cv.best_score_)
0.8199999999999999
print(cv.best_params_)
{'knn__n_neighbors': 12}
SUPERVISED LEARNING WITH SCIKIT-LEARN
Let's practice!
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
Evaluating multiple
models
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
George Boorman
Core Curriculum Manager, DataCamp
Different models for different problems
Some guiding principles
Size of the dataset
Fewer features = simpler model, faster training time
Some models require large amounts of data to perform well
Interpretability
Some models are easier to explain, which can be important for stakeholders
Linear regression has high interpretability, as we can understand the coefficients
Flexibility
May improve accuracy, by making fewer assumptions about data
KNN is a more flexible model, doesn't assume any linear relationships
SUPERVISED LEARNING WITH SCIKIT-LEARN
It's all in the metrics
Regression model performance:
RMSE
R-squared
Classification model performance:
Accuracy
Confusion matrix
Precision, recall, F1-score
ROC AUC
Train several models and evaluate performance out of the box
SUPERVISED LEARNING WITH SCIKIT-LEARN
A note on scaling
Models affected by scaling:
KNN
Linear Regression (plus Ridge, Lasso)
Logistic Regression
Artificial Neural Network
Best to scale our data before evaluating models
SUPERVISED LEARNING WITH SCIKIT-LEARN
Evaluating classification models
import [Link] as plt
from [Link] import StandardScaler
from sklearn.model_selection import cross_val_score, KFold, train_test_split
from [Link] import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from [Link] import DecisionTreeClassifier
X = [Link]("genre", axis=1).values
y = music["genre"].values
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = [Link](X_test)
SUPERVISED LEARNING WITH SCIKIT-LEARN
Evaluating classification models
models = {"Logistic Regression": LogisticRegression(), "KNN": KNeighborsClassifier(),
"Decision Tree": DecisionTreeClassifier()}
results = []
for model in [Link]():
kf = KFold(n_splits=6, random_state=42, shuffle=True)
cv_results = cross_val_score(model, X_train_scaled, y_train, cv=kf)
[Link](cv_results)
[Link](results, labels=[Link]())
[Link]()
SUPERVISED LEARNING WITH SCIKIT-LEARN
Visualizing results
SUPERVISED LEARNING WITH SCIKIT-LEARN
Test set performance
for name, model in [Link]():
[Link](X_train_scaled, y_train)
test_score = [Link](X_test_scaled, y_test)
print("{} Test Set Accuracy: {}".format(name, test_score))
Logistic Regression Test Set Accuracy: 0.844
KNN Test Set Accuracy: 0.82
Decision Tree Test Set Accuracy: 0.832
SUPERVISED LEARNING WITH SCIKIT-LEARN
Let's practice!
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
Congratulations
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N
George Boorman
Core Curriculum Manager, DataCamp
What you've covered
Using supervised learning techniques to build predictive models
For both regression and classification problems
Underfitting and overfitting
How to split data
Cross-validation
SUPERVISED LEARNING WITH SCIKIT-LEARN
What you've covered
Data preprocessing techniques
Model selection
Hyperparameter tuning
Model performance evaluation
Using pipelines
SUPERVISED LEARNING WITH SCIKIT-LEARN
Where to go from here?
Machine Learning with Tree-Based Models in Python
Preprocessing for Machine Learning in Python
Model Validation in Python
Feature Engineering for Machine Learning in Python
Unsupervised Learning in Python
Machine Learning Projects
SUPERVISED LEARNING WITH SCIKIT-LEARN
Thank you!
S U P E R V I S E D L E A R N I N G W I T H S C I K I T- L E A R N