"Feature engineering is the process of

transforming raw data into features that better

represent the underlying problem to the
predictive models, resulting in improved
model accuracy on unseen data."
– Jason Brownlee
 “Coming up with features is difficult,
time-consuming,
requires expert knowledge.
'Applied machine learning' is basically
feature engineering.”
– Andrew Ng
The Dream...

Raw data Dataset Model Task

 … The Reality

?
Features
ML Ready
dataset
? Task
Model

Raw data
 … The Reality
ML Ready Model
dataset

Deep Learning ? Task

Raw data But what architecture?

Scikit-learn can help a lot around here

 What is Scikit-Learn?
Extensions to SciPy (Scientific Python) are
called SciKits. SciKit-Learn provides machine
learning algorithms and much more.
● Algorithms for supervised & unsupervised learning
● Built on SciPy and Numpy
● Standard Python API interface
● Sits on top of c libraries, LAPACK, LibSVM, and Cython
● Open Source: BSD License (part of Linux)

Probably the best general ML framework out there.

More on scikit-learn
- Data loading and pre-processing

- Feature engineering

- Hyperparameter tuning and Model evaluation

- Pipelines
 DataLoading
Data Science and
Steps
pre-processing
● Data Preparation / pre-processing (this is where the magic comes ) [scikit-learn (numpy, Pandas)]
Data cleaning, inputs for missing values, features normalization, outliers detection
Features augumentation / selection / construction / reduction
- Transformers (fit -> transform)
- [Link] (deal with missing values)
- [Link] (features/objects => axis=1/0)
- [Link] (too many correlated features)
- [Link] (transform feature space, e.g., numeric -> Binary)
- [Link] (feature combinations)
- [Link] (transform to cum. distribution)
● Exploration (iterates with pre-processing) [scikit-learn / matplotlib / Seaborn…]
Get some insights on the data, understand its structure, create initial hypothesis
Clustering algorithms, histograms, plots, correlation…
[Link] -> Agglomerative clustering
[Link]
[Link]
Visualization (matplotlib, seaborn,...)
[Link]
...
Data pre-processing and transformation
- Inputing for NaNs
- Normalization / standardization
- Transform nominals
- Transform numeric
- Data Aggregation
- Feature extraction / combination

Mostly Transformer design patterns

- Fit on train data
- Transform both train and test data
 Load dataset
- [Link]
- Some „toy examples“ ready to use
- datasets.load_iris()
- datasets.load_wine()
- Generate artificial data
>>> from [Link] import load_digits

>>> digits = load_digits()

>>> digits .[Link]
(1797, 64)

>>> digits .target_names

[0 1 2 3 4 5 6 7 8 9]
>>> import [Link] as plt

>>> plt .gray()

>>> plt .matshow([Link][0])
>>> plt .show()
 Imputation for missing values
● Datasets contain missing values, often encoded as blanks, NaNs or other
placeholders

● Ignoring rows and/or columns with missing values is possible, but at the price of
loosing data which might be valuable

● Better strategy is to infer them from the known part of data

● Strategies

○ Mean: Basic approach

○ Median: More robust to outliers

○ Mode: Most frequent value

○ Using a model: Can expose algorithmic bias

Imputation for missing values

>>> import numpy as np

>>> from [Link] import Imputer
>>> imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
>>> [Link]([[1, 2], [[Link], 3], [7, 6]])
Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean',
verbose=0)
>>> X = [[[Link], 2], [6, [Link]], [7, 6]]
>>> print([Link](X))
[[ 4. 2. ]
[ 6. 3.666...]
[ 7. 6. ]]

Missing values imputation with scikit-learn

 Binarization
● Transform discrete or continuous numeric features in binary features
Example: Number of user views of the same document
>>> from sklearn import preprocessing
>>> X = [[ 1., -1., 2.],
... [ 2., 0., 0.],
... [ 0., 1., -1.]]

>>> binarizer =
[Link](threshold=1.0)
>>> [Link](X)
array([[ 1., 0., 1.],
[ 1., 0., 0.],
[ 0., 1., 0.]])
Binarization with scikit-learn
 Binning

● Split numerical values into bins and encode with a bin ID

● Can be set arbitrarily or based on distribution
● Fixed-width binning
Does fixed-width binning make sense for this long-tailed distribution?

>>> x = [Link]([0.2, 6.4, 3.0, 1.6])

>>> bins = [Link]([0.0, 1.0, 2.5, 4.0, 10.0])
>>> inds = [Link](x, bins)
>>> inds
array([1, 4, 3, 2])

Most users (458,234,809 ~ 5*10^8) had only 1 pageview during the period.

[Link]
 Binning

● Adaptative or Quantile binning

Divides data into equal portions (eg. by median, quartiles, deciles)

>>> deciles = dataframe['review_count'].quantile([.1, .2, .3, .4, .5, .6, .7,

.8, .9])
>>> deciles
0.1 3.0
0.2 4.0
0.3 5.0
0.4 6.0
0.5 8.0
0.6 12.0
0.7 17.0
0.8 28.0
0.9 58.0

Quantile binning with Pandas

Scaling
● Models that are smooth functions of input features (or utilize similarity) are
sensitive to the scale of the input (eg. Linear Regression, KNN)
● Scale numerical variables into a certain range, dividing values by a
normalization constant (no changes in single-feature distribution)

● Popular techniques

○ MinMax Scaling

○ Standard (Z) Scaling

○ L2 normalization
 Min-max scaling

● Squeezes (or stretches) all values within the range of [0, 1] to add robustness to
very small standard deviations and preserving zeros for sparse data.

>>> from sklearn import preprocessing

>>> X_train = [Link]([[ 1., -1., 2.],
... [ 2., 0., 0.],
... [ 0., 1., -1.]])
...
>>> min_max_scaler =
[Link]()
>>> X_train_minmax =
min_max_scaler.fit_transform(X_train)
>>> X_train_minmax
array([[ 0.5 , 0. , 1. ],
[ 1. , 0.5 , 0.33333333],
[ 0. , 1. , 0. ]])
Min-max scaling with scikit-learn
 Standard (Z) Scaling
After Standardization, a feature has mean of 0 and variance of 1 (assumption of
many learning algorithms)
>>> from sklearn import preprocessing
>>> import numpy as np
>>> X = [Link]([[ 1., -1., 2.],
... [ 2., 0., 0.],
... [ 0., 1., -1.]])
>>> X_scaled = [Link](X)
>>> X_scaled
array([[ 0. ..., -1.22..., 1.33...],
[ 1.22..., 0. ..., -0.26...],
[-1.22..., 1.22..., -1.06...]])
>> X_scaled.mean(axis=0)
array([ 0., 0., 0.])
>>> X_scaled.std(axis=0)
array([ 1., 1., 1.])

Standardization with scikit-learn

 Normalization
● Scales individual samples (rows) to have unit vector, dividing values by
vector’s L2 norm, a.k.a. the Euclidean norm
● Useful for quadratic form (like dot-product) or any other kernel to quantify
similarity of pairs of samples. This assumption is the base of the Vector
Space Model often used in text classification and clustering contexts

Normalized vector

Euclidean (L2) norm

Normalization

>>> from sklearn import preprocessing

>>> X = [[ 1., -1., 2.],
... [ 2., 0., 0.],
... [ 0., 1., -1.]]
>>> X_normalized = [Link](X, norm='l2')
>>> X_normalized
array([[ 0.40..., -0.40..., 0.81...],
[ 1. ..., 0. ..., 0. ...],
[ 0. ..., 0.70..., -0.70...]])

Normalization with scikit-learn

 Log transformation
Compresses the range of large numbers and expand the range of small numbers.
Eg. The larger x is, the slower log(x) increments.
 Log transformation
Smoothing long-tailed data with log

Histogram of # views by user Histogram of # views by user

smoothed by log(1+x)
Feature extraction
 Interaction Features

>>> import numpy as np

>>> from [Link] import PolynomialFeatures
>>> X = [Link](6).reshape(3, 2)
>>> X
array([[0, 1],
[2, 3],
[4, 5]])
>>> poly = poly = PolynomialFeatures(degree=2, interaction_only=False,
include_bias=True)
>>> poly.fit_transform(X)
array([[ 1., 0., 1., 0., 0., 1.],
[ 1., 2., 3., 4., 6., 9.],
[ 1., 4., 5., 16., 20., 25.]])
Polynomial features with scikit-learn
 One-Hot Encoding (OHE)

● Transform a categorical feature with m possible values into m binary features.

● If the variable cannot be multiple categories at once, then only one bit in the
group can be on.

● Sparse format is memory-friendly

● sklearn.feature_extraction.DictVectorizer
Large Categorical Variables

● Common in applications like targeted advertising and fraud detection

● Example:

Some large categorical features from Outbrain Click Prediction competition

 Feature hashing

● Hashes categorical values into vectors with fixed-length.

● Lower sparsity and higher compression compared to OHE

● Deals with new and rare categorical values (eg: new user-agents)

● May introduce collisions

100 hashed columns

[Link]
 Bin-counting

● Instead of using the actual categorical value, use a global statistic of this
category on historical data.

● Useful for both linear and non-linear algorithms

● May give collisions (same encoding for different categories)

● Be careful about leakage

● Strategies

○ Count

○ Average CTR
Bin-counting

or or

Counts Click-Through Rate

P(click | ad) = ad_clicks / ad_views
 Textual features

● sklearn.feature_extraction.text
● Count, Tf-IDF,

Image features

● sklearn.feature_extraction.image
● Patches, img to graph,…
Feature selection
 Feature selection

● Decrease complexity of model

● Remove redundant and irrelevant features
[Link]

● sklearn.feature_selection.SelectKBest
● Scoring function
● K features
● Threshold
● …
Data exploration (Exploratory data analysis, EDA)

- Get the knowledge about your data

- What does the data model look like?

- What is the features distribution?
- What are the features with missing or
inconsistent values?
- What are the most predictive features?
Exploratory Data Analysis (EDA)
- Get the knowledge about your data

- get a basic description of the data

- visualize it
- identify patterns in it
- identify challenges of using the data
- form hypothesis about the data
- …
Exploratory Data Analysis (EDA)
Basic knowledge
-Describe data (quantiles, med, mean, max, min…)
- head(), tail()
- [Link](), sorting,
Challenges
-NaNs, outliers, numeric -> nominal and vice versa
Patterns
- correlation, covariation, linear dependence,
clusters, dimensionality reduction…
Visualization
- boxpolots, histograms, cummulative distribution,
pairplots, scatter plots, corr matrices, …
Patterns
[Link]
- Pairwise correlation for dataframe columns

[Link]
- Linear regression (Estimator class, fit and predict methods)

[Link]
- Several variants of clustering, perhaps hierarchical one would be most appropriate

[Link]
- Linear dimensionality reduction (are my features correlated?)
Visualization
Matplotlib
- Basics

Seaborn
- Extends over matplotlib (compatible calls)
- Easier processing
- Nicer graphs by default
- Several additional graph types (pairplot, facetedGrid) especially useful
- [Link]

Matplotlib slides follows

More reading:
[Link]
[Link]
Supervised ML algorithms
(just the very basics now, more later)
 Classification

Given labeled input data (with two or more labels), fit a

function that can determine for any input, what the label is.
 Regression

Given continuous input data fit a function that is able to

predict the continuous value of input given other data.
K-nearest neighbors
Linear regression
Exploratory Data Analysis (EDA) - pairplot
import pandas as pd
import seaborn as sns #statistical data visualization, based on matplotlib
import [Link] as plt

df = pd.read_csv("C:/Users/peska/Documents/uceni/2017_2018/NPRG065/cviceni/[Link]", delimiter=";")

[Link](data=df,
x_vars= ["age","Medu","Fedu","famrel","freetime","goout","absences","G1","G2"],
y_vars= ['G3'])
[Link]()
Exploratory Data Analysis (EDA) - CDF
import pandas as pd
import [Link] as plt

df = pd.read_csv("C:/Users/peska/Documents/uceni/2017_2018/NPRG065/cviceni/[Link]", delimiter=";")

[Link](cumulative=True, normed=1, bins=len(df.G3))

#[Link](cumulative=True, normed=1, bins=len([Link]))
[Link]()
Model Fitting & Evaluation
 Model Fitting Protocol
How to learn model’s hyperparameters: grid search & cross-validation

Aggregate results
Instead of Train – Test split, you may use additional „outer“
cross-validation
Get results from all parts of the dataset
Never use any knowledge of the test set data
E.g. For mean ratings, object similarities etc
 Model Fitting Protocol
Further variants:
- Monte-Carlo cross validation:
- random splits, arbitrary size (cold start problems)

- Bootstrap validation:
-only one split (if you have abundant data)

-Temporal splits:
-if domain changes over time
accuracy =
t r ue p o s i t i ve s + true
negatives / t o t a l

precision =
true positives / (true
positives + falsepositives)

recall =
true positives / (false
negatives + t r ue p o s i t i v e s )

F1 score =
2 * ( ( precision * r e c a l l) /
(precision + r e c a l l ) )

[Link]
 ROC (reciever-operator curve)

True Positive Rate /

False Positive Rate

AUC (AUROC) = area under ROC

- probability that random positive example
will be ranked higher than random
negative
- calculate empirically from test results

AUPR (area under precision-recall curve)

[Link]
[Link]
[Link]/stable/modules/generated/[Link]#skle
[Link]
 MSE & Coefficient of Determination
MAE / MSE / RMSE
MSE = [Link]((predicted-expected)**2)

Coefficient of determination
R2 is a predictor of “goodness of fit” and is a value ∈
[0,1] where 1 is perfectfit.
-the proportion of the variance in the dependent variable that is
predictable from the independent variable(s)
-independent from the scale of the output feature
Evaluation - sklearn
sklearn.model_selection
-GridSearchCV
-train_test_split()
-LeaveOneOut (predict for each example separately)

- Split data / Learn hyperparameters / Validate metrics

Evaluation - sklearn
[Link]: metric(true, predicted/rank,…)
Classification
-classification_report()
-precision, recall, auc, f1_score, jaccard,…
-confusion matrix

Regression
-r2_score(), MAE, MSE

Clustering
Pairwise
- cosine / euklidean distance, …
…
-[Link]
 Crossdatasets
from sklearn import svm, Validation in Scikit-Learn
from sklearn.model_selection import GridSearchCV, train_test_split
from [Link] import classification_report

iris = datasets.load_iris()
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svc = [Link]()
X = [Link]
y = [Link]
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=0.3, random_state=0)

clf = GridSearchCV(svc, parameters)

[Link](X,y)
print(clf.cv_results_['mean_test_score'])
print(clf.cv_results_['params'])
print(clf.best_params_)

y_pred = [Link](X_test)
print(classification_report(y_test, y_pred))
Other Evaluation
How to evaluate clusters?
Visualization (but only in 2D)
 Unpredictable Future
Machine learning models attempt to predict the future
as new inputs come in - but human systems and
processes are subject to change.

Solution: Precision/Recall tracking over time

Pipelines
Pipelines
[Link](steps)
- Sequentially apply repeatable transformations to final
estimator that can be validated at every step.
- Each step (except for the last) must implement
Transformer, e.g. f i t and transform methods.
- Pipeline itself implements both methods of
Transformer and Estimator interfaces.
>>> from [Link] import PolynomialFeatures
>>> from [Link] import make_pipeline
>>> model = make_pipeline(PolynomialFeatures(2), linear_model.
Ridge())
>>> m o d e l . f i t ( X _ t r a i n , y _ t r a i n )

>>> mean_squared_error(y_test, model. predict(X_test))

3.1498887586451594

>>> [Link](X_test, y_t e s t )

0.97090576345108104

Pipelined Model