Scribd
Upload
25 views5 pages

Normalization Techniques in Machine Learning

The document discusses different techniques for normalizing data, including scaling to a range, clipping, log scaling, and z-scoring. It explains when each technique should be used, such as scaling for uniformly distributed data and clipping for outliers. The goal of normalization is to transform features to a similar scale to improve model performance and training stability.

Uploaded by

Arshad Ali
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
25 views5 pages

Normalization Techniques in Machine Learning

The document discusses different techniques for normalizing data, including scaling to a range, clipping, log scaling, and z-scoring. It explains when each technique should be used, such as scaling for uniformly distributed data and clipping for outliers. The goal of normalization is to transform features to a similar scale to improve model performance and training stability.

Uploaded by

Arshad Ali
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Normalization

The goal of normalization is to transform features to be on a similar


scale. This improves the
performance and training stability of the model.

Normalization Techniques at a Glance

Four common normalization techniques may be useful:

scaling to a range

clipping

log scaling

z-score

The following charts show the effect of each normalization technique on the
distribution of the raw
feature (price) on the left.
The charts are based on the data set from 1985 Ward's Automotive
Yearbook that
is part of the UCI Machine Learning Repository under Automobile Data
Set
 ([Link]

Figure 1. Summary of normalization techniques.

Scaling to a range

Recall from MLCC (/machine-learning/crash-course/representation/cleaning-data)


that scaling
 (/machine-learning/glossary#scaling)
means converting floating-point feature values from their natural
range (for
example, 100 to 900) into a standard range—usually 0 and 1 (or sometimes -1 to
+1). Use
the following simple formula to scale to a range:

\[ x' = (x - x_{min}) / (x_{max} - x_{min}) \]


keyboard_arrow_left
Scaling to a range is a good choice when both of the following conditions are
met:

You know the approximate upper and lower bounds on your data with
few or no outliers.

Your data is approximately uniformly distributed across that range.

A good example is age. Most age values falls between 0 and 90, and every part of
the range has a
substantial number of people.

In contrast, you would not use scaling on income, because only a few people
have very high
incomes. The upper bound of the linear scale for income would be
very high, and most people would
be squeezed into a small part of the scale.

Feature Clipping

If your data set contains extreme outliers, you might try feature
clipping, which caps all feature
values above (or below) a certain
value to fixed value. For example, you could clip all temperature
values
above 40 to be exactly 40.

You may apply feature clipping before or after other normalizations.

Formula: Set min/max values to avoid outliers.

Figure 2. Comparing a raw distribution and its clipped version.

Another simple clipping strategy is to clip by z-score to +-Nσ (for example, limit to
+-3σ). Note that σ
is the standard deviation.

Log Scaling

Log scaling computes the log of keyboard_arrow_left


your values to compress a wide range to a narrow
range.
\[ x' = log(x) \]

Log scaling is helpful when a handful of your values have many points, while
most other values have
few points. This data distribution is known as the power
law distribution. Movie ratings are a good
example. In the chart below, most
movies have very few ratings (the data in the tail), while a few
have lots of
ratings (the data in the head). Log scaling changes the distribution, helping to
improve
linear model performance.

Figure 3. Comparing a raw distribution to its log.

Z-Score

Z-score is a variation of scaling that represents the number of standard


deviations away from the
mean. You would use z-score to ensure your feature
distributions have mean = 0 and std = 1. It’s
useful when there are a few
outliers, but not so extreme that you need clipping.

The formula for calculating the z-score of a point, x, is as follows:

\[ x' = (x - μ) / σ \]

μ is the mean and σ is the standard deviation.

keyboard_arrow_left
Figure 4. Comparing a raw distribution to its z-score distribution.

Notice that z-score squeezes raw values that have a range of ~40000
down into a range from
roughly -1 to +4.

Suppose you're not sure whether the outliers truly are extreme.
In this case, start with z-score unless
you have feature values that
you don't want the model to learn; for example, the values are
the result
of measurement error or a quirk.

Summary

est normalization technique is one that


empirically works well, so try new ideas if you think they'll work well on
your
e distribution.

Normalization
Formula When to Use
Technique

Linear Scaling $$ x' = (x - x_{min}) / (x_{max} - When the feature is more-or-less uniformly distributed
x_{min}) $$ across a fixed range.

Clipping if x > max, then x' = max. if x < min, When the feature contains some extreme outliers.
then x' = min

Log Scaling x' = log(x) When the feature conforms to the power law.

Z-score x' = (x - μ) / σ When the feature distribution does not contain extreme
outliers.

keyboard_arrow_left
erms:

aling (/machine-learning/glossary#scaling) normalization (/machine-learning/glossary#normalization)

Previous

arrow_back Transforming Numeric Data


 (/machine-learning/data-prep/transform/transform-numeric)
Next
Bucketing
 (/machine-learning/data-prep/transform/bucketing) arrow_forward

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License
 ([Link] and code samples are licensed under the Apache 2.0 License
 ([Link] For details, see the Google Developers Site Policies
 ([Link] Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2021-06-13 UTC.

keyboard_arrow_left

Common questions

Powered by AI

Normalizing income data requires careful consideration of outliers. Techniques like scaling to a range may be inefficient due to extreme high values skewing scales; z-score could be compromised by outliers affecting mean and standard deviation calculations. Clipping might be more suitable to control the impact of outliers before applying other techniques for distribution consistency .

Scaling to a range adjusts data into a uniform scale, like 0 to 1, which is suitable for uniformly distributed data with known bounds. Conversely, z-score normalization re-scales data based on its mean and standard deviation to have a mean of 0 and standard deviation of 1, affecting the positioning but not changing extreme values or discovering hidden structure .

Applying z-score normalization on datasets with significant outliers can lead to a skewed distribution where outliers disproportionately affect the mean and standard deviations used in the calculation, potentially leading to biased model outputs .

Log scaling is suitable for data with features adhering to a power law distribution, compressing wide-ranged values, while z-score is best for data with minor outliers to standardize mean and variance. The choice depends on whether the goal is to narrow distribution range or to equalize based on deviation from mean .

The statement suggests that the choice of normalization method should be based on empirical evidence of improved model performance rather than a theoretical preference. This could mean testing multiple techniques and evaluating their impact on an actual dataset to determine which method optimizes the model outcomes most effectively .

Scaling to a range converts feature values into a standard range, typically between 0 and 1, which is effective when the data is uniformly distributed across a known range with few outliers. This is different from techniques like log scaling, which compresses wide ranges, or z-score normalization, which standardizes mean and variance .

Feature clipping might be applied to cap extreme values in a dataset, which can distort the effects of other normalization methods or when extreme outliers might skew the data distribution and impact model learning .

Log scaling is useful in cases where data conforms to a power law distribution, with a few high-frequency instances among low-frequency ones, such as movie ratings. This technique compresses a wide range of values into a narrower range, improving linear model performance by adjusting extreme values proportionally .

Clipping should be prioritized when dealing with extreme outliers that could negatively impact the data analysis or model learning. It limits the impact of these outliers by capping values at maximum or minimum levels, allowing for a cleaner application of other normalization methods .

Normalization techniques are used to transform features to be on a similar scale, which improves the performance and training stability of the model .

You might also like