CHAPTER 5
ANSWERS
Very Short Answer Questions
1. Why is data preprocessing indispensable?
Answer:
Real-world data is often incomplete, inconsistent, and/or is lacking in certain behaviors or trends, and is
likely to contain many errors. Data preprocessing is a proven method of resolving such issues.
2. What are the different options available for cleaning and filling missing data?
Answer:
Inconsistent and dirty data can cause confusion and result in unreliable and poor output. Hence, the first
step in data preprocessing is data cleaning. The cleaning of data is done by filling in missing values,
smoothing noisy data by identifying and/or removing outliers, and resolving inconsistencies. The
following are some data cleaning routines.
Filling in Missing Values: Missing data elements in a database is a common problem in the real-world
scenario. The accuracy of predictions gets affected as missing data elements results in poor-quality
database. Thus, we need to fill in the missing data elements before the database is used for any
prediction or other data mining processes.
Missing data generally happens while people fill in details in online surveys of products. Some fields
in the form cannot be held mandatory as customers find it a nuisance and stop filling the feedback form,
which in turn affects data availability. Some customers share their experience, but not how long they are
using the product; some customers share how long they are using the product and their experience but
not their contact information. Thus, some part of the data is always missing, and it is very common in
real world scenario.
With reindexing, a DataFrame is made with the missing values in it. The output NaN implies it is “not
a number”. To detect missing values in a DataFrame, Pandas provides two functions on DataFrame
objects: isnull() and notnull().
Calculations with missing data: When the sum of the data is calculated, NaN will be treated as zero. If
the entries of data are all NaN, then the sum will be NaN.
Cleaning and Filling Missing Data: Pandas provides different methods for filling the missing values. The
fillna function can “fill in” NaN values with non-null data in different methods.
Replace NaN with a scalar value: Replacing NaN with a different value is very easy to implement. In
general, we tend to replace NaN with value 0. The following program illustrates how to replace NaN
with 0.
Fill NA forward and backward: Forward and backward filling involves filling the missing data with the
data objects above or below the NaN values. If the data object prior to the missing value is filled, then it
is called forward filling; and if the data object next to the missing value is filled, then it is called
backward filling.
Method Action
pad/fill Fill methods forward
bfill/backfill Fill methods backward
Drop Missing Values: If the missing values must be removed from the analysis, then we use
the dropna function along with axis argument. By default, axis = 0 (i.e., along a row), which means that if
any value within a row is NaN then the whole row is excluded.
Replace Missing (or) Generic Values: A generic value can be replaced with a specific value. This is
achieved by applying the replace method. Replacing NaN with a scalar value is the equivalent behavior
of the fillna() function.
3. Why is data smoothened?
Answer:
Data smoothing removes noise from a dataset, allowing important patterns to stand out. Data
smoothing can be done in a variety of different ways, including random, random walk, moving average,
simple exponential, linear exponential, and seasonal exponential smoothing.
4. Why is normalization of variables necessary?
Answer:
In most cases, when you normalize data you eliminate the units of measurement for data, enabling you
to more easily compare data from different places. It is required to make data uniform and comparison
will not result in wrong information generation.
5. How is data integration accomplished? What are the problems associated with data integration?
Answer:
Data integration merges data from multiple sources into a coherent data store. The most well-known
implementation of data integration is building an enterprise’s data warehouse. The benefit of a data
warehouse enables a business to perform analyses based on the data in the data warehouse. This would
not be possible to do on the data available only in the source system. The reason is that the source
systems may not contain corresponding data, even though the data are identically named, they may
refer to different entities.
Problems in Integration:
a) Quick and accurate understanding of data sources: You may start by profiling the data and
come to some general understanding but invariably will encounter a peculiar anomaly about the
data source – which will cause you to rework your integration efforts. This might take the form
of a data element that is intended for one purpose, but contains data content of something else.
b) Handling changes in data over time: This is one of the more complex aspects of data
integration, because different sources can be updated at different intervals. Sometimes source
systems can have very unusual processes for handling updates that can cause assumptions
typically seen in relational databases, such as referential integrity, to be invalid. To combine
data from multiple sources into a cohesive result, one must get down to the detailed field level
mapping of the multiple data sources – all of which have varying consistency, definitions, and
contexts.
c) Data mapping: Business analysis, domain expertise, and technical knowledge about source
systems are required when mapping data from one source to another. While differences in
naming conventions and data formats is a good first step, this task also requires understanding
the relationships of one dataset to another. The business rules embedded in the source system
that produce the data must be considered when applying transformation logic to create the
integrated dataset.
Short Answer Questions
1. What is the difference between covariance and correlation?
Answer:
Covariance is a measure of how much two random variables vary together. It is similar to variance, but
where variance tells you how a single variable varies, covariance tells you how two variables vary
together.
Correlation is defined as covariance normalized by the product of standard deviations, so the correlation
between X and Y is defined as
cov ( X , Y )
cor ( X , Y ) =
var ( X ) var (Y )
Covariance can range between −∞ and ∞ while correlation takes values in [−1, 1] (this is easily proved
with the Cauchy–Schwarz inequality). Two random variables have zero correlation if and only if they
have zero covariance.
2. How many covariance matrices have to be computed for a five-dimensional data?
Answer:
n!
For an n-dimensional dataset, we can calculate different covariance values.
( ( n − 2 )! × 2 )
When n = 5, we have to calculate 10 values.
3. What are the methods of estimating ICA?
Answer:
The two methods of estimating ICA are:
a) Non-linear decorrelation: Non-linear decorrelation method involves finding the matrix W so that for
any i ≠ j, the components yi and yj are completely uncorrelated. The transformed components g(yi)
and h(yj) are also uncorrelated, where g and h are some suitable non-linear functions. In this
method of estimating ICA, if the non-linearities are properly chosen, the method does find the
independent components. The main problem in this method is to address how the non-linearities g
and h are chosen. One of the approaches to select the non-linear functions is to use maximum
likelihood method in information theory.
b) Maximum non-Gaussianity: The maximum non-Gaussianity approach for estimating independent
component involves finding the local maxima of non-Gaussianity of a linear combination, y = Σbixi,
under the constraint that the variance of y is constant. Each local maximum corresponds to one
independent component. In practice, kurtosis is used to measure non-Gaussianity. Kurtosis is a
higher order cumulate method, which involves some ways of generalizations of variance using
higher order polynomials. Cumulants are used for ICA as they have important algebraic and
statistical properties.
4. How is mean deviation, standard deviation, and variance related to each other?
Answer:
Standard deviation is the square root of variance. It is expressed in the same units as mean, whereas the
variance is expressed in squared units.