21CSE355T - DATA MINING AND ANALYTICS

UNIT - 1
Why Data mining? What is Data mining ?
Kinds of data meant for mining
Kinds of patterns that can be mined
Applications suitable for data mining
Issues in Data mining
Data objects and Attribute types
Statistical descriptions of data
Need for data preprocessing and data quality
Data cleaning
Data integration
Data reduction
Data transformation
Data cube and its usage
Why Data mining?

• The Explosive Growth of Data: from terabytes to petabytes

• Data collection and data availability
• Automated data collection tools, database systems, Web, computerized society
• Major sources of abundant data
• Business: Web, e-commerce, transactions, stocks, …
• Science: Remote sensing, bioinformatics, scientific simulation, …
• Society and everyone: news, digital cameras, YouTube

• We are drowning in data, but starving for knowledge!

• “Necessity is the mother of invention”—Data mining—Automated analysis of massive data sets

3
What is Data mining ?

Data mining (knowledge discovery from data)

• Extraction of interesting (non-trivial, implicit, previously unknown and
potentially useful) patterns or knowledge from huge amount of data
• Data mining: a misnomer?

Alternative names
• Knowledge discovery (mining) in databases (KDD), knowledge extraction,
data/pattern analysis, data archeology, data dredging, information
harvesting, business intelligence, etc.

4
Knowledge discovery process
Data processing
1. Data cleaning (remove noise and inconsistent data)
2. Data integration (multiple data sources maybe
combined)
3. Data selection (data relevant to the analysis task
are retrieved from database)
4. Data transformation (data transformed or
consolidated into forms appropriate for mining)
(Done with data preprocessing-standardizing formats)
5. Data mining (an essential process where intelligent
methods are applied to extract data patterns)
6. Pattern evaluation (identify the truly interesting
patterns)
7. Knowledge presentation (mined knowledge is
presented to the user with visualization or
representation techniques)
5
Try me…

[Link]

6
What Kind of Data Can Be mined?
• Database-oriented data sets and applications
• Relational database, data warehouse, transactional database
• Advanced data sets and advanced applications.
• Object-Relational Databases
• Temporal Databases, Sequence Databases, Time-Series databases
• Spatial Databases and Spatiotemporal Databases
• Text databases and Multimedia databases
• Heterogeneous Databases and Legacy Databases
• Data Streams
• The World-Wide Web

7
Relational Databases
• DBMS – database management system,
contains a collection of
interrelated data’s from databases.
• Each table contains set of
attributes(columns) and set of tuples(rows),
with columns as attributes of data and rows
as records.
• Tables can be used to represent the
relationships between or among multiple
tables.
• Each tuple in the relational table represents
an object identified by a unique key and
described by a set of attribute values.
• Relational data can be accessed by database
queries (SQL).
8
Data Warehouses
•A repository of information
collected from multiple sources,
stored under a unified schema, and
that usually resides at a single site.

• Constructed via a process of data

cleaning, data integration, data
transformation, data loading and
periodic data refreshing.

• User can perform drill-down or roll-

up operation to view the data at
different degrees of summarization.

9
Transactional Databases
• Consists of a file where each record
represents a transaction.

• A transaction typically includes a

unique transaction ID and a list of the
items making up the transaction.

• Easy to identify items that are

frequently sold together.

• Market Basket Analysis enable you to

bundle groups of items together. Eg:
printers are commonly purchased
together with computers.
10
Object-Relational Databases
• Object-relational databases are constructed based on an object-relational data model.
• This model extends the relational model by providing a rich data type for handling complex
objects and object orientation.
• Inherits the essential concepts of object-oriented databases.
• AllElectronics example, objects can be individual employees, customers, or items.
• Data and code relating to an object are encapsulated into a single unit.
• Objects that share a common set of properties can be grouped into an object class.

• A set of variables that describe the objects.

• A set of messages that the object can use to communicate with other objects.
• A set of methods, where each method holds the code to implement a message.

11
Temporal Databases, Sequence Databases, and Time-
Series Databases
• A temporal database typically stores relational
data that include time-related attributes.

• A sequence database stores sequences of ordered

events, with or without a concrete notion of time.
E.g.: customer shopping sequences.

• A time-series database stores sequences of values

or events obtained over repeated measurements of
time (e.g., hourly, daily, weekly).
E.g.: data collected from the stock exchange, inventory
control.

12
Spatial Databases and Spatiotemporal Databases

• Spatial databases contain spatial-related information.

E.g.: geographic(map) databases commonly used in vehicle navigation and dispatching
systems.

• A spatial database that stores spatial objects that change with time
is called a spatiotemporal database.

13
Text Databases and Multimedia Databases
• Text databases are databases that
contain word descriptions for objects.
E.g.: product specifications, error or bug reports,
warning messages etc.

• Multimedia databases store image,

audio, and video data. They are used in
applications such as picture content-
based retrieval, voice-mail systems,
video-on-demand systems etc.

14
Heterogeneous Databases and Legacy Databases

• A heterogeneous database consists of a set of interconnected, autonomous

component databases.

• A legacy database is a group of heterogeneous databases that combines

different kinds of data systems.
E.g.: relational or object-oriented databases, hierarchical databases, network
databases, spread sheets, multimedia databases, or file systems.

15
Data Streams
• Many applications involve the generation and
analysis of a new kind of data, called stream data.
• where data flow in and out of an observation
platform (or window) dynamically.

Unique features
• huge or possibly infinite volume.
• dynamically changing.
• flowing in and out in a fixed order.
• allowing only one or a small number of scans.
• demanding fast (often real-time) response time.

E.g.: scientific and engineering data, time-series data,

stock exchange,video surveillance, network traffic,
weather or environment monitoring.
16
The World Wide Web
• Capturing user access patterns in such distributed
information environments is called Web usage mining
(or Weblog mining).

• Authoritative Web page analysis based on linkages

among Web pages can help rank Web pages based on
their importance, influence, and topics.

• Automated Web page clustering and classification

help group and arrange Web pages in a
multidimensional manner based on their contents.

• Web community analysis helps identify hidden Web

social networks and communities and observe their
evolution.
17
Kinds of pattern that can be mined

Data mining functionalities

1. Characterization and discrimination
2. The mining of frequent patterns, associations, and correlations
3. Classification and regression
4. Clustering analysis
5. Outlier analysis

Data Mining Tasks

• Descriptive mining tasks characterize the general properties of the data in
the database.
• Predictive mining tasks perform inference on the current data in order to
make predictions.
18
Data can be associated with classes or concepts.
E.g. classes of items – computers, printers, …
concepts of customers – bigSpenders, budgetSpenders, …

Data characterization
• summarizing the general characteristics of a target class of data.
E.g. summarizing the characteristics of customers who spend more than $1,000 a
year at AllElectronics.

Data Discrimination
• comparing the target class with one or a set of comparative classes.
• E.g. Compare the general features of software products whole sales increase by
10% in the last year with those whose sales decrease by 30% during the same
period
19
Mining Frequent Patterns, Associations and Correlations

• Frequent itemset: a set of items that frequently appear together in a transactional data
set. (e.g. milk and bread)

• Association Analysis: find frequent patterns

• E.g. a sample analysis result – an association rule:
buys(X, “computer”) => buys(X, “software”) [support = 1%, confidence = 50%]
[Support measures how often an item or set of items appear in a dataset, while confidence
measures how often a relationship between items is true. ]
• Correlation Analysis: additional analysis to find statistical correlations between
associated pairs

20
Classification and Prediction
Classification
• The process of finding a model that describes and
distinguishes the data classes or concepts, for the
purpose of being able to use the model to predict
the class of objects whose class label is unknown.

• The model can be represented in classification

(IF-THEN) rules, decision trees, neural
networks, etc.

Prediction
• Predict missing or unavailable numerical data
values.
21
• Classification (Predicts a
category)
• Categorical (discrete, unordered)
labels,

• Regression(Predicts a
numerical value)
• Models continuous-valued functions

22
Cluster Analysis

• Class label is unknown: group data to form new

classes

• Clusters of objects are formed based on the

principle of maximizing intra-class similarity &
minimizing interclass similarity
E.g. Identify homogeneous subpopulations of
customers. These clusters may represent
individual target groups for marketing.

23
Outlier Analysis & Evolution Analysis

Outlier Analysis
• Data that do no comply with the general behavior or
model.
• Outliers are usually discarded as noise or exceptions.
• Useful for fraud detection.
E.g. Detect purchases of extremely large amounts

Evolution Analysis
• Describes and models regularities or trends for objects
whose
behavior changes over time.
E.g. Identify stock evolution regularities for overall
stocks and for the stocks of
particular companies.

24
Try me…

[Link]

25
Applications Suitable for data Mining
• Highly application-driven
discipline

26
Business Intelligence
• Business intelligence (BI)
technologies provide historical,
current, and predictive views of
business operations.

27
A typical DM System Architecture
• Data mining is the process of discovering interesting
knowledge from large amounts of data stored in databases,
data warehouses or other important repositories.

• Database, data warehouse, WWW or other information

repository (store data)

• Database or data warehouse server (fetch and combine data)

• Knowledge base (turn data into meaningful groups according

to domain knowledge)

• Data mining engine (perform mining tasks)(like

Characterization, association, correlation analysis,
classification, prediction, cluster analysis, outlier analysis and
evolution analysis.

• Pattern evaluation module (find interesting patterns)

(integrated/interacts with Data Mining module)

• User interface (interact with the user)

28
[Link]

29
Issues in Data Mining
• Mining methodology and User interaction
Mining different kinds of knowledge
• DM should cover a wide spectrum of data analysis and knowledge discovery tasks
• Enable to use the database in different ways
• Require the development of numerous data mining techniques

Interactive mining of knowledge at multiple levels of abstraction

• Difficult to know exactly what will be discovered
• Allow users to focus the search, refine data mining requests
• Data cube(multidimensional data structure) can be used.
• Multidimensional data mining can be used.

Incorporation of background knowledge

• Guide the discovery process
• Allow discovered patterns to be expressed in concise terms and different levels
of abstraction.
Domain knowledge related to databases, such as integrity constraints and deduction rules speed up a data
mining process.
30
Issues in Data Mining
Data mining query languages and ad hoc data mining
• High-level query languages need to be developed
• Should be integrated with a DB/DW query language

Presentation and visualization of results

• Knowledge should be easily understood and directly usable.
• High level languages, visual representations or other expressive forms.
• Require the DM system to adopt the above techniques.

Handling noisy or incomplete data

• Require data cleaning methods and data analysis methods that can handle
noise.

Pattern evaluation – the interestingness problem

• How to develop techniques to access the interestingness of discovered patterns,
especially with subjective measures bases on user beliefs or expectations.
31
Issues in Data Mining
Performance Issues
• Efficiency and scalability
• Huge amount of data
• Running time must be predictable and acceptable
• Parallel, distributed and incremental mining algorithms
• Divide the data into partitions and processed in parallel
• Incorporate database updates without having to mine the entire
data again from scratch.

Issues relating to Diversity of Database Types

• Other database that contain complex data objects, multimedia data,
spatial data, etc.
• Expect to have different DM systems for different kinds of data
• Heterogeneous databases and global information systems
• Web mining becomes a very challenging and fast-evolving field in
data mining
32
Issues in Data Mining
Diversity of Database Types
• Handling complex types of data (handle structured, semi structured,
unstructured data)
• Mining dynamic, networked, and global data repositories

Data Mining and Society

• Social impacts of data mining
• Privacy-preserving data mining
• Invisible data mining
(E.g.: Online shopping)

33
Data Objects and Attribute Types
• A data object represents an entity.
E.g.: 1. Sales database - customers, store items, sales
2. Medical database - patients
[Link] database -students, professors,
courses.
• Data sets are made up of data objects.

• Data objects are typically described by attributes. Data

objects can also be referred to as samples, examples,
instances, data points, or objects.

• If the data objects are stored in a database, they are data

tuples.
34
Attributes
• Data objects are typically described by
attributes

• Represents a characteristic or feature of a data

object

• Observed values for a given attribute are

known as observations

• A set of attributes used to describe a given

object is called an attribute vector

35
Types Of Attributes
• Nominal
• Binary
• Ordinal
• Numeric
• Discrete
• Continuous

36
Nominal Attributes
• Nominal means “relating to names.”

• The values of a nominal attribute are

symbols or names of things.

• Also referred to as categorical.

• The values do not have any meaningful

order.

Marital status • values of a nominal attribute are symbols or

0 – Single “names of things,” it is possible to represent
1 – Married such symbols or “names” with numbers.
2 – divorced E.g.: Marital status, Zip code, Hair colour.
3 - widowed

37
Binary Attributes
• Two categories or states: 0 or 1
0 – absent
1 – present
• Binary attributes are referred to as Boolean if
the two states correspond to true and false.
Symmetric
• Both states are equally valuable and carry same
weight.
E.g.: Gender – Male & Female
Asymmetric
• Both states are not equally important
• rarest one, by 1 (e.g., Covid19 positive)
• other by 0 (e.g., Covid19 negative).
38
Ordinal Attribute
• An ordinal attribute is an attribute with
possible values that have a meaningful order
or ranking among them, but the magnitude
between successive values is not known.

• Order is important
E.g.: Grades {O, A+, A, B+, B, C}
professional rank {professors, associate,
assistant}
• Ordinal attributes are often used in surveys
for ratings.

39
Interval-Scaled Variables
• Interval-scaled attributes are measured on a scale
of equal-size units.

• The values of interval-scaled attributes have order

and can be positive, 0, or negative.

E.g.: temperature, Calendar dates.

Temperature of 20◦C is five degrees higher
than a temperature of 15◦C.
years 2002 and 2010 are eight years apart

• Interval-scaled attributes are numeric.

40
Ratio Scaled Attribute
• A ratio-scaled attribute is a numeric attribute with
an inherent zero-point.

• Value as being a multiple (or ratio) of another value.

• Values are ordered.

• Compute the difference between values, as well as

the mean, median, and mode.
E.g.: Growth of a bacteria population
Decay of a radioactive element
years of experience
Temperature in kelvin
(true zero-point (0◦K = −273.15◦C))

41
Discrete versus Continuous Attributes
• A discrete attribute has a finite or countably infinite set of
values, which may or may not be represented as integers.

E.g.: Attributes countably finite - hair color, smoker,

medical test, drink size have a finite number of
values
E.g.: Attribute countably infinite - customer ID, Zip
codes

• If an attribute is not discrete, it is continuous.

• Continuous attributes are typically represented as floating-

point variables.

42
[Link]
nd

43
Statistical Descriptions of Data
• Statistical descriptions can be used to identify
properties of the data and highlight which data
values should be treated as noise or outliers.

Measuring the Central Tendency: Mean, Median,

and Mode
• The most common and effective numeric measure
of the “center” of a set of data is the (arithmetic)
mean. Let x1,x2,...,xN be a set of N values or
observations, such as for some numeric attribute
X, like salary. The mean of this set of values is

44
Weighted arithmetic mean
• Some cases each value of xi in a set may be associated with a weight wi for i =
1,...,N.

• The weights reflect the significance, importance, or occurrence frequency

attached to their respective values.

• This is called the weighted arithmetic mean or the weighted average.

45
Drawbacks of Mean Calculation
• Sensitive to extreme values.
• Even a small number of extreme values can corrupt the mean
E.g.: The mean salary at a company may be substantially pushed
up by that of a few highly paid managers.

Solution
• Trimmed mean
• Mean obtained after chopping off values at the high and low extremes.
• Sort the values observed for salary
• Remove the top and bottom 2% before computing the mean

46
Median
• Middle value in a set of ordered data values.
• It is the value that separates the higher half
of a data set from the lower half.
If N is odd
• Then the median is the middle value of
the ordered set.
If N is even
• Then the median is not unique
• It is the two middlemost values and any
value in between.
• The median is expensive to compute when
we have a large number of observations.

47
Mode
• The mode - value that occurs most frequently in the
set.

• It is possible for the greatest frequency to

correspond to several different values, which results
in more than one mode.

• One mode – Unimodal

• Two modes – Bimodal
• Three modes – Trimodal

• Data set with two or more modes is multimodal.

• If each data value occurs only once, then there is no

mode.
48
Midrange
• The midrange can also be used to assess the central
tendency of a numeric data set.
• It is the average of the largest and smallest values in
the set.

• In a unimodal frequency curve with perfect

symmetric data distribution, the mean, median, and
mode are all at the same center value

Positively Skewed Data

• The mode occurs at a value that is smaller than the
median
Negatively Skewed Data
• The mode occurs at a value greater than the median

49
Range, Quartiles, Outliers and Boxplots
• The range of the set is the difference between the
largest (max()) and smallest (min()) values.

• The most commonly used percentiles other than

the median are quartiles.

• The distance between the first and third quartiles

is a simple measure of spread that gives the range
covered by the middle half of the data.

• This distance is called the interquartile range

(IQR) and is defined as
IQR = Q3 - Q1
50
Five-Number Summary, Boxplots, and Outliers
Five-Number summary
• The five-number summary of a distribution consists of the
median (Q2), the quartiles Q1 and Q3, and the smallest and
largest individual observations, written in the order of
Minimum, Q1, Median, Q3, Maximum.

Boxplots
• Boxplots are a popular way of visualizing a
distribution. A boxplot incorporates the five-number
summary as follows:
✔ Typically, the ends of the box are at the quartiles so
that the box length is the interquartile range.
✔ The median is marked by a line within the box.
✔ Two lines (called whiskers) outside the box extend
to the smallest (Minimum) and largest (Maximum)
observations.
51
Five-Number Summary, Boxplots

52
Outliers
Outliers stand for data points that are indicative of a much higher variability than other
observations in a given dataset.

This can result in skewing statistical studies and wrong conclusions after all the
variables are not adequately identified and handled.

Identifications of outliers are very relevant for the financial sector, healthcare industry
and decision-making processes that depend on data analysis.

A common rule of thumb for identifying suspected outliers is to single out values
falling at least 1.5 × IQR above the third quartile or below the first quartile.

53
 Outliers Example

Steps to find Outliers

54
Try…

[Link]
nd

55
Variance and Standard Deviation
•

56
Graphic Displays
• Graphic displays include quantile plots, quantile–
Quantile Plot quantile plots, histograms, and scatter plots.

Quantile Plot
• A quantile plot is a simple and effective way to look at
a univariate data distribution. First, it displays all of
the data for the given attribute. Second, It plots
quantile information.
Quantile–Quantile Plot

Quantile–Quantile Plot
• A quantile–quantile plot, or q-q plot, graphs the
quantiles of one univariate distribution against the
corresponding quantiles of another

57
Graphic Displays
Histograms
• “Histos” means pole, and “gram” means chart, so a
histogram is a chart of poles.
• If X is nominal, such as automobile model or item type,
then a pole or vertical bar is drawn for each known value
of X.
• The resulting graph is more commonly known as a bar
chart.
• If X is numeric, the term histogram is preferred.
• The range of values for X is partitioned into disjoint
consecutive subranges. The subranges, referred to as
buckets or bins. The range of a bucket is known as the
width.

58
Graphic Displays
Positive Correlation Negative Correlation
• A scatter plot is one of the most effective
graphical methods for determining if there
appears to be a relationship, pattern, or trend
between two numeric attributes.

• To construct a scatter plot, each pair of values is

treated as a pair of coordinates in an algebraic
No Correlation sense and plotted as points in the plane.

• Two attributes, X, and Y, are correlated if one

attribute implies the other. Correlations can be
positive, negative, or null.

59
Need for data preprocessing and data quality

• Data have quality if they satisfy the

requirements of the intended use.
Factors comprising Data Quality
✔ Accuracy
✔ Completeness Reasons of Bad Quality Data
✔ Consistency ✔ The data collection instruments used may be
✔ Timeliness faulty.
✔ Believability ✔ Disguised missing data.
✔ Interpretability ✔ Technology limitations.
✔ Inconsistencies in naming conventions
Bad Data Quality
• Incomplete
• Inaccurate or noisy
• Inconsistent

60
Data Cleaning
Real-world data
✔ Incomplete
✔ Noisy
✔ Inconsistent

Data cleaning (or data cleansing)

⮚ Fill in missing values
⮚ Smooth out noise while identifying outliers
⮚ Correct inconsistencies in the data.

61
Handle Missing Values
1. Ignore the tuple 4. Use a measure of central tendency for the
✔ usually done when the class label is attribute (mean or median) to fill in
missing. missing value.
✔ This method is not very effective, unless
5. Use the attribute mean or median for all
the tuple contains several attributes with
samples belonging to the same class as the
missing values.
given tuple.

2. Fill in the missing value manually 6. Use the most probable value to fill in the
✔ Time consuming missing value
✔ May not be feasible given a large data set
with many missing values

3. Use a global constant to fill in the missing

value
✔ Replace all missing attribute values by the
same constant such as a label like
“Unknown” or −∞ 62
Noisy Data
• Noise is a random error or variance in a measured variable.
Binning
• Binning methods smooth a sorted data value by consulting its
“neighborhood,” that is, the values around it.
• The sorted values are distributed into a number of “buckets,” or
bins.
Smoothing by bin means
• Each value in a bin is replaced by the mean value of the bin.

Smoothing by bin medians

• Each bin value is replaced by the bin median.

Smoothing by bin boundaries

• The minimum and maximum values in a given bin are identified as
the bin boundaries.
• Each bin value is then replaced by the closest boundary value.
63
Binning – Equal depth (frequency)
Equal-width (distance) partitioning
✔ Divides the range into N intervals of equal size: uniform
grid
✔ if A and B are the lowest and highest values of the
attribute, the width of intervals will be: W = (B –A)/N.
✔ The most straightforward, but outliers may dominate
presentation
✔ Skewed data is not handled well

Equal-depth (frequency) partitioning

✔ Divides the range into N intervals, each containing
approximately same number of data points
✔ Good data scaling
✔ Managing categorical attributes can be tricky

64
Binning – Equal width (distance)
• Sorted data is given in the order - 5,10,11,13,15,35,50,55,72,92,204,215

65
Noisy Data
Regression
• Linear regression - finding the “best” line to fit two attributes (or variables) so that one
attribute can be used to predict the other.
• Multiple linear regression - is an extension of linear regression, where more than two
attributes are involved, and the data are fit to a multidimensional surface.

Outlier analysis
• Outliers may be detected by clustering, for example, where similar values are organized into
groups, or “clusters.”

66
Data Cleaning as process
• Missing values, noise, and inconsistencies contribute to inaccurate data.
• The first step in data cleaning as a process is discrepancy detection.
Factors causes Discrepancies
✔ Poorly designed data entry forms
✔ Human error in data entry, deliberate errors
✔ Data decay
Rules
• Unique Rule - Each value of the given attribute must be different from all other values
for that attribute.
• Consecutive Rule - There can be no missing values between the lowest and highest
values for the attribute, and that all values must also be unique (e.g., check numbers).
• Null Rule - Use of blanks, question marks, special characters, or other strings that may
indicate the null condition.

67
Data Cleaning as process
Field overloading
• Another source of errors that typically results when developers squeeze new attribute
definitions into unused (bit) portions of already defined attributes.

Discrepancy detection tools

• Data scrubbing tools - use simple domain knowledge to detect errors and make corrections
in the data.

• Data auditing tools - find discrepancies by analyzing the data to discover rules and
relationships and detecting data that violate such conditions.

68
Give a try…

[Link]

69
Data Integration
Data integration
• Combines data from multiple sources into a coherent store
• Schema integration: e.g., [Link]-id = [Link]-#
• Integrate metadata from different sources

Entity identification problem

• Identify real world entities from multiple data sources, e.g., Bill Clinton = William Clinton

Detecting and resolving data value conflicts

• For the same real world entity, attribute values from different sources are different
• Possible reasons: different representations, different scales, e.g., metric vs. British units

70
Handling Redundancy in Data Integration
• Redundant data occur often when integration of multiple databases.

✔ Object identification: The same attribute or object may have different names in
different databases.
✔ Derivable data: One attribute may be a “derived” attribute in another table, e.g., annual
revenue.

• Redundant attributes may be able to be detected by correlation analysis.

• Careful integration of the data from multiple sources may help reduce/avoid redundancies
and inconsistencies and improve mining speed and quality.

71
Data Integration

• For numerical attributes, we can evaluate the correlation between two attributes,
A and B, by computing the correlation coefficient.

• If rA,B > 0, A and B are positively correlated (A’s values increase as B’s). The higher,
the stronger correlation.

• rA,B = 0: uncorrelated;

• rA,B < 0: negatively correlated.

72
Data Integration
Chi-Square Test
• For categorical (discrete) data, a correlation relationship between two
attributes, A and B, can be discovered by a (chi-square) test.

• where oij is the observed frequency (i.e., actual count) of the joint event
(Ai;Bj) and eij is the expected frequency of (Ai;Bj), which can be computed as

73
Chi-Square Calculation Example

e12 = 1200*450/1500 = 360

e21 = 300*1050/1500 = 210
e22 = 1200*1050/1500 = 840

74
Data Transformation
• Smoothing: which works to remove noise from the data. Such techniques include binning,
regression, and clustering.

• Aggregation: where summary or aggregation operations are applied to the data.

Eg: the daily sales data may be aggregated so as to compute monthly and annual total
amounts.

• Generalization of the data, where low-level or “primitive” (raw) data are replaced by higher-
level concepts through the use of concept hierarchies.
Eg: categorical attributes, like street, can be generalized to higher-level concepts, like
city or country.

• Numerical attributes, like age, may be mapped to higher-level concepts, like youth, middle-
aged, and senior.

75
Data Transformation
• Normalization: where the attribute data are scaled so as to fall within a small specified
range, such as - 1.0 to 1.0, or 0.0 to 1.0.

• Attribute construction (or feature construction),where new attributes are constructed and
added from the given set of attributes to help the mining process.

There are many methods for data normalization.

✔ Min-max normalization
✔ z-score normalization
✔ Normalization by decimal scaling.

76
Min-max normalization
• Min-max normalization preserves the relationships among the original data values.

Example
• Suppose that the minimum and maximum values for the attribute income are
$12,000 and $98,000, respectively. We would like to map income to the range
[0.0;1.0]. By min-max normalization, a value of $73,600 for income is transformed.

73600 – 12000/98000 – 12000 (1.0 – 0.0) +0 = 0.716

77
z-score normalization (or zero-mean normalization)
• The values for an attribute, A, are normalized based on the mean and standard
deviation of A.

• Where Ᾱ and σA are the mean and standard deviation, respectively, of attribute A.

Example
• Suppose that the mean and standard deviation of the values for the attribute income
are $54,000 and $16,000, respectively. With z-score normalization, a value of
$73,600 for income is transformed

73600 – 54000 / 16000 = 1.225

78
Normalization by decimal scaling
• Normalizes by moving the decimal point of values of attribute A. The number of decimal
points moved depends on the maximum absolute value of A.

• Suppose that the recorded values of A range from - 986 to 917. The maximum absolute value
of A is 986.

• To normalize by decimal scaling, we therefore divide each value by 1,000 (i.e., j = 3) so that
- 986 normalizes to – 0.986 and 917 normalizes to 0.917.

79
Give a try…

[Link]

80
Data Reduction
• Data reduction techniques can be applied to obtain a reduced representation of the data set
that is much smaller in volume.
• closely maintains the integrity of the original data.
Strategies for data reduction
✔ Data cube aggregation: where aggregation operations are applied to the data in the
construction of a data cube.
✔ Attribute subset selection: where irrelevant, weakly relevant, or redundant attributes or
dimensions may be detected and removed.
✔ Dimensionality reduction: where encoding mechanisms are used to reduce the data set
size.
✔ Numerosity reduction: where the data are replaced or estimated by alternative.
✔ Discretization and concept hierarchy generation: where raw data values for attributes are
replaced by ranges or higher conceptual levels.

81
Data cube and its usage

• Data cubes store multidimensional aggregated information.

• Each cell holds an aggregate data value, corresponding to the data point in
multidimensional space.
• Data cubes provide fast access to precomputed summarized data.
• Base Cuboid Vs Apex Cuboid. (the base cuboid is the most specific cuboid,
while the apex cuboid is the most generalized)

Data Cube
Data Cube Aggregation
82
Attribute Subset Selection

• Definition: Attribute Subset Selection refers to the process of selecting a subset of

relevant features (attributes) for use in model construction.
• Goal: To reduce the dimensionality of the data and improve computational
efficiency, accuracy, and model interpretability.
• Methods:
 Filter Methods: Ranking attributes based on statistical tests.
 Wrapper Methods: Using predictive models to evaluate feature subsets.
 Embedded Methods: Feature selection during model training (e.g., decision trees).
Importance of Attribute Subset Selection
• Improves Accuracy: Reduces noise and overfitting by removing irrelevant or
redundant features.
• Reduces Computational Cost: Fewer attributes mean faster algorithms.
• Enhances Interpretability: Easier to understand the models when fewer attributes are
used.

83