Data Exploration and Data

Preprocessing
• Data Mining
• In general terms, “Mining” is the process of extraction of some valuable
material from the earth e.g. coal mining, diamond mining, etc.
• In the context of computer science, “Data Mining” can be referred to
as knowledge mining from data, knowledge extraction, data/pattern
analysis, data archaeology, and data dredging.
• It is basically the process carried out for the extraction of useful
information from a bulk of data or data warehouses.
• We can think of Data Mining as a step in the process of Knowledge
Discovery or Knowledge Extraction.
• We can say it is a process of extracting interesting knowledge from large
amounts of data.
• That is stored in many data sources. Such as file systems, databases, data
warehouses. Also, knowledge used to contributes a lot of benefits to
business and individual.
• Nowadays, data mining is used in almost all places where a large
amount of data is stored and processed.
• For example, banks typically use ‘data mining’ to find out their
prospective customers who could be interested in credit cards,
personal loans, or insurance as well.
• Since banks have the transaction details and detailed profiles of
their customers, they analyze all this data and try to find out patterns
that help them predict that certain customers could be interested in
personal loans, etc.

Customers Data Credit

Transaction Mining Card(Yes/
s Techniqu No)
e
• Data->Information->knowledge

Data Informatio Knowledg

n e
Known facts about entity , Event ,
transaction etc.
Data is unorganized and unprocessed facts
Knowledge:
• Heart Disease Detection: Problem Area
• Patients (weight, height, sugar, bp, BMI, HeartDisease)
• 80 5 120 100 5.5 yes Information
• 70 5.1 100 80 5.0 No

• weight, height, sugar, bp, BMI, HeartDisease

• 60 4.8 100 95 5.0 ? Knowledge
Data Mining Architecture
• Data mining Architecture system contains many components.
• That is a data source, data warehouse server, data mining engine, and
knowledge base.
a. Data Sources
• There are so many documents present.
• That is a database, data warehouse, World Wide Web (WWW).
• That are the actual sources of data. Sometimes, data may reside even in plain text
files or spreadsheets.
• World Wide Web or the Internet is another big source of data.

b. Database or Data Warehouse Server

• The database server contains the actual data that is ready to be processed.
• Hence, the server handles retrieving the relevant data. That is based on the data
mining request of the user.
c. Data Mining Engine
•In data mining system data mining engine is the core component.
•As It consists a number of modules.
•That we used to perform data mining tasks. That includes association,
classification, characterization, clustering, prediction, etc.
d. Pattern Evaluation Modules
•This module is mainly responsible for the measure of interestingness of the
pattern. For this, we use a threshold value.
•Also, it interacts with the data mining engine. That’s main focus is to search
towards interesting patterns.
e. Graphical User Interface
•We use this interface to communicate between the user and the data mining
system.
•Also, this module helps the user use the system easily and efficiently. They
don’t know the real complexity of the process.
•When the user specifies a query, this module interacts with the data mining
system.
•Thus, displays the result in an easily understandable manner.
• What is Data Mining?
• It is a process of discovering hidden valuable knowledge by analyzing a
large amount of data. Also, we have to store that data in different
databases.
• As data mining is a very important process.
• It becomes an advantage for various industries. Such as manufacturing,
marketing, etc. to increase their business efficiency. Therefore, the needs
for a standard data mining process increased dramatically.

• Stages of Data Mining Process

• Data Mining Process is classified into two stages: Data preparation or
data preprocessing and data mining
Stages of Data Mining Process
Data Mining Process is classified into two stages: Data
preparation or data preprocessing and data mining
• a. Data Cleaning
• In the phase of data mining process, data gets cleaned.
• As we know data in the real world is noisy, inconsistent and incomplete.
• It includes a number of techniques. Such as filling in the missing values,
combined compute.
• The output of the data cleaning process is adequately cleaned data.
• b. Data Integration
• In this phase of Data Mining process data in integrated from different data sources
into one. As data lies in different formats in a different location.
• We can store data in a database, text files, spreadsheets, documents, data cubes,
and so on.
• Although, we can say data integration is so complex, tricky and difficult task.
That is because normally data doesn’t match the different sources.
• We use metadata to reduce errors in the data integration process. Another issue
faced is data redundancy.
• In this case, the same data might be available in different tables in the same
database.
• Data integration tries to reduce redundancy to the maximum possible level. As
without affecting the reliability of data
• c. Data Selection
• This is the process by which data relevant to the analysis is retrieved from the
database.
• As this process requires large volumes of historical data for analysis.
• So, usually, the data repository with integrated data contains much more data than
actually required.
• From the available data, data of interest needs to be selected and stored.
• d. Data Transformation
• In this process, we have to transform and consolidate the data into different
forms. That must be suitable for mining.
• Normally this process includes normalization, aggregation, generalization etc.
• For example, a data set available as “-5, 37, 100, 89, 78” can be transformed as “-
0.05, 0.37, 1.00, 0.89, 0.78”. Here data becomes more suitable for data mining.
After data integration, the available data is ready for data mining.
• e. Data Mining
• In this phase of Data Mining process, we have applied methods to extract patterns
from the data.
• As these methods are complex and intelligent. Also, this mining includes several
tasks. Such as classification, prediction, clustering, time series analysis and so on.
f. Pattern Evaluation
•The pattern evaluation identifies the truly interesting patterns.
•That is representing knowledge based on different types of interesting
measures.
•A pattern is considered to be interesting if it is potentially useful.
•Also, easily understandable by humans.

•g. Knowledge Representation

•In the phase of Data Mining process, we have to represent data to the user in
an appealing way.
• Also, that information is mined from the data. To generate output different
techniques are need to be applied.
• Data Mining phases:
• The whole process of Data Mining consists of three main phases:
• Data Pre-processing – Data cleaning, integration, selection, and
transformation takes place
• Data Extraction – Occurrence of exact data mining
• Data Evaluation and Presentation – Analyzing and presenting results
• What is the KDD Process?

• The term Knowledge Discovery in Databases, or KDD for short, refers to

the broad process of finding knowledge in data, and emphasizes the
"high-level" application of particular data mining methods.
• It is of interest to researchers in machine learning, pattern recognition,
databases, statistics, artificial intelligence, knowledge acquisition for
expert systems, and data visualization.

• KDD process is to extract knowledge from data in the context of large

databases.

• It does this by using data mining methods (algorithms) to extract

(identify) what is deemed knowledge, according to the specifications of
measures and thresholds, using a database along with any required
preprocessing, subsampling, and transformations of that database.
KDD Process:
• The overall process of finding and interpreting patterns from data involves the
repeated application of the following steps:
• Developing an understanding of
• the application domain
• the relevant prior knowledge
• the goals of the end-user
• Creating a target data set:
• selecting a data set, or focusing on a subset of variables, or data samples, on
which discovery is to be performed.
• Data cleaning and preprocessing.
• Removal of noise or outliers.
• Collecting necessary information to model or account for noise.
• Strategies for handling missing data fields.
• Accounting for time sequence information and known changes.
• Data reduction and projection.
• Finding useful features to represent the data depending on the goal of the task.
• Using dimensionality reduction or transformation methods to reduce the
effective number of variables under consideration or to find invariant
representations for the data.
• Choosing the data mining task.
• Deciding whether the goal of the KDD process is classification, regression,
clustering, etc.
• Choosing the data mining algorithm(s).
• Selecting method(s) to be used for searching for patterns in the data.
• Deciding which models and parameters may be appropriate.
• Matching a particular data mining method with the overall criteria of the
KDD process.
• Data mining.
• Searching for patterns of interest in a particular representational form or a
set of such representations as classification rules or trees, regression,
clustering, and so forth.
• Interpreting mined patterns.
• Consolidating discovered knowledge.
• KDD refers to the overall process of discovering useful knowledge from data.
• It involves the evaluation and possibly interpretation of the patterns to make the
decision of what qualifies as knowledge.
• It also includes the choice of encoding schemes, preprocessing, sampling, and
projections of the data prior to the data mining step.
Data mining refers to the application of algorithms for extracting patterns from
data without the additional steps of the KDD process.
• Definitions Related to the KDD Process
• Knowledge discovery in databases is the non-trivial process of identifying valid,
novel, potentially useful, and ultimately understandable patterns in data.
• Applications of Data Mining

• Financial Analysis
• Biological Analysis
• Scientific Analysis
• Intrusion Detection
• Fraud Detection
• Research Analysis
• Data Mining Applications & Use Cases
• Following are the applications of data mining in various sectors:
• a. Data Mining in Finance
• We have to Increase customer loyalty by collecting and analyzing customer
behavior data. Also, one needs to help banks that predict customer behavior
and launch relevant services and products.
• Helps in Discovering hidden correlations between various financial
indicators that need to detect suspicious activities with a high potential risk.
• Generally, it identifies fraudulent or non-fraudulent actions. As it done by
collecting historical data. And then turning it into valid and useful
information.
• b. Data Mining in Healthcare
• Basically, it provides government, regulatory and competitor information
that can fuel competitive advantage. Although, it supports the R&D process.
And then go-to-market strategy with rapid access to information at every
phase.
• Generally, it discovers the relationships between diseases and the
effectiveness of treatments.
• That is to identify new drugs or to ensure that patients receive appropriate,
timely care.
• Also, it supports healthcare insurers in detecting fraud and abuse.
c. Data Mining for Intelligence
• Generally, it reveals hidden data related to money laundering, narcotics
trafficking, etc.
• Also, helps in Improving intrusion detection with a high focus on anomaly
detection. And identify suspicious activity from a day one.
• Basically, convert text-based crime reports into word processing files.
• That can be used to support the crime-matching process.
• d. Data Mining in Telecommunication
• In this, data mining gains a competitive advantage and reduce customer
churn by understanding demographic characteristics and predicting
customer behavior.
• Increases customer loyalty and improve profitability by providing
customized services.
• As it supports customer strategy by developing appropriate marketing
campaigns and pricing strategies.
• f. Data Mining in Marketing and Sales
• Basically, it enables businesses to understand the hidden patterns inside
historical purchasing transaction data. Thus helping in planning and
launching new marketing campaigns.
• Generally, the following illustrates several data mining applications in sale
and marketing.
• We use it for market basket analysis. That is to provide information on what
product combinations have to purchased together. This information helps
businesses promote their most profitable products and maximize the profit. In
addition, it encourages customers to purchase related products.
• Retail companies use data mining to identify customer’s behavior buying
patterns.
• g. Data Mining in E-commerce
• Many E-commerce companies are using data mining business Intelligence to
offer cross-sells through their websites.
• One of the most famous of these is, of course, Amazon. They use
sophisticated mining techniques to drive their ‘People who viewed that
product. Also liked this’ functionality.
• h. Data Mining in Education
• There is a newly emerging field, called Educational Data Mining. As it
concerns with developing methods. That discover knowledge from data
originating from educational Environments.

• The goals of EDM are identified as predicting students’ future learning

behavior, studying. We use data mining by an institution to take
accurate decisions. And also to predict the results of the student.

• With the results, the institution can focus on what to teach and how to
teach. Learning pattern of the students can be captured. And used to
develop techniques to teach them.
• [Link]
Main Purpose of Data
Mining
• Basically, Data mining has been integrated with many other techniques from
other domains such as statistics, machine learning, pattern recognition,
database and data warehouse systems, information retrieval, visualization,
etc. to gather more information about the data and to helps predict hidden
patterns, future trends, and behaviors and allows businesses to make
decisions.
• Technically, data mining is the computational process of analyzing data
from different perspectives, dimensions, angles and
categorizing/summarizing it into meaningful information.
Getting to Know Your Data

• Data Objects and Attribute Types

• Basic Statistical Descriptions of Data
• Measuring Data Similarity and Dissimilar
• What is Data?
• Data sets are made up of data objects.
• A data object represents an entity.
• Also called sample, example, instance, data point, object, tuple.
• Data objects are described by attributes.
• An attribute is a property or characteristic of a data object. –
Examples: eye color of a person, temperature, etc.
• Attribute is also known as variable, field, characteristic, or feature
• A collection of attributes describe an object.
• Attribute values are numbers or symbols assigned to an attribute.
 Data Objects

■ Data sets are made up of data objects.

■ A data object represents an entity.
■ Examples:
■sales database: customers, store items, sales
■medical database: patients, treatments
■university database: students, professors, courses
■ Also called samples , examples, instances, data points,
objects, tuples.
■ Data objects are described by attributes.
■ Database rows -> data objects; columns ->attributes.
35
 Types of Data Sets
■ Record
■ Relational records
■ Data matrix, e.g., numerical matrix,
crosstabs
■ Document data: text documents: term-
frequency vector
■ Transaction data
■ Graph and network
■ World Wide Web
■ Social or information networks
■ Molecular Structures
■ Ordered
■ Video data: sequence of images
■ Temporal data: time-series
■ Sequential Data: transaction sequences
■ Genetic sequence data
■ Spatial, image and multimedia:
■ Spatial data: maps
■ Image data:
■ Video data:

36
A Data Object

database rows 🡺 data objects

database columns 🡺 attributes
Attributes:
• Attribute (or dimensions, features, variables): a data field, representing a
characteristic or feature of a data object.
• E.g., customer _ID, name, address
• Attribute values are numbers or symbols assigned to an attribute.
• The attribute can be defined as a field for storing the data that represents
the characteristics of a data object.
• The attribute is the property of the object.
• The attribute represents different features of the object.
• For example, hair color is the attribute of a lady.
• Similarly, rollno, and marks are attributes of a student.
• An attribute vector is commonly known as a set of attributes that are used
to describe a given object.
Type of attributes:
We need to differentiate between different types of attributes during Data-
preprocessing.
• There are different types of attributes. some of these attributes are
mentioned below.
• So firstly, we need to differentiate between qualitative and quantitative
attributes.
1. Qualitative Attributes such as Nominal, Ordinal, and Binary
Attributes.
2. Quantitative Attributes such as Discrete and Continuous Attributes.
• Attributes can store all kinds of different descriptive information, however,
depending on the descriptive information stored in each attribute, some
properties are implied, so operations are permitted, and other operations
and properties are not allowed and not applied respectively.
• Descriptive information does not imply any order, size, or any other
quantitative information.
• Attributes can store all kinds of different descriptive information.
.
• We can break attributes down into four different categories: nominal, ordinal,
interval, and ratio.
• The first attribute category is the nominal attribute category.
• What’s important here is that this That means that you cannot state that one attribute
is greater than or less than another attribute or you cannot multiply attributes
together, so for instance, it does not make sense to multiply the color blue by the
color red.
• The only comparisons you can do with nominal attributes, are to check whether to
attributes are equal or not equal.
Types of Attributes

41
 Qualitative Attributes:
[Link] Attributes –
• A nominal attribute provides descriptive information about the object such
as the color of the object, the name of an object so for instance a city name,
or the type of an object.
●related to names: The values of a Nominal attribute are names of things,
some kind of symbols.
●Nominal scales: Measurements where a value is used to represent
something or someone.
● Nominal values are typically coded, or converted to numeric values for
later statistical analysis.
●Values of Nominal attributes represents some category or state and that’s
why nominal attribute also referred as categorical attributes and there is no
order (rank, position) among values of the nominal attribute.
Example :

42
• Nominal Attributes:
• Nominal means “relating to names”.
• The values of a nominal attribute are symbols or names of things.
• Each value represents some kind of category, code, or state,
• Nominal attributes are also referred to as categorical attributes.
• The values of nominal attributes do not have any meaningful order.
• Example: The attribute marital_status can take on the values single, married
hair color,Occupation etc.
• Because nominal attribute values do not have any meaningful order about them
and they are not quantitative.
• It makes no sense to find the mean (average) value or median (middle) value
for such an attribute.
• Mode is one of the measures of Central tendency.
• Properties of nominal scale are:
• Mutually exclusive
• Categories are distinct and homogeneous
• They cannot be measured or ordered but can be counted
• Data can reflect that they are different from each other but cannot be
ordered as smaller or greater.
• The only mathematical operation that can be performed is count or say
frequency.
• Nominal Scale, also called the categorical variable scale, is defined as a
scale used for labeling variables into distinct classifications and doesn’t
involve a quantitative value or order.
• This scale is the simplest of the four variable measurement scales.
• Nominal scale is often used in research surveys and questionnaires where
only variable labels hold significance.
• For instance, a customer survey asking “Which brand of smartphones do
you prefer?” Options : “Apple”- 1 , “Samsung”-2, “OnePlus”-3.
• Nominal Attributes: Binary attribute
• A binary attribute is a special nominal attribute with only two states: 0 or 1.
• A binary attribute is symmetric if both of its states are equally valuable and
carry the same weight.
• Example: the attribute gender having the states male and female.
• A binary attribute is asymmetric if the outcomes of the states are not equally
important.
• Example: Positive and negative outcomes of a medical test.
• By convention, we code the most important outcome, which is usually the
rarest one, by 1 (e.g. positive) and the other by 0 (e.g. negative).
2. Binary Attributes:
•Binary data has only 2 values/states.
•For Example yes or no, affected or unaffected, true or false.
•Symmetric: Both values are equally important (Gender).
•Asymmetric: Both values are not equally important (Result).

48
3. Ordinal Attributes:
•An ordinal attribute is an attribute with possible values that have a meaningful
order or ranking among them, but the magnitude between successive values is not
known.
•Example: An ordinal attributes include grade (eg. A+,A,B+,B etc).
•This attribute has three possible values: small, medium, and large.
•The values have a meaningful sequence (which corresponds to increasing )
•Professional Rank etc.
•The central tendency of an ordinal attribute can be represented by its mode and
its median (middle value in an ordered sequence), but the mean cannot be defined
3. Ordinal Attributes :
●The Ordinal Attributes contains values that have a meaningful
sequence or ranking(order) between them, but the magnitude
between values is not actually known, the order of values that
shows what is important but don’t indicate how important it is.

50
 Ordinal Scale Properties:
•It has unequal units
•It displays from highest to lowest by different measurement points
•It has no zero point i.e. it is arbitrary or absolute
•Interval size is unequal and unknown
•Mutually exclusive
•Categories are distinct and homogeneous
•They cannot be measured but can be counted and ordered/ranked
•Data can express that one is different from another and one is greater or
smaller than the other.
•However, data cannot say that one is ‘X’ units or ‘X’times greater or smaller
than the other.
Examples: Academic performance: School-1, College-2, Bachelor-3,
Masters-4
Disease severity: mild-1, moderate-2, severe-3
 3. Ordinal Attributes :
Characteristics of Ordinal Variable
● It is an extension of nominal data.
● It has no standardized interval scale.
● It establishes a relative rank.
● It measures qualitative traits.
● The median and mode can be analyzed.
● It has a rank or order.

Interval size is unequal and unknown

Mutually exclusive
Categories are distinct and homogeneous
They cannot be measured but can be counted and
ordered/ranked
52
• Ordinal Scale is defined as a variable measurement scale used to simply
depict the order of variables and not the difference between each of the
variables.
• These scales are generally used to depict non-mathematical ideas such as
frequency, satisfaction, happiness, a degree of pain, etc.
• Origin of this scale is absent due to which there is no fixed start or “true
zero”.
• example
Quantitative Attributes:

1. Numeric: A numeric attribute is quantitative because, it is a measurable

quantity, represented in integer or real values. Numerical attributes are of 2
types, interval, and ratio.

● An interval-scaled attribute has values, whose differences are

interpretable, but the numerical attributes do not have the correct
reference point, or we can call zero points.
● Data can be added and subtracted at an interval scale but can not be
multiplied or divided.
● Consider an example of temperature in degrees Centigrade.
● If a day’s temperature of one day is twice of the other day we
cannot say that one day is twice as hot as another day.

54
Numeric Attributes:
• A Numeric Attribute is quantitative , it is measurable quantity , represented in
integer or real values. Numeric attributes can be interval-scaled or ratio-scaled.
• Interval-Scaled Attributes:
• Interval-Scaled Attributes are measured on a scale of equal-size units.
• The values of interval-scaled attributes have order and can be positive,0,negative.
• Thus, in addition to providing a ranking of values , such attributes allow and
quantify the difference between values.
Eg. temprature attribute is interval scaled.
This type of data represents quantitative data with equal
intervals between consecutive values.
Interval data has no absolute zero point, and therefore,
ratios cannot
be computed.
Examples of interval data include temperature, IQ scores,
and time.
Interval data is used in data mining for clustering and
• Interval
• An interval scale is one where there is order and the difference between
two values is meaningful.
• Examples of interval variables include:
• temperature (Farenheit), temperature (Celcius), pH, SAT score (200-800),
credit score (300-850).
Quantitative Attributes:

● A ratio-scaled attribute is a numeric attribute with a fix zero-point.

● If a measurement is ratio-scaled, we can say of a value as being a
multiple (or ratio) of another value.
● The values are ordered, and we can also compute the difference
between values, and the mean, median, mode, Quantile-range, and Five
number summary can be given.
● This type of data is similar to interval data, but with an absolute zero
point. In ratio data, it is possible to compute ratios of two values, and
this makes it possible to make meaningful comparisons. Examples of
ratio data include height, weight, and income.
● Ratio data is used in data mining for prediction and association rule
mining tasks.
57
Characteristics of interval scale:
•It has equal units
•It has arbitrary (absolute) zero which is just a reference point
•Interval size is known, equal and constant
•Measurement is taken on both sides of zero
•Mutually exclusive.
•Categories are distinct and homogeneous.
•Can be measured and ranked/ordered.
•Data can show that one is different from other, one is greater or smaller than
other, one is ‘X’ unit greater or smaller than other but cannot show that one is
‘X’ times greater or smaller than the other.
•Data can be added or subtracted but cannot be divided or multiplied.
. Ratio scale
•Ratio scale is a system of assigning number to the variable to label them for
identification and ranking based on a scale having equal interval size with
absolute zero that allows measurement on only one side of zero.
•Here, a variable is categorized in different subgroups in ascending or
descending order and intervals between the successive categories are equal
and constant.
•Same as the interval scale except zero has the true value i.e. zero represent
the absolute value
•All the mathematical operations are applicable in this scale.
•Examples include: weight, height, sales figures, ruler measurements, number
of children.
•Here, weight 0 means no weight and weight 60 kgs mean it is 30 units more
than 30 kgs and two times of 30 kgs.
Characteristics of ratio scale:
•It has equal units
•Has absolute zero
•Interval size is known, equal and constant.
•Measures on only one side of zero
•Mutually exclusive.
•Categories are distinct and homogeneous.
•Can be measured and ranked/ordered.
•Data can show that one is different from other, one is greater or smaller than
other, one is ‘X’ unit greater or smaller than other, one is ‘X’ times greater or
smaller than the other.
•Data can be added, subtracted, multiplied or divided.
•Ratio of two numbers can be meaningfully calculated and interpreted
• Comparisons of Four Scales of Measurement:
 · Counting
· Ranking · Ordered category with equal and known interval size and
· absolute zero
Measurement · Can say one is different from the other
Ratio
· Addition · Can say one is greater or smaller than the other Body weight, height etc.
Scale · Subtraction
· Can say one is ‘X’ units greater or smaller than the other
·
Multiplication · Can say one is ‘X’ times greater or smaller than the other
· Division
• Ratio
• A ratio variable, has all the properties of an interval variable, and also has a
clear definition of 0.0. When the variable equals 0.0, there is none of that
variable.
• Examples of ratio variables include:
• enzyme activity, dose amount, reaction rate, flow rate, concentration, pulse,
weight, length, temperature in Kelvin (0.0 Kelvin really does mean “no
heat”), survival time.
• For example, because weight is a ratio variable, a weight of 4 grams is twice
as heavy as a weight of 2 grams.
Quantitative Attributes:

[Link] : Discrete data have finite values it can be numerical and can also
be in categorical form. These attributes has finite or countably infinite set of
values.

Example:

66
Quantitative Attributes:

3..Continuous: Continuous data have an infinite no of states. Continuous data

is of float type. There can be many values between 2 and 3.
Example :

67
• Attribute Summary
Attribute Types – Categorical/Qualitative

■ Nominal: categories, states, or “names of things”

■ Hair_color = {auburn, black, blond, brown, grey, red, white}
■ marital status, occupation, ID numbers, zip codes
■ Binary
■ Nominal attribute with only 2 states (0 and 1)
■ Symmetric binary: both outcomes equally important
■ e.g., gender
■ Asymmetric binary: outcomes not equally important.
■ e.g., medical test (positive vs. negative)
■ Convention: assign 1 to most important outcome
(e.g., positive/Negative)
■ Ordinal
■ Values have a meaningful order (ranking) but magnitude
between successive values is not known.
■ Size = {small, medium, large}, grades, army rankings

69
 Numeric Attribute Types
■Quantity (integer or real-valued)
■Interval
■ Measured on a scale of equal-sized units
■ Values have order
■ E.g., temperature in C˚or F˚, calendar
dates
■ No true zero-point
■Ratio
■ Inherent zero-point
■ We can speak of values as being an order of
magnitude larger than the unit of
measurement (10 K˚ is twice as high as 5
K˚).
■ e.g., temperature in Kelvin, length,
70
 Discrete vs. Continuous
Attributes
■ Discrete Attribute
■Has only a finite or countably infinite set of values
■E.g., zip codes, profession, or the set of words
in a collection of documents
■Sometimes, represented as integer variables
■Note: Binary attributes are a special case of discrete
attributes
■ Continuous Attribute
■Has real numbers as attribute values
■E.g., temperature, height, or weight
■Practically, real values can only be measured and
represented using a finite number of digits
■Continuous attributes are typically represented as
floating-point variables
71
• Example of attribute
• In this example, RollNo, Name, and Result are attributes of the object named
as a student.

• Nominal Attributes
• Nominal data is in alphabetical form and not in an integer.
• Nominal Attributes are Qualitative Attributes.
• Examples of Nominal attributes
• Binary Attributes
• Binary data have only two values/states.
• For example, here Disease detected can be only Yes or No.
• Binary Attributes are Qualitative Attributes.
• Examples of Binary Attributes

Attribute Value

Disease detected Yes, No

Result Pass, Fail

• The binary attribute is of two types;
• Symmetric binary
• Asymmetric binary
• Examples of Symmetric data
• Both values are equally important. For example, if we have open
admission to our university, then it does not matter, whether you are a male
or a female.
• Example:

Attribute Value

Gender Male, Female

• Examples of Asymmetric data
• Both values are not equally important.
• For example, Disease detected is more important than Disease not detected.
• If a patient is with Disease and we ignore him, then it can lead to death but if a person is
not Disease detected and we ignore it, then there is no special issue or risk.
• Example:
Attribute Value

Disease detected Yes, No

Result Pass, Fail

• Ordinal Attributes
• All Values have a meaningful order.
• For example, Grade-A means highest marks, B means marks are less than A, C means
marks are less than grades A and B, and so on. Ordinal Attributes are Quantitative
Attributes.
• Examples of Ordinal Attributes

Attribute Value

Grade A, B, C, D, F

BPS- Basic pay scale 16, 17, 18

• Discrete Attributes
• Discrete data have a finite value. It can be in numerical
form and can also be in a categorical form.
• Discrete Attributes are Quantitative Attributes.
• Examples of Discrete Data
Attribute Value

Profession Teacher, Bussiness Man, Peon etc

Postal Code 42200, 42300 etc

Example of Continuous Attribute

• Continuous data technically have an infinite number of steps.
• Continuous data is in float type.
• There can be many numbers in between 1 and 2.
• These attributes are Quantitative Attributes.
• Example of Continuous Attribute

Attribute Value

Height 5.4…, 6.5….. etc

Weight 50.09…. etc

• Properties of Attribute Values
• The type of an attribute depends on which of the following properties it
possesses:
• Distinctness: =
• Order: < >
• Addition: + -
• Multiplication: * /
• Nominal attribute: Distinctness
• Ordinal attribute: distinctness & order
• Interval attribute: distinctness, order & addition
• Ratio attribute: all 4 properties
Properties of Attribute Values
Attribute Description Examples
Type

Nominal zip codes, employee ID numbers,

The values of a nominal attribute are just different eye color, sex: {male, female}
names,
i.e., nominal attributes provide only enough
information to distinguish one object from another. (=,
)

Ordinal The values of an ordinal attribute provide enough

information to order objects. (<, >) hardness of minerals, {good,
better, best}, grades, street
numbers

Interval calendar dates, temperature in

For interval attributes, the differences between values
Celsius or Fahrenheit
are meaningful,
i.e., a unit of measurement exists. (+, - )

Ratio For ratio variables, both differences and ratios are

meaningful. (*, /) temperature in Kelvin, monetary
quantities, counts, age, mass,
length,
• Attribute Types
• Categorical (Qualitative) and Numeric (Quantitative)
• Nominal and Ordinal attributes are collectively referred to as
categorical or qualitative attributes.
• qualitative attributes, such as employee ID, lack most of the properties of
numbers.
• Even if they are represented by numbers, i.e. , integers, they should be
treated more like symbols .
• Mean of values does not have any meaning.
• Interval and Ratio are collectively referred to as quantitative or numeric
attributes.
• Quantitative attributes are represented by numbers and have most of the
properties of numbers .
• Note that quantitative attributes can be integer-valued or continuous.
• Numeric operations such as mean, standard deviation are meaningful
• Discrete vs. Continuous Attributes
• Discrete Attribute
• Has only a finite or countably infinite set of values
• zip codes, profession, or the set of words in a collection of documents
• Sometimes, represented as integer variables
• Note: Binary attributes are a special case of discrete attributes
• Binary attributes where only non-zero values are important are called
asymmetric binary attributes.
• Continuous Attribute
• Has real numbers as attribute values
• temperature, height, or weight
• Practically, real values can only be measured and represented using a finite
number of digits
• Continuous attributes are typically represented as floating-point variables
• Example of attribute
• In this example, RollNo, Name, and Result are attributes of the object named
as a student.

• Nominal Attributes
• Nominal data is in alphabetical form and not in an integer.
• Nominal Attributes are Qualitative Attributes.
• Examples of Nominal attributes
• Binary Attributes
• Binary data have only two values/states.
• For example, here Disease detected can be only Yes or No.
• Binary Attributes are Qualitative Attributes.
• Examples of Binary Attributes

Attribute Value

Disease detected Yes, No

Result Pass, Fail

• The binary attribute is of two types;
• Symmetric binary
• Asymmetric binary
• Examples of Symmetric data
• Both values are equally important. For example, if we have open
admission to our university, then it does not matter, whether you are a male
or a female.
• Example:

Attribute Value

Gender Male, Female

• Examples of Asymmetric data
• Both values are not equally important.
• For example, Disease detected is more important than Disease not detected.
• If a patient is with Disease and we ignore him, then it can lead to death but if a person is
not Disease detected and we ignore it, then there is no special issue or risk.
• Example:
Attribute Value

Disease detected Yes, No

Result Pass, Fail

• Ordinal Attributes
• All Values have a meaningful order.
• For example, Grade-A means highest marks, B means marks are less than A, C means
marks are less than grades A and B, and so on. Ordinal Attributes are Quantitative
Attributes.
• Examples of Ordinal Attributes

Attribute Value

Grade A, B, C, D, F

BPS- Basic pay scale 16, 17, 18

• Discrete Attributes
• Discrete data have a finite value. It can be in numerical
form and can also be in a categorical form.
• Discrete Attributes are Quantitative Attributes.
• Examples of Discrete Data
Attribute Value

Profession Teacher, Bussiness Man, Peon etc

Postal Code 42200, 42300 etc

Example of Continuous Attribute

• Continuous data technically have an infinite number of steps.
• Continuous data is in float type.
• There can be many numbers in between 1 and 2.
• These attributes are Quantitative Attributes.
• Example of Continuous Attribute

Attribute Value

Height 5.4…, 6.5….. etc

Weight 50.09…. etc

Discrete data represents discrete variables, which you can count in a finite
amount of time. The key feature here is that these variables are countable
instead of measurable.
• Basic Statistical Descriptions of Data
• Statistical description can be used to identify properties of the data and
highlight which data values should be treated as noise or outliers.
• These descriptive statistics are of great help in understanding the
distribution of the data.
• Three areas of basic statistical descriptions:
• Measure of Central Tendency : Which measure the location of the
middle or center of a data distribution.
• Measures of central tendency include mean, median, mode, and midrange.
• Measures of data dispersion spread of the data which include quartiles,
interquartile range (IQR), and variance.
• Graphic Display of Basic Statistical Description of Data:
• Graphical data representation of basic statistical descriptions(Quantile Plot,
Histogram, Scatter Plot etc)
• Measures of Central Tendency & Dispersion
• Measures that indicate the approximate center of a distribution are called measures
of central tendency.
• Measures that describe the spread of the data are measures of dispersion.
• These measures include the mean, median, mode, range, upper and lower
quartiles, variance, and standard deviation.
 Measure of Central Tendency
• A measure of central tendency is an important aspect of quantitative data.
• Three of the many ways to measure central tendency are
the mean, median and mode.
• A measure of central tendency is a single value that attempts to describe a set of
data by identifying the central position within that set of data.
• Measures of central tendency are sometimes called measures of central location.
• The mean (often called the average) is most likely the measure of central
tendency that you are most familiar with, but there are others, such as the median
and the mode.
• It is also called as categorized as summary statistics.
Measuring the Central Tendency
■ Mean (algebraic measure) (sample vs. population):
Note: n is sample size and N is population size.
■ Weighted arithmetic mean:
■ Trimmed mean: chopping extreme values
■ Median:
■ Middle value if odd number of values, or average
of the middle two values otherwise
■ Estimated by interpolation (for grouped data):

■ Mode
■ Value that occurs most frequently in the data
■ Unimodal, bimodal, trimodal

90
• Measures of Central Tendency
• Measures that indicate the approximate center of a distribution are called
measures of central tendency.
A. Finding the Mean
• The mean of a set of data is the sum of all values in a data set divided by the
number of values in the set. It is also often referred to as an arithmetic average.
The Greek letter (“mu”) is used as the symbol for population mean and the
symbol is used to represent the mean of a sample. To determine the mean of
a data set:
1. Add together all of the data values.
2. Divide the sum from Step 1 by the number of data values in the set.
• Measuring Central Tendency: Mean

• The most common and most effective numerical measure of the “center” of a
set of data is the arithmetic mean.

• Sometimes, each value xi in a set may be associated with a weight wi.

• The weights reflect the significance and importance attached to their
respective values.
• Measuring Central Tendency: Mean
• Although the mean is the single most useful quantity for describing a data
set, it is not always the best way of measuring the center of the data.
• A major problem with the mean is its sensitivity to extreme (outlier)
values.
• Even a small number of extreme values can corrupt the mean.
• To offset the effect caused by a small number of extreme values, we can
instead use the trimmed mean,
• Trimmed mean can be obtained after chopping off values at the high and
low extremes.
• Measuring Central Tendency: Median
• Another measure of the center of data is the median.
• Suppose that a given data set of N distinct values is sorted in numerical
order.
• If N is odd, the median is the middle value of the ordered set;
• If N is even, the median is the average of the middle two values.
• In probability and statistics, the median generally applies to numeric data.
however, we may extend the concept to ordinal data.
• Suppose that a given data set of N values for an attribute X is sorted in
increasing order.
• If N is odd, then the median is the middle value of the ordered set.
• If N is even, then the median may not be not unique.
• In this case, the median is the two middlemost values and any value in
between.
• For example: Let's consider the data: 48, 20, 50, 69, 73. What is the
median?
• Solution:
• Arranging in ascending order, we get: 20, 48, 50, 69, 73. Here, n ([Link]
observations) = 5
• So, to find the median of odd data we use the formula:
• [(n+1)/2] = (5 + 1)/2 = 6/2 = 3
• Therefore, Median = 3rd observation
• Median = 50.
• What is Median of Grouped Data?
• Median of a grouped data is data that is arranged in ascending order and is
written in a continuous manner.
• The data is in the form of a frequency distribution table that divides the higher
level of data from the lower level of data.
• One of the simplest methods of finding the median of grouped data is by using
the formula.
• As finding the middle value or median of a grouped data might be tough.
• Therefore, to find the median for grouped data we can use the following steps
and formula:
• Step 1: Find the total number of observations.
• Step 2: Define the class size, and divide the data into different classes.
• Step 3: Calculate the cumulative frequency of each class.
• Step 4: Identify the class in which the median falls. (Median Class is the class
where n/2 lies.)
• Step 5: Find the lower limit of the median class(l), and the cumulative
frequency of the median class (c).
• Finding the Median of Grouped Data
• The marks obtained in English test by 17 students were recorded. What is
the median marks of the students?
• Measuring Central Tendency: Mode
• Another measure of central tendency is the mode.
• The mode for a set of data is the value that occurs most frequently in the
set.
• It is possible for the greatest frequency to correspond to several different
values, which results in more than one mode.
• Data sets with one, two, or three modes: called unimodal, bimodal, and
trimodal. – At the other extreme, if each data value occurs only
once, then there is no mode.
• Central Tendency Measures for Numerical Attributes: Mean, Median,
Mode
• Central Tendency Measures for Categorical Attributes: Mode
• Central Tendency Measures for Nominal Attributes: Mode
• Central Tendency Measures for Ordinal Attributes: Mode, Median
• Measuring Central Tendency Mean, Median, Mode
• Median, mean and mode of symmetric, positively and negatively skewed
data
 • Measuring Central Tendency: Example

• What are central tendency measures (mean, median, mode)for the

following attributes?
• attr1 = {2,4,4,6,8,24}
• attr2 = {2,4,7,10,12}
• attr3 = {xs,s,s,s,m,m,l}
• Measuring Central Tendency: Example
• What are central tendency measures (mean, median, mode)for the
following attributes? attr1 = {2,4,4,6,8,24}
• mean = (2+4+4+6+8+24)/6 = 8 average of all values
• median = (4+6)/2 = 5 avg. of two middle values
• mode = 4 attr2 = {2,4,7,10,12} most frequent item
• mean = (2+4+7+10+12)/5 = 7 average of all values
• median = 7 middle value
• mode = any of them (no mode) all of them has same freq.
• attr3 = {xs,s,s,s,m,m,l} mean is meaningless for categorical attributes.
• median = s middle value mode = s most frequent item
standard deviation σ (s) is the square root of varianceσ2 (s2)
• In the case of a moderately skewed distribution, i.e. in general, the
difference between mean and mode is equal to three times the difference
between the mean and median. Thus, in this case, the empirical
relationship is expressed as, Mean – Mode = 3 (Mean – Median).
• In the case of a frequency distribution that has a symmetrical frequency
curve, the empirical relation states that mean = median = mode.
• In the case of a positively skewed frequency distribution curve, mean >
median > mode.
• In the case of negatively skewed frequency distribution, mean < median <
mode.
• What is Dispersion in Statistics?
• Dispersion is the state of getting dispersed or spread. Statistical dispersion
means the extent to which a numerical data is likely to vary about an
average value.
• In other words, dispersion helps to understand the distribution of the data.
 Measuring the Dispersion of
Data
■ Quartiles, outliers and boxplots
■ Range, Quartiles: Q1 (25th percentile), Q3 (75th percentile)
■ Inter-quartile range: IQR = Q3 – Q1
■ Five number summary: min, Q1, median, Q3, max
■ Boxplot: ends of the box are the quartiles; median is marked; add
whiskers, and plot outliers individually
■ Outlier: usually, a value higher/lower than 1.5 x IQR
■ Variance and standard deviation (sample: s, population: σ)
■ Variance: (algebraic, scalable computation)

■ Standard deviation s (or σ) is the square root of variance s 2 (or σ2)

128
• Range: It is simply the difference between the maximum value and the
minimum value given in a data set. Example: 1, 3,5, 6, 7 => Range = 7 -
1= 6
• Distance between first and third quartile is simple measure of spread of
Data that gives the range covered by the middle half of the data.
• This distance is called IQR.
• IQR=Q3-Q1
 Boxplot Analysis

■ Five-number summary of a distribution

■ Minimum, Q1, Median, Q3, Maximum
■ Boxplot
■ Data is represented with a box
■ The ends of the box are at the first and
third quartiles, i.e., the height of the box is
IQR
■ The median is marked by a line within the
box
■ Whiskers: two lines outside the box
extended to Minimum and Maximum
■ Outliers: points beyond a specified outlier
threshold, plotted individually
133
• With boxplots, this is done using something called “fences”. The idea is
that anything outside the fences is a potential outlier and shouldn’t be
included in the main group that we graph.
 Visualization of Data Dispersion: 3-D
Boxplots

Data Mining: Concepts and

* Techniques 135
 Properties of Normal Distribution
Curve

■ The normal (distribution) curve

■From μ–σ to μ+σ: contains about 68% of the
measurements (μ: mean, σ: standard deviation)
■ From μ–2σ to μ+2σ: contains about 95% of it
■From μ–3σ to μ+3σ: contains about 99.7% of it

136
 Problems
■ Given the set:
■ 10, 13, 4, 7, 8, 11 10, 16, 18, 12, 3, 6, 9, 9, 4, 13, 20, 7, 5, 10, 17, 10,
16, 14, 8, 18
■ Find their mean, mode, quartiles and variance.

137
 Problems

A pediatrician has obtained the following table which

represents the number of children who begin to walk
for the first time at different ages:
Months Children
9 1
10 4
11 9
12 16
13 11
14 8
15 1

1. Draw the respective frequency plot.

2. Calculate the mode, median, mean and variance .

138
 Problems
■ The result of throwing two dice 120 times is represented by the table

Sum
2 3 4 5 6 7 8 9 10 11 12
s
No.
of
3 8 9 11 20 19 16 13 11 6 4
Time
s
■ Calculate the mean and standard deviation.
■ draw the box plot for the above data

139
Graphic Displays of Basic Statistical
Descriptions

■ Boxplot: graphic display of five-number summary

■ Histogram: x-axis are values, y-axis repres. frequencies

■ Quantile plot: each value xi is paired with fi indicating

that approximately 100 fi % of data are ≤ xi

■ Quantile-quantile (q-q) plot: graphs the quantiles of one

univariant distribution against the corresponding quantiles
of another
■ Scatter plot: each pair of values is a pair of coordinates
and plotted as points in the plane
140
 Histogram Analysis
■ Histogram: Graph display of
tabulated frequencies, shown as
bars
■ It shows what proportion of cases
fall into each of several categories
■ Differs from a bar chart in that it is
the area of the bar that denotes the
value, not the height as in bar
charts, a crucial distinction when the
categories are not of uniform width
■ The categories are usually specified
as non-overlapping intervals of
some variable. The categories
(bars) must be adjacent

141
Histograms Often Tell More than
Boxplots

■ The two histograms

shown in the left
may have the same
boxplot
representation
■The same values
for: min, Q1,
median, Q3, max
■ But they have
rather different data
distributions

142
 Quantile Plot
■ Displays all of the data (allowing the user to assess both
the overall behavior and unusual occurrences)
■ Plots quantile information
■For a data xi data sorted in increasing order, fi
indicates that approximately 100 fi% of the data are
below or equal to the value xi

Data Mining: Concepts and

Techniques 143
 Quantile-Quantile (Q-Q) Plot
■ Graphs the quantiles of one univariate distribution against the
corresponding quantiles of another
■ View: Is there is a shift in going from one distribution to another?
■ Example shows unit price of items sold at Branch 1 vs. Branch 2 for
each quantile. Unit prices of items sold at Branch 1 tend to be lower
than those at Branch 2.

144
 Scatter plot
■ Provides a first look at bivariate data to see clusters of
points, outliers, etc
■ Each pair of values is treated as a pair of coordinates and
plotted as points in the plane

145
Positively and Negatively Correlated
Data

■ The left half fragment is positively

correlated
■ The right half is negative correlated

146
Uncorrelated Data

147
Chapter 2: Getting to Know Your
Data

■ Data Objects and Attribute Types

■ Basic Statistical Descriptions of Data

■ Data Visualization

■ Measuring Data Similarity and Dissimilarity

■ Summary

148
Chapter 2: Getting to Know Your
Data

■ Data Objects and Attribute Types

■ Basic Statistical Descriptions of Data

■ Data Visualization

■ Measuring Data Similarity and Dissimilarity

■ Summary

149
 Similarity and Dissimilarity
■ Similarity
■Numerical measure of how alike two data objects
are
■Value is higher when objects are more alike
■Often falls in the range [0,1]
■ Dissimilarity (e.g., distance)
■Numerical measure of how different two data
objects are
■Lower when objects are more alike
■Minimum dissimilarity is often 0
■Upper limit varies
■ Proximity refers to a similarity or dissimilarity
150
 Data Matrix and Dissimilarity
Matrix
■ Data matrix
■n data points with p
dimensions
■Two modes

■ Dissimilarity matrix
■n data points, but
registers only the
distance
■A triangular matrix
■Single mode
151
 Proximity Measure for Nominal
Attributes
■ Can take 2 or more states, e.g., red, yellow, blue, green
(generalization of a binary attribute)
■ Method 1: Simple matching
■ m: # of matches, p: total # of variables

■ Method 2: Use a large number of binary attributes

■ creating a new binary attribute for each of the M nominal
states

152
 Proximity Measure for Binary
Attributes
Object
j
■ A contingency table for binary data
Object
i
■ Distance measure for symmetric
binary variables:
■ Distance measure for asymmetric
binary variables:
■ Jaccard coefficient (similarity
measure for asymmetric binary
variables):
■ Note: Jaccard coefficient is the same as “coherence”:

153
 Dissimilarity between Binary
Variables
■ Example

■ Gender is a symmetric attribute

■ The remaining attributes are asymmetric binary
■ Let the values Y and P be 1, and the value N 0

154
 Example:
Data Matrix and Dissimilarity Matrix
Data Matrix

Dissimilarity Matrix
(with Euclidean Distance)

155
Distance on Numeric Data: Minkowski
Distance
■ Minkowski distance: A popular distance measure

where i = (xi1, xi2, …, xip) and j = (xj1, xj2, …, xjp) are two
p-dimensional data objects, and h is the order (the
distance so defined is also called L-h norm)
■ Properties
■ d(i, j) > 0 if i ≠ j, and d(i, i) = 0 (Positive
definiteness)
■ d(i, j) = d(j, i) (Symmetry)
■ d(i, j) ≤ d(i, k) + d(k, j) (Triangle Inequality)
■ 156
Special Cases of Minkowski Distance

■ h = 1: Manhattan (city block, L1 norm) distance

■ E.g., the Hamming distance: the number of bits that are
different between two binary vectors

■ h = 2: (L2 norm) Euclidean distance

■ h → ∞. “supremum” (Lmax norm, L∞ norm) distance.

■ This is the maximum difference between any component
(attribute) of the vectors

157
Example: Minkowski Distance
Dissimilarity Matrices
Manhattan
(L1)

Euclidean
(L2)

Supremu
m

158
 Ordinal Variables

■ An ordinal variable can be discrete or continuous

■ Order is important, e.g., rank
■ Can be treated like interval-scaled
■replace xif by their rank
■map the range of each variable onto [0, 1] by
replacing i-th object in the f-th variable by

■compute the dissimilarity using methods for

interval-scaled variables

159
 Attributes of Mixed Type
■ A database may contain all attribute types
■Nominal, symmetric binary, asymmetric binary,
numeric, ordinal
■ One may use a weighted formula to combine their effects

■f is binary or nominal:
dij(f) = 0 if xif = xjf , or dij(f) = 1 otherwise
■f is numeric: use the normalized distance
■f is ordinal
■Compute ranks rif and
■Treat zif as interval-scaled
160
 Cosine Similarity
■ A document can be represented by thousands of attributes, each
recording the frequency of a particular word (such as keywords) or
phrase in the document.

■ Other vector objects: gene features in micro-arrays, …

■ Applications: information retrieval, biologic taxonomy, gene feature
mapping, ...
■ Cosine measure: If d1 and d2 are two vectors (e.g., term-frequency
vectors), then
cos(d1, d2) = (d1 ∙ d2) /||d1|| ||d2|| ,
where ∙ indicates vector dot product, ||d||: the length of vector d

161
 Example: Cosine Similarity

■ cos(d1, d2) = (d1 ∙ d2) /||d1|| ||d2|| ,

where ∙ indicates vector dot product, ||d|: the length of vector d

■ Ex: Find the similarity between documents 1 and 2.

d1 = (5, 0, 3, 0, 2, 0, 0, 2, 0, 0)
d2 = (3, 0, 2, 0, 1, 1, 0, 1, 0, 1)

d1∙d2 = 5*3+0*0+3*2+0*0+2*1+0*1+0*1+2*1+0*0+0*1 = 25
||d1||= (5*5+0*0+3*3+0*0+2*2+0*0+0*0+2*2+0*0+0*0) 0.5=(42)0.5 = 6.481
||d2||= (3*3+0*0+2*2+0*0+1*1+1*1+0*0+1*1+0*0+1*1) 0.5=(17)0.5 =
4.12
cos(d1, d2 ) = 0.94

162
Chapter 2: Getting to Know Your
Data

■ Data Objects and Attribute Types

■ Basic Statistical Descriptions of Data

■ Data Visualization

■ Measuring Data Similarity and Dissimilarity

■ Summary

163
 Summary
■ Data attribute types: nominal, binary, ordinal, interval-scaled, ratio-
scaled
■ Many types of data sets, e.g., numerical, text, graph, Web, image.
■ Gain insight into the data by:
■ Basic statistical data description: central tendency, dispersion,
graphical displays
■ Data visualization: map data onto graphical primitives
■ Measure data similarity
■ Above steps are the beginning of data preprocessing.
■ Many methods have been developed but still an active area of
research.
• Measuring Dispersion of Data:
• Example
• Consider following two attribute values:
• attr1: {2,3,4,5,6,7,8,9}
• attr2: {1,5,9,10,11,12,18,30}
• Which attribute has biggest standard deviation? Do not compute standard
deviations.
• Give interquartile ranges of attribute values?
• Are there any outliers (wrt IQR) in these datasets?
• Give a 4 element dataset whose standard deviation is zero?
• Measuring Dispersion of Data: Example
• Consider following two attribute values:
• attr1: {2,3,4,5,6,7,8,9}
• attr2: {1,5,9,10,11,12,18,30}
• Give interquartile ranges of attribute values?
• attr1:
• Q1: (3+4)/2=3.5 Q3:(7+8)/2=7.5 IQR:3.5-7.5 = 4
• attr2: Q1: (5+9)/2=7 Q3:(12+18)/2=15 IQR:7-15 = 8
• Are there any outliers (wrt IQR) in these datasets?
• Yes.
• 30 in attr2. 30 > 15+1.5*IQR
• Give a 4 element dataset whose standard deviation is zero?
• {1,1,1,1}
• Graphic Displays of Basic Statistical Descriptions
• Boxplot: graphic display of five-number summary
• Bar Chart: compare data across different categories.
• Histogram: x-axis are values, y-axis represent frequencies
• Quantile plot: each value xi is paired with fi indicating that approximately 100
fi % of data are < xi
• Quantile-quantile (q-q) plot: graphs the quantiles of one univariant distribution
against the corresponding quantiles of another.
• Scatter plot: each pair of values is a pair of coordinates and plotted as points in
the plane.
Histogram
• Data visualization provides an important suite of tools for identifying a qualitative
understanding. This can be helpful when we try to explore the dataset and extract
some information to know about a dataset and can help with identifying patterns,
corrupt data, outliers, and much more.
• What is Data Visualization?
• Data visualization is defined as a graphical representation that contains the
information and the data.

• By using visual elements like charts, graphs, and maps, data visualization
techniques provide an accessible way to see and understand trends, outliers, and
patterns in data.
• It is used in many areas such as:
• To model complex events.
• Visualize phenomenons that cannot be observed directly, such
as weather patterns, medical conditions, or mathematical
relationships.
• The basic uses of the Data Visualization technique are as
follows:
• It is a powerful technique to explore the data
with presentable and interpretable results.
• In the data mining process, it acts as a primary step in the
pre-processing portion.
• It supports the data cleaning process by finding incorrect data
and corrupted or missing values.
• It also helps to construct and select variables, which means
we have to determine which variable to include and discard in
the analysis.
• In the process of Data Reduction, it also plays a crucial role
while combining the categories.
• Different Types of Analysis for Data Visualization
• Mainly, there are three different types of analysis for Data Visualization:

• Univariate Analysis: In the univariate analysis, we will be using a single

feature to analyze almost all of its properties.

• Bivariate Analysis: When we compare the data between exactly 2

features then it is known as bivariate analysis.

• Multivariate Analysis: In the multivariate analysis, we will be comparing

more than 2 variables.
• Univariate Analysis Techniques for Data Visualization
• 1. Distribution Plot
• It is one of the best univariate plots to know about the distribution of data.
• When we want to analyze the impact on the target variable(output) with
respect to an independent variable(input), we use distribution plots a lot.
• This plot gives us a combination of both probability density functions(pdf)
and histogram in a single plot.
• Implementation:
• The distribution plot is present in the Seaborn package.
• The code snippet is as follows:
• [Link](hb,hue='SurvStat',size=5).map([Link],'age').add_legend()
• data visualization techniques | distribution plot
• Some conclusions inferred from the above distribution plot:
• From the above distribution plot we can conclude the following observations:
• We have observed that we created a distribution plot on the feature
‘Age’(input variable) and we used different colors for the Survival
status(output variable) as it is the class to be predicted.
• There is a huge overlapping area between the PDFs for different
combinations.
• In this plot, the sharp block-like structures are called histograms, and the
smoothed curve is known as the Probability density function(PDF).
• The Probability density function(PDF) of a curve can help us
to capture the underlying distribution of that feature which
is one major takeaway from Data visualization or
Exploratory Data Analysis(EDA).
• 2. Box and Whisker Plot
• This plot can be used to obtain more statistical
details about the data.
• The straight lines at the maximum and minimum are also
called whiskers.
• Points that lie outside the whiskers will be considered as an
outlier.
• The box plot also gives us a description of the 25th,
50th,75th quartiles.
• With the help of a box plot, we can also determine
the Interquartile range(IQR) where maximum details of
the data will be present. Therefore, it can also give us a
clear idea about the outliers in the dataset.
• Some conclusions inferred from the above box plot:
• From the above box and whisker plot we can conclude the
following observations:
• How much data is present in the 1st quartile and how many
points are outliers etc.
• For class 1, we can see that it is very little or no data is
present between the median and the 1st quartile.
• There are more outliers for class 1 in the feature
named axil_nodes.
• NOTE:
• We can get details about outliers that will help us to well
prepare the data before feeding it to a model since outliers
influence a lot of Machine learning models.
• Bivariate Analysis Techniques for Data Visualization
• 1. Line Plot
• This is the plot that you can see in the nook and corners of
any sort of analysis between 2 variables.
• The line plots are nothing but the values on a series of data
points will be connected with straight lines.
• The plot may seem very simple but it has more applications
not only in machine learning but in many other areas.