.

IN
S. J. P. N. TRUST’S
HIRASUGAR INSTITUTE OF TECHNOLOGY, NIDASOSHI

C
Accredited at 'A+' Grade by NAAC
Programmes Accredited by NBA: CSE & ECE

N
Department of Computer Science & Engineering

SY
U
VT

Module 1:
Chapter 2: Understanding of Data
Dr. S. V. Manjaragi
Asso. Prof. , Dept. of Computer Science & Engg.,
Hirasugar Institute of Technology, Nidasoshi
3/11/2025
 .IN
What is Data?

C
N
SY
• Data are facts
• Facts are in the form of numbers, audio, video, and
image U
VT
• Today, business organizations are accumulating vast
and growing amounts of data of the order of
gigabytes, tera bytes, exabytes.
• Data is available in different data sources like flat
files, databases, or data warehouses.
• Need to analyze data for taking decisions
 .IN
Characteristics of Big Data

C
• Volume: Since there is a reduction in the cost of storing devices,

N
there has been a tremendous growth of data.

SY
• Big Data is measured in terms of petabytes (PB) and exabytes (EB).
One exabyte is 1 million terabytes.

U
• Velocity: The fast arrival speed of data and its increase in data
volume is noted as velocity.
VT
• Variety : The variety of Big Data includes:
– Form: There are many forms of data. Data types range from
text, graph, audio, video, to maps.
– Function: These are data from various sources like human
conversations, transaction records, and old archive data.
– Source of data: The data source can be classified as open/public
data, social media data and multimodal data.
• Validity: Validity is the accuracy of the data for taking decisions
• Value: Value is the characteristic of big data that indicates the value
of the information that is extracted from the data.
 .IN
Types of Data

C
N
• In Big Data, there are three kinds of data.

SY
• Structured data
• Unstructured data
U
VT
• Semi-structured data
 .IN
Structured data

C
• In structured data, data is stored in an organized manner such
as a database where it is available in the form of a table.

N
• The data can also be retrieved in an organized manner using

SY
tools like SQL.
• Record data: We have a collection of objects in a dataset and
U
each object has a set of measurements.
VT
– The measurements can be arranged in the form of a
matrix.
– Rows in the matrix represent an object and can be called
as entities, cases, or records.
– The columns of the dataset are called attributes, features,
or fields.
• Graphics data: It involves the relationships among objects.
– For example, a web page can refer to another web page.
– This can be modeled as a graph.
– The nodes are web pages and the hyperlink is an edge that
connects the nodes.
 .IN
Structured data (Contd…)

C
• Data matrix: It is a variation of the record type

N
because it consists of numeric attributes.

SY
 The standard matrix operations can be applied on these
data.
• U
Ordered data : Ordered data objects involve attributes
VT
that have an implicit order among them.
₋ Temporal data : It is the data whose attributes are
associated with time. E.g: customer purchasing patterns
during festival time
– Sequence data: sequential data but does not have time
stamps, This data involves the sequence of words or
letters.
– Spatial data: It has attributes such as positions or areas.
For example, maps are spatial data where the points are
related by location.
 .IN
Unstructured data

C
• Video, image, programs

N
• Blog data

SY
• 80 % of the data is unstructured
U
VT
 .IN
Semi-structured data

C
• Semi-structured data are partially structured and

N
partially unstructured

SY
• XML/JSON objects
U
• RSS (Really Simple Syndication) feeds
VT
• Hierarchical records
 .IN
Data Storage: Flat Files

C
• Approaches to organize and manage data in storage files
and systems from flat file to data warehouses.

N
• Flat Files: These are the simplest and most commonly

SY
available data source.
• It is also the cheapest way of organizing the data.
U
• These flat files are the files where data is stored in plain
VT
ASCII format.
• Minor changes of data in flat files affect the results of the
data mining algorithms.
• Flat file is suitable only for storing small dataset.
• Some of the popular spreadsheet formats are listed below:
– CSV files - CSV stands for comma-separated value files
– TSV files - TSV stands for Tab separated values files where values
are separated by Tab.
• There are many tools like Google Sheets and Microsoft
Excel to process these files.
 .IN
Data Storage: Database System

C
• It normally consists of database files and a database
management system (DBMS).

N
• Database files contain original data and metadata.

SY
• DBMS aims to manage data and improve operator
performance .
U
• A relational database consists of sets of tables.
VT
• The tables have rows and columns.
• The columns represent the attributes and rows represent
tuples.
• A tuple corresponds to either an object or a relationship
between objects.
• A user can access and manipulate the data in the
database using SQL.
• Different types of databases
– A transactional database
– Time-series database : Log Files
 .IN
Data Storage: Other Types

C
• World Wide Web (WWW): It provides a diverse,
worldwide online information source.

N
– The objective of data mining algorithms is to mine

SY
interesting patterns of information present in WWW
• XML (eXtensible Markup Language): It is both human
U
and machine interpretable data format that can be used
VT
to represent data that needs to be shared across the
platforms.
• Data Stream It is dynamic data, which flows in and out of
the observing environment.
– Typical characteristics of data stream are huge volume of data,
dynamic, fixed order movement, and real-time constraints.
• RSS (Really Simple Syndication) It is a format for sharing
instant feeds across services.
• JSON (JavaScript Object Notation) It is another useful
data interchange format that is often used for many
machine learning algorithms.
 .IN
BIG DATA ANALYTICS

C
• The primary aim of data analysis is to assist business

N
organizations to take decisions.

SY
• For example, a business organization may want to know
which is the fastest selling product, in order for them to
U
market activities.
VT
• Data analysis and data analytics are terms that are used
interchangeably to refer to the same concept.
• Data analytics is a general term and data analysis is a part
of it.
• Data analytics refers to the process of data collection,
preprocessing and analysis.
• Data analysis is just analysis and is a part of data
analytics.
• Data analytics, instead, concentrates more on future and
helps in prediction.
 .IN
TYPES OF ANALYTICS

C
• There are four types of data analytics:

N
1. Descriptive analytics

SY
2. Diagnostic analytics
U
3. Predictive analytics
VT
4. Prescriptive analytics
 .IN
Descriptive Analytics

C
• It is about describing the main features of the data.

N
• It is often stated that analytics is essentially

SY
statistics.

U
• There are two aspects of statistics - Descriptive and
VT
Inference.
• Descriptive analytics only focuses on the description
part of the data.
• Inference is the process of drawing conclusions by
reasoning, logic and analytical thinking.
 .IN
Diagnostic Analytics

C
• It deals with the question - 'Why?'.

N
• This is also known as causal analysis, as it aims to

SY
find out the cause and effect of the events.

U
• For example, if a product is not selling, diagnostic
VT
analytics aims to find out the reason.
 .IN
Predictive Analytics

C
• It deals with the future.

N
• It deals with the question — 'What will happen in

SY
future given this data?'.

U
• This involves the application of algorithms to
VT
identify the patterns to predict the future.
• The entire course of machine learning is mostly
about predictive analytics
 .IN
Prescriptive Analytics

C
• Prescriptive analytics goes beyond prediction and

N
helps in decision making by giving a set of actions.

SY
• It helps the organizations to plan better for the

U
future and to mitigate the risks that are involved.
VT
 .IN
BIG DATA ANALYSIS FRAMEWORK

C
• Big data framework is a layered architecture.

N
• A 4-layer architecture has the following layers:

SY
1. Data connection layer
U
2. Data management layer
VT
3. Data analytics later
4. Presentation layer
 .IN
Data Connection Layer

C
• It has data ingestion mechanisms and data

N
connectors.

SY
• Data ingestion means taking raw data and
U
importing it into appropriate data structures.
VT
• It performs the tasks of ETL process.
• By ETL, it means extract, transform and load
operations.
 .IN
Data Management Layer

C
• It performs preprocessing of data.

N
• The purpose of this layer is to allow parallel

SY
execution of queries, and read, write and data
U
management tasks.
VT
• There may be many schemes that can be
implemented by this layer such as data-in-
place, where the data is not moved at all, or
constructing data repositories such as data
warehouses and pull data on-demand
mechanisms.
 .IN
Data Analytic Layer

C
• It has many functionalities such as statistical

N
tests, machine learning algorithms to

SY
understand, and construction of machine
U
learning models.
VT
• Types of Processing:
– Cloud Computing: IaaS, PaaS, SaaS, public, private,
hybrid cloud
– Grid Computing: Parallel and distributed
computing framework
– H-Computing or HPC
 .IN
Presentation Layer

C
• It has mechanisms such as dashboards, and

N
applications that display the results of

SY
analytical engines and machine learning
algorithms.
U
VT
• Big Data processing cycle involves data
management that consists of the following
steps.
– Data collection
– Data preprocessing
– Applications of machine learning algorithm
– Interpretation of results and visualization of
machine learning algorithm
 .IN
Data Collection

C
• The first task of gathering datasets are the collection

N
of data.
• A good quality data yields a better result.

SY
• Properties of ‘Good Data’:
U
– Timeliness: The data should be relevant and not stale or
VT
obsolete data.
– Relevancy : The data should be relevant and ready for the
machine learning or data mining algorithms.
– Knowledge about the data: The data should be
understandable and interpretable
• Classification of data source:
– Open or public data source : It is a data source that does
not have any stringent copyright rules or restrictions. E.g:
Government census data
– Social media: Twitter, Facebook, YouTube, and Instagram.
– Multimodal data: It includes data that involves many
modes such as text, video, audio and mixed types.
 .IN
Data Preprocessing

C
• In real world, the available data is 'dirty'.

N
• By this word 'dirty', it means:
–

SY
Incomplete data
– Inaccurate data
– Outlier data
– U
Data with missing values
VT
– Data with inconsistent value
– Duplicate data
• Data preprocessing improves the quality of the data mining
techniques.
• The raw data must be preprocessed to give accurate
results.
• The process of detection and removal of errors in data is
called data cleaning.
• Data wrangling means making the data processable for
machine learning algorithms.
 Illustration of 'Bad' Data

.IN
C
Date of Birth
Patient ID Name Age Fever Salary

N
(DoB)

SY
1. John 21 Low -1500

U
VT
1. Andre 36 High Yes
,, ,,
1. David 5 10/10/1980 Low

1. Raju 136 High Yes

Outliers are data that exhibit the characteristics that

are different from other data and have very unusual
values.
 .IN
Missing Data Analysis

C
• Data cleaning process is missing data analysis.

N
• Data cleaning routines attempt to fill up the

SY
missing values, smoothen the noise while
U
identifying the outliers and correct the
VT
inconsistencies of the data.
• This enables data mining to avoid overfitting
of the models.
 .IN
Missing Data Analysis:
solve the problem of missing data

C
N
• Ignore the tuple - A tuple with missing data, especially the
class label, is ignored.

SY
• Fill in the values manually - Here, the domain expert can
analyse the data tables and carry out the analysis and fill in
U
the values manually.
VT
• A global constant can be used to fill in the missing
attributes. The missing values may be 'Unknown' or be
'Infinity'.
• The attribute value may be filled by the attribute value.
Say, the average income can replace a missing value.
• Use the attribute mean for all samples belonging to the
same class. Here, the average value replaces the missing
values of all tuples that fall in this group.
• Use the most possible value to fill in the missing value. The
most probable value can be obtained from other methods
like classification and decision tree prediction.
 .IN
Removal of Noisy or Outlier Data

C
• Noise is a random error or variance in a measured

N
value.

SY
• It can be removed by using binning, which is a method
where the given data values are sorted and distributed
U
into equal frequency bins.
VT
• The bins are also called as buckets.
• The binning method then uses the neighbor values to
smooth the noisy data.
• 'smoothing by means' where the mean of the bin
removes the values of the bins
• 'smoothing by bin medians' where the bin median
replaces the bin values
• 'smoothing by bin boundaries' where the bin value is
replaced by the closest bin boundary. The maximum
and minimum values are called bin boundaries.
 .IN
Example

C
• Consider the following set:

N
SY
S = {12, 14, 19, 22, 24, 26, 28, 31, 34}. Apply
various binning techniques and show the result.
U
VT
• Solution: By equal-frequency bin method, the
data should be distributed across bins.
• Let us assume the bins of size 3, then the
above data is distributed across the bins
 Noisy Data

.IN
BINNING TECHNIQUE

C
N
SY
U
VT
 .IN
Data Integration and Data Transformations

C
• Data integration involves routines that merge data

N
from multiple sources into a single data source.

SY
• this may lead to redundant data.
U
• The main goal of data integration is to detect and
VT
remove redundancies that arise from integration.
• Data transformation routines perform operations like
normalization to improve the performance of the data
mining algorithms.
• In normalization, the attribute values are scaled to fit
in a range (say 0-1)
• Some of the normalization procedures used are:
– Min-Max
– z-Score
 .IN
Min-Max Procedure

C
• It is a normalization technique where each

N
variable V is normalized by its difference with

SY
the minimum value divided by the range to a
U
new range, say 0-1.
VT

• Here max-min is the range.

• Min and max are the minimum and maximum
of the given data
• new max and new min are the minimum and
maximum of the target range, say 0 and 1
 .IN
Min-Max Procedure : Example
• Consider the set: V = {88, 90, 92, 94}. Apply Min-Max procedure

C
and map the marks to a new range 0-1.

N
SY
U
VT
 .IN
z-Score Normalization
• This procedure works by taking the difference between the

C
field value and mean value, and by scaling this difference by

N
standard deviation of the attribute.

SY
U
VT
 .IN
DESCRIPTIVE STATISTICS

C
• Descriptive statistics is a branch of statistics that

N
does dataset summarization.

SY
• It is used to summarize and describe data.
U
• Data visualization is a branch of study that is
VT
useful for investigating the given data.
• Mainly, the plots are useful to explain and
present data to customers.
• Descriptive analytics and data visualization
techniques help to understand the nature of the
data,
• Exploratory Data Analysis (EDA): helps to
determine the kinds of machine learning or data
mining tasks that can be applied to the data
 .IN
Dataset and Data Types
• A dataset can be assumed to be a collection of

C
N
data objects.

SY
• The data objects may be records, points, vectors,
patterns, events, cases, samples or observations.
U
• These records contain many attributes.
VT
• An attribute can be defined as the property or
characteristics of an object.
• The type of attribute determines the data types,
often referred to as measurement scale types.
 .IN
Types of data

C
N
SY
U
VT
 .IN
Categorical or Qualitative Data
• The categorical data can be divided into two types.

C
N
They are nominal type and ordinal type.

SY
– Nominal Data - In Table 2.2, patient ID is nominal data.
Nominal data are symbols and cannot be processed like a
number.
U
– Nominal data type provides only information but has no
VT
ordering among data.
– Only operations like (=, *) are meaningful for these data.
– Ordinal Data - It provides enough information and has
natural order. For example, Fever= (Low, Medium, High) is
an ordinal data.
 .IN
Numeric or Quantitative Data
• It can be divided into two categories. They are

C
interval type and ratio type.

N
– Interval Data - Interval data is a numeric data for

SY
which the differences between values are
meaningful.
U
– For example, there is a difference between 30 degree
VT
and 40 degree.
– Ratio Data - For ratio data, both differences and ratio
are meaningful.
• Discrete Data This kind of data is recorded as
integers. Employee identification number such as
10001 is discrete data.
• Continuous Data It can be fitted into a range and
includes decimal point. For example, age is a
continuous data.
 .IN
C
N
• classifying the data is based on the number of

SY
variables used in the dataset
U
• the data can be classified as univariate data,
VT
bivariate data, and multivariate data
• In case of univariate data, the dataset has only
one variable. A variable is also called as category.
• Bivariate data indicates that the number of
variables used are two and multivariate data uses
three or more variables.
 .IN
UNIVARIATE DATA ANALYSIS AND

C
VISUALIZATION

N
• Univariate analysis is the simplest form of statistical

SY
analysis.
U
• As the name indicates, the dataset has only one
VT
variable.
• A variable can be called as a category.
• Univariate does not deal with cause or relationships.
• The aim of univariate analysis is to describe data and
find patterns.
• Univariate data description involves finding the
frequency distributions, central tendency measures,
dispersion or variation, and shape of the data.
 .IN
Data Visualization

C
N
• Data visualization helps to understand data.

SY
• It helps to present information and data to
U
VT
customers.
• Some of the graphs that are used in
univariate data analysis are:
– bar charts,
– histograms,
– frequency polygons and
– pie charts.
 .IN
Bar Chart

C
• A Bar chart (or Bar graph) is used to display the frequency

N
distribution for variables.

SY
• Bar charts are used to illustrate discrete data.
• The charts can also help to explain the counts of nominal
data. U
VT
• It also helps in comparing the frequency of different
groups.
 .IN
Pie Chart

C
• These are equally helpful in illustrating the univariate data.

N
• The percentage frequency distribution of students' marks

SY
[122, 22, 40, 40, 70, 70, 70, 85, 90, 90]

U
VT
 Histogram

.IN
• It plays an important role in data mining for showing frequency

C
distributions.
• Histograms can be used as charts to show frequency, skewness

N
present in the data, and shape./ Mode(peak)

SY
• The histogram for students' marks {45, 60, 60, 80, 85} in the group
range of 0-25, 26-50, 51-75, 76-100 is

U
VT
 Dot Plots

.IN
• These are similar to bar charts.

C
• They are less clustered as compared to bar

N
charts, as they illustrate the bars only with single

SY
points
U
VT
 Central Tendency

.IN
• Summary of the data is called central

C
tendency.

N
SY
• Central tendency can explain the
characteristics of data and that further helps
U
VT
in comparison.
• Mass data have tendency to concentrate at
certain values, normally in the central
location. It is called measure of central
tendency (or averages).
• Popular measures are mean, median and
mode.
 Central Tendency-Mean

.IN
• Mean - Arithmetic average (or mean) is a measure of

C
central tendency that represents the 'center' of the

N
dataset.

SY
• E.g: average income or average traffic.

U
VT

• Weighted mean - Unlike arithmetic mean that gives the

weightage of all items equally, weighted mean gives
different importance to all items as the item importance
varies.
 Central Tendency-Mean

.IN
• Geometric mean - Let x1, x2, ••• xN be a set of 'N'

C
values or observations. Geometric mean is the Nth

N
root of the product of N items.

SY
• The formula for computing geometric mean is given
U
as follows:
VT
 Central Tendency-Meadian

.IN
• Median - The middle value in the distribution is

C
called median.

N
• If the total number of items in the distribution is odd,

SY
then the middle value is called median.
U
• If the numbers are even, then the average value of
VT
two items in the centre is the median.
 Central Tendency-Mode

.IN
• Mode - Mode is the value that occurs more

C
frequently in the dataset.

N
• In other words, the value that has the highest

SY
frequency is called mode.
U
• The procedure for finding the mode is to calculate
VT
the frequencies for an the values in the data, and
mode is the value (or values) with the highest
frequency.
• Normally, the dataset is classified as unimodal,
bimodal and trimodal with modes 1, 2 and 3,
respectively.
 Dispersion

.IN
• The spreadout of a set of data around the central

C
tendency (mean, median or mode) is called dispersion.

N
• Dispersion is represented by various ways such as range,

SY
variance, standard deviation, and standard error.

U
• Range Range is the difference between the maximum
VT
and minimum of values of the given list of data.
• Standard deviation is the average distance from the
mean of the dataset to each point.
 Quartiles and Inter Quartile Range

.IN
• Percentiles are about data that are less than the

C
coordinates by some percentage of the total value.

N
• Kth percentile is the property that the k% of the data lies

SY
at or below Xi .
• median is 50th percentile and can be denoted as Q0.50.
(Q2)
U
• The 25th percentile is called first quartile (Q1)
VT
• The 75th percentile is called third quartile (Q3).
• Another measure that is useful to measure dispersion is
Inter Quartile Range (IQR).
• The IQR is the difference between Q3 and Q1.
• Interquartile percentile = Q3 — Q1
• Outliers are normally the values falling apart at least by
the amount 1.5 x IQR above the third quartile or below
the first quartile.
 Quartiles and Inter Quartile Range

.IN
• E.g: For patients' age list {12, 14, 19, 22, 24, 26, 28,

C
31, 34}, find the IQR.

N
• The median is in the fifth position. In this case, 24 is

SY
the median.
• The first quartile is median of the scores below the
U
mean i.e., {12, 14, 19, 22}.
VT
• In this case, the median is the average of the second
and third values, that is, Q0.25 = 16.5.
• Similarly, the third quartile is the median of the
values above the median, that is {26, 28, 31, 34}
• Q0.75 is the average of the seventh and eighth score.
In this case, it is 28 + 31/2 = 59/2 = 29.5.
• IQR= Q0.75 — Q0.25= 29.5-16.5=13
 Five-point Summary and Box Plots

.IN
• The median, quartiles Q1 and Qy and minimum and

C
maximum written in the order < Minimum, Q1,

N
Median, Q3 Maximum > is known as five-point

SY
summary.
U
• Find the 5-point summary of the list {13, 11, 2, 3, 4,
VT
8, 9}.
• The minimum is 2 and the maximum is 13.
• The Q1, Q2 and Q3 are 3, 8 and 11, respectively.
• 5-point summary is {2, 3, 8, 11, 13}
5-point summary is {2, 3, 8, 11, 13} : Box Plot

.IN
C
N
SY
U
VT
 .IN
Shape
• Skewness and Kurtosis (called moments)

C
N
indicate the symmetry/asymmetry and peak

SY
location of the dataset.
• Ideally, skewness should be zero as in ideal
U
VT
normal distribution.
• More often, the given dataset may not have
perfect symmetry
 .IN
Shape
• Skewness and Kurtosis (called moments)

C
N
indicate the symmetry/asymmetry and peak

SY
location of the dataset.
• Ideally, skewness should be zero as in ideal
U
VT
normal distribution.
• More often, the given dataset may not have
perfect symmetry
 .IN
Shape
• The dataset may also either have very high values or

C
extremely low values.

N
• If the dataset has far higher values, then it is said to be

SY
skewed to the right.

U
• On the other hand, if the dataset has far more low values
VT
then it is said to be skewed towards left.
• If the tail is longer on the left-hand side and hump on the
right-hand side, it is called positive skew. Otherwise, it is
called negative skew.
 .IN
Shape
• if the data is skewed, then there is a greater chance of outliers

C
in the dataset.

N
• This affects the mean and median. Hence, this may affect the

SY
performance of the data mining algorithm.
• A perfect symmetry means the skewness is zero.
U
• In the case of skew, the median is greater than the mean.
VT
• In positive skew, the mean is greater than the median.
• for negatively skewed distribution, the median is more than
the mean.
• The relationship between skew and the relative size of the
mean and median can be summarized by a convenient
numerical skew index known as Pearson 2 skewness
coefficient
 .IN
Shape
• The relationship between skew and the relative size of the

C
mean and median can be summarized by a convenient

N
numerical skew index known as Pearson 2 skewness

SY
coefficient

U
VT
 Kurtosis

.IN
• Kurtosis also indicates the peaks of data.

C
• If the data is high peak, then it indicates higher kurtosis and

N
SY
vice versa.
• Kurtosis is the measure of whether the data is heavy tailed or
U
light tailed relative to normal distribution.
VT
• It can be observed that normal distribution has bell-shaped
curve with no long tails.
• Low kurtosis tends to have light tails.
• The implication is that there is no outlier data.
 .IN
Mean Absolute Deviation (MAD)
• MAD is another dispersion measure and is robust to outliers.

C
• Normally, the outlier point is detected by computing the

N
SY
deviation from median and by dividing it by MAD.
• Here, the absolute deviation between the data and mean is
taken.
U
VT
• Thus, the absolute deviation is given as:
 .IN
Coefficient of Variation (CV)

C
• Coefficient of variation is used to compare datasets

N
with different units.

SY
• CV is the ratio of standard deviation and mean, and
U
%CV is the percentage of coefficient of variations.
VT
 .IN
Special Univariate Plots

C
• The ideal way to check the shape of the dataset is a

N
stem and leaf plot.

SY
• A stem and leaf plot are a display that help us to
U
know the shape and distribution of the data.
VT
• In this method, each value is split into a 'stem' and a
'leaf.
• The last digit is usually the leaf and digits to the left
of the leaf mostly form the stem.
 .IN
Special Univariate Plots

C
N
SY
U
VT
 .IN
A Q-Q plot

C
• A Q-Q plot can be used to assess the shape of the

N
dataset.

SY
• The Q-Q plot is a 2D scatter plot of an univariate
U
data against theoretical normal distribution data or
VT
of two datasets — the quartiles of the first and
second datasets.