MODULE -3
Machine Learning (ML) is a promising and flourishing field. It can enable top management
of an organization to extract the knowledge from the data stored in various archives of the
business organizations to facilitate decision making. Such decisions can be useful for
organizations to design new products, improve business processes, and to develop decision
support systems.
1.1 NEED FOR MACHINE LEARNING
Machine learning has become so popular because of three reasons:
1. High volume of available data to manage:
● Big companies such as Facebook, Twitter, and YouTube generate huge
amount of data that grows at a phenomenal rate.
● It is estimated that the data approximately gets doubled every year
2. The cost of storage has reduced.
The hardware cost has also dropped. Therefore, it is easier now to capture, process,
store, distribute, and transmit the digital information.
3. Availability of complex algorithms now
With the advent of deep learning, many algorithms are available for machine
learning.
Knowledge Pyramid
Data Information
● Processed data is called information.
● It includes patterns, associations, or relationships among data.
● Example: analyzing sales data to find the fastest-selling product.
Information Knowledge
● Condensed or summarized information is called knowledge.
● It involves understanding historical patterns and predicting future trends.
● Example: insights from past sales data used to forecast demand.
Knowledge Intelligence
● Knowledge becomes useful only when applied.
● Intelligence is actionable knowledge used for decision-making and actions.
● Computer systems have been successful up to this stage.
Intelligence Wisdom
● The highest level of the knowledge pyramid is wisdom.
● It represents maturity of mind, judgment, and deeper
understanding.
● So far, wisdom is considered a uniquely human capability.
1.2 MACHINE LEARNING EXPLAINED
● Machine Learning is a branch of Artificial Intelligence that enables systems to learn
from data without explicit programming (Arthur Samuel).
● Replaces traditional step-by-step programming and expert systems like MYCIN, which
could not adapt.
● Learns patterns from data to build models for prediction and decision-making.
● Core idea: y = f(x) (maps input to output).
● A model is a global representation of patterns (equations, trees, rules, clusters).
● Learning follows (E, T, P): Experience (data), Task, Performance (improves over time).
● Process: Data collection Abstraction Generalization.
●
Similar to humans: data algorithm model (like experience thinking decision).
● Better data quality leads to more accurate results.
1.3 MACHINE LEARNING IN RELATION TO OTHER FIELDS
● Machine learning combines concepts from Artificial Intelligence, Data Science,
and Statistics.
It is an interdisciplinary field formed by integrating ideas from multiple domains.
1.3.1 Machine Learning and Artificial Intelligence
● Machine learning is a sub-branch of Artificial Intelligence, which focuses on
building intelligent agents.
These agents can be robots, humans, or autonomous systems capable of decision-
making.
● Early AI aimed to create human-like intelligence using logic and reasoning.
However, progress slowed during periods known as AI winters
due to limitations of rule-based systems.
● AI regained momentum with data-driven approaches that focus on finding patterns
in data.
Machine learning enables systems to learn from data and make predictions on new inputs.
● Machine learning includes techniques like learning from examples and
reinforcement learning.
It allows models to handle unknown instances and generate useful outputs.
● Deep learning is a subfield of machine learning that uses neural networks.
These networks are inspired by human neurons and use activation functions to
process information.
1.3.2 Machine Learning, Data Science, Data Mining, and Data Analytics
● Data science is a broad “umbrella” field that includes many disciplines.
Machine learning is one of its key components and relies heavily on data.
● Data science focuses on collecting, processing, and analyzing data for insights.
Machine learning uses this data to build models for prediction and decision-making.
● Data mining involves extracting hidden patterns from large datasets.
Data analytics focuses on interpreting data to support decisions and gain insights.
● All these fields are interconnected and work together.
They transform raw data into meaningful information, knowledge, and intelligent
actions.
Big Data
● Big data is a part of data science that deals with very large and complex
datasets.
It is characterized by three main factors: Volume, Variety, and Velocity.
● Volume refers to the massive amount of data generated by platforms like
social media.
Variety means data exists in multiple forms such as text, images, and videos.
Velocity indicates the speed at which data is generated and processed.
● Big data supports machine learning applications like language translation and
image recognition.
It also plays a major role in the growth of deep learning, which uses neural
networks.
Data Mining
● Data mining originated in business to discover useful information from large
datasets.
It is similar to mining valuable resources from the earth.
● It focuses on extracting hidden patterns and relationships in data. These insights
help organizations make better decisions.
● Though often confused with machine learning, they differ slightly. Data mining
extracts patterns, while machine learning uses them for prediction.
Data Analytics
● Data analytics is another branch of data science focused on analyzing raw
data.
Its goal is to extract meaningful insights for decision-making.
● It includes different types such as predictive analytics. Predictive analytics uses
past data to forecast future outcomes.
● Machine learning is closely related to data analytics.
Many algorithms used in predictive analytics are shared with machine learning.
Pattern Recognition
● Pattern recognition is an engineering field that uses machine learning
techniques.
It focuses on identifying patterns and regularities in data.
● It involves feature extraction and classification of patterns. This makes it a
specific application area of machine learning.
1.3.3 Machine Learning and Statistics
● Statistics is a mathematical field with a strong theoretical foundation.
Like machine learning, it also learns from data.
● Statistical methods focus on identifying patterns and relationships in data.
They often begin with a hypothesis and test it through experiments.
● The key difference is in approach.
Statistics is hypothesis-driven, while machine learning is more data-driven and
predictive.
1.4 TYPES OF MACHINE LEARNING
Machine learning is broadly divided into four types.
Each type is based on how the system learns from data.
● Supervised Learning
○ Uses labelled data where the correct output is known.
○ Includes tasks like classification (categorizing data) and regression
(predicting values).
● Unsupervised Learning
○ Works with unlabelled data where no predefined output is given.
○ Includes clustering, association mining, and dimension reduction.
● Semi-Supervised Learning
○ Combines both labelled and unlabelled data.
○ Useful when labelled data is limited but unlabelled data is abundant.
● Reinforcement Learning
○ Learning happens through interaction with an environment.
○ The system improves by receiving rewards or penalties for its actions.
Labelled and Unlabelled Data
● Data is a collection of raw facts, usually represented in tabular form.
Each row represents a data point (sample), and each column represents a feature
(attribute).
● A label is a special feature that we want to predict.
Based on this, data is classified into labelled and unlabelled types.
1.4.1 Supervised Learning
● Supervised learning uses labelled data, where each input has a corresponding
correct output.
A “supervisor” or teacher provides this data to guide the learning process.
● The goal is to build a model that can learn from this data and predict outputs
for new, unseen inputs.
It is one of the most commonly used types of machine learning.
Two Stages of Supervised Learning
● Stage 1: Learning (Training Phase)
○ Similar to a teacher teaching a student.
○ The model is given input data along with correct answers (labels).
○ The model learns patterns and relationships from this data.
○ At this stage, it is not yet clear how well the model has understood.
● Stage 2: Testing (Evaluation Phase)
○ The model is tested using new questions (test data).
○ Its predictions are compared with the correct answers.
○ Based on performance, errors are identified and
improvements are made.
● Through repeated training and testing, the model improves its accuracy.
This process helps the system make reliable predictions on unknown data
Supervised Learning Methods
● Supervised learning has two main methods:
○ Classification
○ Regression
Classification
● Classification is a supervised learning technique used when the output is
categorical (discrete values).
The goal is to assign a label to input data based on learned patterns.
● In classification:
○ Input features are called independent variables.
○ Output feature is called label or dependent variable.
○ A classification model represents the relationship between inputs and labels.
Working of Classification
● Classification works in two main stages:
● 1. Training Stage
○ A labelled dataset is given to the algorithm.
○ The model learns patterns from the data.
○ A classification model is created based on this learning.
● 2. Testing Stage
○ The trained model is given new, unseen data.
○ The model predicts the correct label for the input.
○ This evaluates how well the model has learned.
Example of Classification
● Example: Dog vs Cat image classification
○ The model is trained using labelled images of dogs and cats.
○ After training, it can classify a new image as either dog or cat.
● Another example: Iris dataset
○ Input: (6.3, 2.9, 5.6, 1.8, ?)
○ Output: The model predicts the correct flower class.
● Applications include:
○ Image recognition
○ Disease detection (e.g., cancer classification)
○ Plant species identification
Classification Algorithms
● Some important classification algorithms include Decision Tree, Random Forest,
Support Vector Machines (SVM), Naïve Bayes, Artificial Neural Networks
(ANN), and deep learning models like CNN.
These algorithms are used to assign data into predefined categories or labels.
Regression Models
● Regression models are supervised learning techniques used to predict
continuous numerical values such as price, sales, or temperature.
Unlike classification, the output here is a number rather than a category.
● A regression model fits a function of the form y=f(x) often represented
as a line in linear regression.
For example, sales prediction can be modeled as: y=0.66x+0.54y where x is the
week and y is product sales.
● In this equation, coefficients like 0.66 and 0.54 are learned from training data.
● Linear regression is one of the most widely used regression techniques and
forms the basis of many predictive models.
Classification vs Regression (Key Difference)
Aspect Classification Regression
Type Supervised Learning Supervised Learning
Output Discrete labels Continuous values
Goal Assign category Predict quantity
Example Spam / Not spam Price, Temperature
Evaluation Accuracy Error (e.g., difference from actual)
Nature Qualitative Quantitative
1.4.2 Unsupervised Learning
● Unsupervised learning is a self-learning process where there is no teacher or
supervisor.
The system learns on its own by observing data and identifying patterns.
● Since there are no labels provided, the algorithm relies on trial and error and
pattern discovery.
It groups similar data together based on shared characteristics.
● In this method, the input consists of objects without labels.
The algorithm analyzes the data and organizes it into meaningful structures.
Types of Unsupervised Learning Algorithms
● The main examples include:
○ Cluster analysis
○ Dimensionality reduction
Cluster Analysis
● Cluster analysis is a technique used to group similar data objects into clusters.
Objects within the same cluster are highly similar, while objects in different clusters are
significantly different.
● It works by identifying patterns in the data and forming natural groupings.
Each cluster represents a specific structure or category within the dataset.
● Applications of clustering include:
○ Image segmentation (identifying regions in images)
○ Detecting abnormalities in medical images
○ Grouping gene patterns in biological databases
Example of Clustering
● In a dataset containing images of dogs and cats, clustering can automatically
group them.
Without labels, the algorithm forms two clusters: one for dogs and one for cats.
Similar images are grouped together, while different ones are separated.
Common Clustering Algorithms
● Some key clustering techniques include:
○ K-means algorithm
○ Hierarchical clustering algorithm
Dimensionality Reduction
● Dimensionality reduction is an unsupervised learning technique.
It takes high-dimensional data as input and converts it into lower-
dimensional data.
● The main idea is to reduce the number of features while preserving important
information.
This is done by using the variance present in the data to retain meaningful structure.
● It simplifies datasets without losing their overall generality.
This helps improve efficiency and visualization of complex data.
1.4.3 Semi-Supervised Learning
● Semi-supervised learning occurs when a dataset contains both labelled and
unlabelled data.
Labeling data is often expensive and time-consuming, so this approach is widely
used.
● The algorithm first assigns pseudo-labels to unlabelled data. Then, both labelled
and pseudo-labelled data are combined for training.
● This method improves learning performance when labelled data is limited.
It reduces dependency on fully labelled datasets.
1.4.4 Reinforcement Learning
● Reinforcement learning is inspired by how humans and animals learn from
interaction.
An agent learns by interacting with an environment and receiving feedback.
● The agent can be a human, robot, animal, or software program.
It takes actions and receives rewards or punishments based on those actions.
● The goal of the agent is to maximize cumulative reward over time.
Positive rewards reinforce good behavior, while negative rewards discourage wrong
actions.
● Over time, the agent improves its decision-making through experience.
This leads to learning optimal strategies for different situations.
1.5 Challenges of Machine Learning
Overview of Challenges
Machine learning systems can perform complex computational tasks faster than
humans. However, they still struggle in areas that require understanding, perception,
and high-quality data.
The performance of machine learning models heavily depends on data quality and
system design. Many real-world problems remain difficult due to practical limitations.
1. Well-Posed vs Ill-Posed Problems
● Machine Learning works well for well-posed problems with clearly defined inputs.
● Struggles with ill-posed problems where information is incomplete or unclear.
2. Huge Data Requirement
● Requires large amounts of high-quality data.
● Data must be clean and consistent (avoid missing or incorrect values).
3. High Computational Power
● Needs significant computational resources for large data and complex models.
● Uses advanced hardware like GPUs and TPUs.
4. Algorithm Complexity
● Involves selecting and evaluating suitable algorithms.
● Requires expertise for tuning and optimization.
5. Bias–Variance Tradeoff
● Models must balance bias and variance for good performance.
● Overfitting (too complex) and underfitting (too simple) are common issue
1.6 MACHINE LEARINING PROCESS
● The emerging process model for the data mining solutions for business
organizations is CRISP-DM.
● Since machine learning is like data mining, except for the aim, this process can be
used for machine learning. CRISP-DM stands for Cross Industry Standard Process –
Data Mining.
● This process involves six steps:
1. Understanding the Business
● Define business objectives and problem clearly.
● Identify suitable approach/algorithm.
2. Understanding the Data
● Collect and explore data patterns and relationships.
● Understand data meaning for modeling.
3. Data Preparation
● Clean and transform data for analysis.
● Handle missing/poor-quality data.
4. Modeling
● Apply Machine Learning algorithms.
● Build a model to capture patterns.
5. Evaluation
● Test model performance (accuracy, effectiveness).
● Use analysis and domain knowledge.
6. Deployment
● Implement model in real-world systems.
● Use results for decision-making/improvement.
1.7 Machine Learning Applications
Sentiment Analysis
● Sentiment analysis is an NLP application that converts text into emotions like
happy, sad, or angry.
It is widely used in movie and product reviews.
● It can automatically assign ratings such as stars based on user opinions.
Recommendation Systems
● Recommendation systems provide personalized suggestions to users.
They analyze user behavior and preferences.
● Examples include Amazon recommending books and Netflix suggesting
movies or shows.
Voice Assistants
● Voice assistants like Alexa, Siri, Cortana, and Google Assistant use machine
learning.
They understand voice commands and perform actions accordingly.
● These systems function as intelligent chatbots.
Navigation Systems
● Applications like Google Maps and Uber use machine learning techniques.
They help in finding shortest or fastest routes for navigation.
● This improves efficiency and reduces travel time.
◻ 1. Understanding Data
What is Data?
All facts are data. In computer systems, bits encode facts present in numbers, text, images,
audio, and video. Data can be directly human interpretable (such as numbers or texts) or
diffused data such as images or video that can be interpreted only by a computer.
1. Volume - Since there is a reduction in the cost of storing devices, there has been a
tremendous growth of data. Small traditional data is measured in terms of gigabytes
(GB) and terabytes (TB), but Big Data is measured in terms of petabytes (PB) and
exabytes (EB). One exabyte is 1 million terabytes.
2. Velocity - The fast arrival speed of data and its increase in data volume is noted as
velocity. The availability of IoT devices and Internet power ensures that the data is
arriving at a faster rate. Velocity helps to understand the relative growth of big data
and its accessibility by users, systems and applications.
3. Variety - The variety of Big Data includes:
a. Form -There are many forms of data. Data types range from text, graph,
audio, video, to maps. There can be composite data too, where one media
can have many other sources of data, for example, a video can have an audio
song.
b. Function - These are data from various sources like human conversations,
transaction records, and old archive data.
c. Source of data - This is the third aspect of variety. There are many sources of
data. Broadly, the data source can be classified as open/public data, social
media data and multimodal data. These are discussed in Section 2.3.1 of this
chapter.
4. Veracity of data – Veracity of data deals with aspects like conformity to the facts,
truthfulness, believability, and confidence in data. There may be many sources of
error such as technical errors, typographical errors, and human errors. So, veracity is
one of the most important aspects of data.
5. Validity - Validity is the accuracy of the data for taking decisions or for any other goals
that are needed by the given problem.
6. Value - Value is the characteristic of big data that indicates the value of the
information that is extracted from the data and its influence on the decisions that are
taken based on it.
Data refers to raw facts, figures, or observations collected for analysis.
● Example: numbers, text, images, videos, logs, etc.
● Data becomes information when processed and interpreted.
Types of Data
In Big Data, there are three kinds of data. They are structured data, unstructured data, and
semi-structured data.
Structured Data
In structured data, data is stored in an organized manner such as a database where it is
available in the form of a table. The data can also be retrieved in an organized manner using
tools like SQL.
The structured data frequently encountered in machine learning are listed below:
1. Record Data :
a. A dataset is a collection of measurements taken from a [Link] have a
collection of objects in a dataset and each object has a set of measurements.
The measurements can be arranged in the form of a matrix.
b. Rows in the matrix represent an object and can be called as entities, cases, or
records.
c. The columns of the dataset are called attributes, features, or fields. The table
is filled with observed data.
d. Label is the term that is used to describe the individual observations.
2. Data Matrix It is a variation of the record type because it consists of numeric
attributes. The standard matrix operations can be applied on these data. The data is
thought of as points or vectors in the multidimensional space where every attribute is
a dimension describing the object. Graph Data It involves the relationships among
objects. For example, a web page can refer to another web page. This can be
modeled as a graph. The modes are web pages and the hyperlink is an edge that
connects the nodes.
3. Ordered Data
Ordered data objects involve attributes that have an implicit order among them. The
examples of ordered data are:
a. Temporal data - It is the data whose attributes are associated with time. For
example, the customer purchasing patterns during festival time is sequential
data. Time series data is a special type of sequence data where the data is a
series of measurements over time.
b. Sequence data - It is like sequential data but does not have time stamps. This
data involves the sequence of words or letters. For example, DNA data is a
sequence of four characters -ATG C.
c. Spatial data – It has attributes such as positions or areas. For example, maps
are spatial data where the points are related by location.
Unstructured Data
Unstructured data includes video, image, and audio. It also includes textual documents,
programs, and blog data. It is estimated that 80% of the data are unstructured data.
Semi-Structured Data
Semi-structured data are partially structured and partially unstructured. These include data
like XML/JSON data, RSS feeds, and hierarchical data.
BIG DATA ANALYTICS AND TYPES OF
ANALYTICS
The primary aim of data analysis is to assist business organizations to take decisions. For
example, a business organization may want to know which is the fastest selling product, in
order for them to market activities. Data analysis is an activity that takes the data and
generates useful information and insights for assisting the organizations.
There are four types of data analytics:
1. Descriptive analytics
2. Diagnostic analytics
3. Predictive analytics
4. Prescriptive analytics
Descriptive Analytics
● It is about describing the main features of the data.
● After data collection is done, descriptive analytics deals with the collected data and
quantifies it.
● It is often stated that analytics is essentially statistics.
● There are two aspects of statistics - Descriptive and Inference.
● Descriptive analytics only focuses on the description part of the data and not the
inference part.
Diagnostic Analytics
● It deals with the question - 'Why?".
● This is also known as causal analysis, as it aims to find out the cause and effect of the
events.
● For example, if a product is not selling diagnostic analytics aims to find out the reason.
● There may be multiple reasons and associated effects are analyzed as part of it.
Predictive Analytics
● It deals with the future.
● It deals with the question - 'What will happen in future given this data?".
● This involves the application of algorithms to identify the patterns to predict the
future.
● The entire course of machine learning is mostly about predictive analytics and forms
the core of this book.
Prescriptive Analytics
● It is about the finding the best course of action for the business organizations.
● Prescriptive analytics goes beyond prediction and helps in decision making by giving
a set of actions.
● It helps the organizations to plan better for the future and to mitigate the risks that are
involved.
BIG DATA ANALYSIS FRAMEWORK
For performing data analytics, many frameworks are proposed. All proposed analytics
frameworks have some common factors. Big data framework is a layered architecture. Such
an architecture has many advantages such as genericness. A 4-layer architecture has the
following layers:
1. Date connection layer
2. Data management layer
3. Data analytics later
4. Presentation layer
Data Connection Layer
It has data ingestion mechanisms and data connectors. Data ingestion means taking raw
data and importing it into appropriate data structures. It performs the tasks of ETL process.
By ETL, it means extract, transform and load operations.
Data Management Layer
It performs preprocessing of data. The purpose of this layer is to allow parallel execution of
queries, and read, write and data management tasks. There may be many schemes that can
be implemented by this layer such as data-in-place, where the data is not moved at all, or
constructing data repositories such as data warehouses and pull data on-demand
mechanisms.
Data Analytic Layer
It has many functionalities such as statistical tests, machine learning algorithms to
understand, and construction of machine learning models. This layer implements many
model validation mechanisms too.
Presentation Layer
It has mechanisms such as dashboards, and applications that display the results of analytical
engines and machine learning algorithms. Thus, the Big Data processing cycle involves data
management that consists of the following steps.
1. Data collection
2. Data preprocessing
3. Applications of machine learning algorithm
4. Interpretation of results and visualization of machine learning algorithm
This is an iterative process and is carried out on a permanent basis to ensure that data is
suitable for data mining. Application and interpretation of machine learning algorithms
constitute the basis for the rest of the book. So, primarily, data collection and data
preprocessing are covered as part of this chapter. The following section covers data
collection in detail.
[Link] Collection
The first task of gathering datasets are the collection of data.
It is often estimated that most of the time is spent for collection of good quality data.
A good quality data yields a better result. It is often difficult to characterize a 'Good
data'.
'Good data' is one that has the following properties:
1. Timeliness -
The data should be relevant and not stale or obsolete data.
2. Relevancy -
The data should be relevant and ready for the machine learning or data
mining algorithms. All the necessary information should be available and there
should be no bias in the data.
3. Knowledge about the data -
The data should be understandable and interpretable, and should be self-
sufficient for the required application as desired by the domain knowledge
engineer.
[Link] Preprocessing
In real world, the available data is 'dirty'. By this word 'dirty', it means:
Incomplete data
Inaccurate data
Outlier data
Data with missing values
Data with inconsistent values
Duplicate data
1. Data preprocessing improves the quality of the data mining techniques.
2. The raw data must be preprocessed to give accurate results.
3. The process of detection and removal of errors in data is called data cleaning.
4. Data wrangling means making the data processable for machine learning algorithms.
5. Some of the data errors include human errors such as typographical errors or
incorrect measurement and structural errors like improper data formats.
6. Data errors can also arise from omission and duplication of attributes.
7. Noise is a random component and involves distortion of a value or introduction of
spurious objects. Often, the noise is used if the data is a spatial or temporal
component. Certain deterministic distortions in the form of a streak are known as
artifacts.
[Link] Visualization
To understand data, graph visualization is must. Data visualization helps to understand data.
It helps to present information and data to customers. Some of the graphs that are used in
univariate data analysis are bar charts, histograms, frequency polygons and pie charts. The
advantages of the graphs are presentation of data, summarization of data, description of
data, exploration of data, and to make comparisons of data. Let us consider some forms of
graphs now:
Bar Chart
A Bar chart (or Bar graph) is used to display the frequency distribution for variables. Bar
charts are used to illustrate discrete data. The charts can also help to explain the counts of
nominal data. It also helps in comparing the frequency of different groups.
[Link] Making
A framework is the architectural "pipeline" that allows data to flow from its source to a final
insight.
Data Acquisition & Ingestion: Collecting data from sensors, web logs, or APIs.
Data Preparation (ETL): Standing for Extract, Transform, and Load. This
involves cleaning the data, removing duplicates, and handling missing values.
Storage and Management: Using distributed systems like Hadoop (HDFS) or
cloud storage to hold massive amounts of data across multiple servers.
Computation Layer: The engine that processes the data, such as Apache Spark,
which allows for high-speed, parallel processing.
Analysis & Visualization: The final layer where algorithms are applied and results
are displayed in charts or dashboards for human interpretation.
◻ 4. Descriptive Statistics
Descriptive statistics provide a summary of the data so you can understand its general
"shape" without looking at every individual point.
Central Tendency
One cannot remember all the data. Therefore, a condensation or summary of the data is
necessary. This makes the data analysis easy and simple. One such summary is called central
tendency.
1. Mean
Arithmetic average (or mean) is a measure of central tendency that represents the
'center of the dataset. This is the commonest measure used in our daily conversation
such as average income or average traffic. It can be found by adding all the data and
dividing the sum by the number of observations. Mathematically, the average of all the
values in the sample (population) is denoted as x. Letx, x2..,x be a set of 'N' values or
observations, then the arithmetic mean is given as:
o Weighted mean – Unlike arithmetic mean that gives the weightage of all items
equally, weighted mean gives different importance to all items as the item
importance varies. Hence, different weightage can be given to items.
o Geometric mean - Letx,, x₂...,x be a set of 'N' values or observations.
Geometric mean is the Nth root of the product of N items. The formula for
computing geometric mean given as follows:
● Median
The middle value in the distribution is called median. If the total number of items in
the distribution is odd, then the middle value is called median. If the numbers are
even, then the average value of two items in the centre is the median. It can be
observed that the median is the value whereere x, is divided into two equal halves,
with half of the values being lower than the median and half higher than the median. A
median class is that class where (N/2)th item is present. In the continuous case, the
median is given by the formula:
● Mode - Mode is the value that occurs more frequently in the dataset. In other words,
the value that has the highest frequency is called mode. Mode is only for discrete data
and is not applicable for continuous data as there are no repeated values in continuous
data.
Measures of Shape
Skewness and Kurtosis (called moments) indicate the symmetry/asymmetry and peak
location of the dataset.
● Skewness The measures of direction and degree of symmetry are called measures
of third order. Ideally, skewness should be zero as in ideal normal distribution. More
often, the given dataset may not have perfect symmetry
Kurtosis
Kurtosis also indicates the peaks of data. If the data is high peak, then it indicates higher
kurtosis and vice versa. Kurtosis is the measure of whether the data is heavy tailed or light
tailed relative to normal distribution.
It can be observed that normal distribution has bell-shaped curve with no long tails. Low
kurtosis tends to have light tails. The implication is that there is no outlier data. Letx,Xy,XN
be a set of 'N' values or observations. Then, kurtosis is measured using the formula given
below:
◻ 5. Univariate Data Analysis
Univariate analysis is the study of a single variable. The goal is to describe the distribution
of that one variable across a population.
Frequency Distribution: Counting how often each value occurs.
Univariate Visualization Tools:
o Histograms: Groups continuous data into "bins" to show the frequency of
ranges (e.g., how many people are in the 20–30 age bracket).
o Box Plots: A "five-number summary" showing the minimum, first quartile,
median, third quartile, and maximum. It is the best tool for spotting
outliers.
o Bar Charts: Used for categorical (non-numerical) data, like counting the
number of "Red," "Blue," and "Green" cars.
o Density Plots: A smooth curve that represents the "shape" of the data
distribution, showing where
values are most concentrated
BIVARIATE DATA AND MULTIVARIATE DATA
Bivariate Data involves two variables. Bivariate data deals with causes of relationships.
Scatter plot is used to visualize bivariate data. It is useful to plot two variables with or without
nominal variables, to illustrate the trends, and also to show differences. It is a plot between
explanatory and response variables. It is a 2D graph showing the relationship between two
variables.
The scatter plot (Refer Figure 2.11) indicates strength, shape, direction and the presence of
Outliers. It is useful in exploratory data before calculating a correlation coefficient or fitting
regression curve.