Machine Learning (BAI602)

MODULE 1
INTRODUCTION:

1.1 NEED FOR MACHINE LEARNING

Business organizations use huge amount of data for their daily activities. They have now
started to use the latest technology, machine learning, to manage the data.
Machine learning has become so popular because of three reasons:
1. High volume of available data to manage: Big companies such as Facebook, Twitter, and
YouTube generate huge amount of data that grows at a phenomenal rate. It is estimated
that the data approximately gets doubled every year.
2. Second reason is that the cost of storage has reduced. The hardware cost has also
[Link], it is easier now to capture, process, store, distribute, and transmit the
digital information.
3. Third reason for popularity of machine learning is the availability of complex algorithms
now. Especially with the advent of deep learning, many algorithms are available for
machine learning.
let us establish these terms - data, information, knowledge, intelligence, and wisdom using a
knowledge pyramid as shown in Figure 1.1.

Figure 1.1: The Knowledge Pyramid

 All facts are data. Data can be numbers or text that can be processed by a computer. Today,
organizations are accumulating vast and growing amounts of data with data sources such
as flat files, databases, or data warehouses in different storage formats.
 Processed data is called information. This includes patterns, associations, or relationships
among data. For example, sales data can be analyzed to extract information like which is
the fast selling product.
 Condensed information is called knowledge. For example, the historical patterns and
future trends obtained in the above sales data can be called knowledge. Unless knowledge
is extracted, data is of no use. Similarly, knowledge is not useful unless it is put into
action.
 Intelligence is the applied knowledge for actions. An actionable form of knowledge is
called intelligence. Computer systems have been successful till this stage.
 The ultimate objective of knowledge pyramid is wisdom that represents the maturity of
mind that is, so far, exhibited only by humans.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
The objective of machine learning is to process these archival data for organizations to take
better decisions to design new products, improve the business processes, and to develop
effective decision support systems.
1.2. MACHINE LEARNING EXPLAINED
Machine Learning Definition:
 Machine learning is a sub-branch of Artificial Intelligence (AI).
 Arthur Samuel defined it as the field where computers learn without being explicitly
programmed.
Traditional Programming vs. Machine Learning:
 In traditional programming, explicit instructions (like algorithms or flowcharts) are
written to perform tasks.
 For complex tasks (e.g., puzzles, image recognition), manually programming these rules
can be difficult.
Expert Systems:
 Early AI used expert systems like MYCIN (for medical diagnosis), where human
knowledge was converted into rules for the system.
 These systems lacked real intelligence as they relied too much on human expertise.
Shift to Data-Driven Approach:
 AI then shifted to machine learning, which uses data to automatically learn models or
rules.
 This enables the system to predict outcomes for unknown data, without human input.
Human vs. Machine Learning:
 Humans make decisions based on experience, while machines use learned models based
on data patterns to make predictions or decisions.

Figure 1.2: (a) A Learning System for Humans (b) A Learning System for Machine Learning

Role of Data in Machine Learning

Data Quality and Learning: The quality of data directly affects the performance of a
learning system.
In statistical learning, the input (x) is mapped to an output (y) using a learning function, f.
Goal : Learn the function f from data to accurately predict the outcomes for new inputs.
Limitations of Heuristics: Heuristics are educated guesses and may fail occasionally. Course
corrections are made through evaluation and improved formulations.
Dr. MADHURA GANGAIAH BGSCET
Machine Learning (BAI602)
Concepts in Machine Learning:
Model in Statistical Learning: A model represents patterns in the data and can take different
forms:
 Mathematical equations
 Relational diagrams (e.g., trees/graphs)
 Logical rules (e.g., if/else)
 Groupings (e.g., clusters)
Pattern vs. Model: A pattern applies to certain attributes locally, while a model fits the entire
dataset [Link] example, a model can help decide if an email is spam or not, based on
patterns learned from data.
Tom Mitchell’s Definition of Machine Learning:A program learns from experience (E)
based on a task (T) and performance measure (P).
Example: For object detection (T), the system learns from a training dataset (E), and
performance is measured by accuracy (P).
Human vs. Machine Learning: Humans gain experience through observation, imitation, and
trial & error.
Machines learn by:
1. Data collection: Gathering data to build knowledge.
2. Abstraction: Creating concepts from the data, similar to how humans recognize objects
(e.g., an elephant).
3. Generalization: Using abstract concepts to form actionable knowledge (similar to forming
heuristics, or "rules of thumb").
4. Evaluation: Checking the effectiveness of the model, making adjustments (course
correction) when needed.

1.3 MACHINE LEARNING IN RELATION TO OTHER FIELDS

Machine Learning and AI: Machine Learning (ML) is a key subfield of Artificial
Intelligence (AI).AI’s goal is to develop intelligent agents (like robots, humans, or
autonomous systems).AI initially focused on replicating human-like intelligence through
logic and reasoning but faced setbacks, known as AI [Link] resurgence of AI was
driven by data-driven systems, with a focus on finding patterns in data.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
 Machine learning, a part of AI, specifically aims to extract patterns from data to make
predictions and decisions. It includes techniques like learning from examples and
reinforcement learning.
AI and Machine Learning Relationship: AI is broader and includes many areas, with
machine learning being one of its important subfields. In this relationship, machine learning
helps AI by providing the tools to learn from data and improve predictions over time.

Figure 1.3: Relationship of AI with Machine Learning

Deep learning is a sub branch of machine learning. In deep learning, the models are
constructed using neural network technology. Neural networks are based on the human
neuron models. Many neurons form a network connected with the activation functions
that trigger further neurons to perform tasks.
1.3.2 MACHINE LEARNING, DATA SCIENCE, DATA MINING AND DATA
ANALYTICS

Figure 1.4: Relationship of Machine Learning with Other Major Fields

1. Data Science Overview:

Data Science is an umbrella term that covers various fields, including machine learning.
Machine Learning is a branch of data science, and both are closely interlinked since
machine learning starts with data.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)

2. Big Data:
Big Data is a key area within data science and deals with large-scale data. It has three main
characteristics:
 Volume: Huge amounts of data, e.g., from platforms like Facebook and Twitter.
 Variety: Data comes in different formats like text, images, and videos.
 Velocity: The speed at which data is generated and processed.
Machine learning algorithms use big data for tasks like language translation and image
recognition, and it also drives fields like Deep Learning.
3. Data Mining:
Data Mining is about uncovering hidden patterns in data, similar to how mining finds
valuable resources. It is often seen as closely related to machine learning. However, data
mining focuses on extracting hidden patterns, while machine learning uses those patterns
for prediction.
4. Data Analytics: Data Analytics is another branch of data science that extracts useful
knowledge from raw data.
Predictive Analytics is used to make predictions, and machine learning plays a big role here,
as it shares many of the same algorithms.
5. Pattern Recognition: Pattern Recognition is an engineering field that uses machine
learning algorithms to identify and classify patterns. It can be seen as a specific application of
machine learning in areas like image recognition or speech processing.
The relationship between these fields is interconnected, as machine learning plays a central
role in many aspects of data science, from data mining to predictive analytics.
1.3.3 MACHINE LEARNING AND STATISTICS
1. Statistics Overview:
Statistics is a branch of mathematics focused on analyzing data and drawing conclusions
from [Link] looks for patterns in the data by setting a hypothesis and performing experiments
to validate [Link] methods are more theory-driven, requiring in-depth knowledge of
mathematical procedures and the guidance of [Link] models in statistics are often
complex equations and involve assumptions based on the type of data being analyzed.
Theoretical foundation: Statistics has a rigorous and coherent theoretical framework.
2. Machine Learning Overview: Machine Learning (ML) also learns from data, but it is
less focused on creating hypotheses upfront.
ML is more practical and data-driven, requiring fewer assumptions and less statistical
expertise.
It often involves using automated tools and algorithms to process data and learn patterns,
making it less reliant on deep theoretical knowledge compared to statistics.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
The Key Differences are :
 Assumptions: Statistics often requires many assumptions about the data, while machine
learning has fewer assumptions.
 Knowledge: Statistics requires strong mathematical and statistical knowledge, whereas
machine learning is more tool-driven and often more automated.
 Modeling Approach: Statistical models tend to be more theory-based with complex
equations, while machine learning models are often based on data patterns and may
require less prior theoretical structure.
Some argue that machine learning is an evolution of traditional statistics, applying
statistical principles in a more automated and flexible way to handle larger and more complex
datasets.
1.4 TYPES OF MACHINE LEARNING
What does the word 'learn' mean? Learning, like adaptation, occurs as the result of interaction
of the program with its environment. It can be compared with the interaction between a
teacher and a student.
There are four types of machine learning as shown in Figure

Figure 1.5: Types of Machine Learning

Before discussing the types of learning, it is necessary to discuss about data.

Labelled and Unlabeled Data: Data is a raw fact. Normally, data is represented in the form
of a table.

To illustrate labelled data, let us take one example dataset called Iris flower dataset or Fisher's
Iris dataset. The dataset has 50 samples of Iris - with four attributes, length and width of
sepals and petals.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
• The target variable is called class. There are three classes - Iris setosa, Iris virginica, and
Iris versicolor.
• The partial data of Iris dataset is shown in Table 1.1.
• Data with target attributes (labels) used for supervised learning.
• For example : Iris Dataset include features (eg: Sepal Length , Petal Length ) with a target
label (class of iris data)

Data without target attributes (labels) used for unsupervised [Link] Algorithm
identifies patterns or groups without explicit supervision.
1.4.1 Supervised Learning:
Uses labelled dataset. there is a supervisor or teacher component in supervised learning. A
supervisor provides labelled data so that the model is constructed and generates test data.
Supervised Learning has two methods: 1. Classification and 2. Regression
Two Key Methods:
[Link]: Predicts discrete labels/ classes. Eg: classifying images as cat or dog.
Algorithms:
• Decision trees
• Random Forest
• Support Vector Machine
• Naive bayes
• Artificial Neural networks (CNNs for Deep Learning)

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)

In classification, learning occurs in two stages. During the training stage, the algorithm learns
from a labelled dataset and builds a model. In the testing stage, the model is used to predict
the label of new, unseen data.
For example, given a test sample like (6.3, 2.9, 5.6, 1.8, ?), the model would classify it based
on what it has learned. Common applications include image recognition, disease
classification, and plant species identification.
Classification models can be categorized into generative models, which focus on data
generation (e.g., probabilistic models), and discriminative models, which focus on classifying
the data. Popular classification algorithms include Decision Trees, Random Forests, Support
Vector Machines (SVM), Naïve Bayes, and Artificial Neural Networks (ANNs), including
Deep Learning models like CNN.
2. Regression Models: Regression models, unlike classification algorithms, predict
continuous variables like price. In other words, it is a number. A fitted regression model is
shown in Figure 1.8 for a dataset that represent weeks input x and product sales y.

Figure 1.8: A Regression Model of the Form y = ax + b

In regression, the model takes an input x (independent variable) and generates a model in the
form of a fitted line, represented by the equation y = f(x), where y is the dependent variable.
For example, in linear regression, the model might predict product sales = 0.66 * Week +
0.54, where 0.66 and 0.54 are coefficients learned from the data. This allows for predictions,
such as estimating sales for an unknown week by substituting the week number into the
equation. Both regression and classification are supervised algorithms involving training and
testing. The key difference is that regression predicts continuous variables (e.g., product
price), while classification predicts discrete labels (e.g., categories or classes).
1.4.2 Unsupervised Learning
Unsupervised Learning is a type of learning where there is no supervisor or teacher guiding
the process. The algorithm learns by itself through trial and error, observing examples, and
identifying patterns.
In this method, the data does not come with labels, and the algorithm groups similar objects
together based on their attributes.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
A key example of unsupervised learning is Cluster Analysis and Dimensionality reduction
algorithms.
1. Cluster analysis
which aims to group objects into disjoint clusters. Objects within the same cluster are
similar in some way, while objects in different clusters are significantly different.
• Common applications of clustering include image segmentation, detecting abnormal
growth in medical images, and finding gene patterns in databases. For example, a
clustering algorithm can group images of dogs and cats into separate clusters, where the
samples within each cluster are similar to each other but different from samples in other
clusters.
• Some of the key clustering algorithms are: k-means algorithm, Hierarchical

algorithms

2. Dimensionality Reduction
Dimensionality reduction algorithms are examples of unsupervised algorithms. It takes
a higher dimension data as input and outputs the data in lower dimension by taking
advantage of the variance of the data. It is a task of reducing the dataset with few features
without losing the generality.

The differences between supervised and unsupervised learning are listed in the
following Table 1.2.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)

Table 1.2: Differences between Supervised and Unsupervised Learning

[Link]. Supervised Learning Unsupervised Learning
1. There is a supervisor No supervisor component
component
2. Uses Labelled data Uses Unlabeled data
3. Assigns categories or labels Performs grouping process such that similar
objects will be in one cluster

1.4.3 Semi-supervised Learning

Is used when there is a large amount of unlabeled data and a small amount of labeled data,
which can be expensive and time-consuming to obtain. In this approach, semi-supervised
algorithms assign pseudo-labels to the unlabeled data and then combine the labeled and
pseudo-labeled data to improve the learning process.
1.4.4 Reinforcement Learning mimics human learning by allowing an agent (which can be
a human, animal, robot, or program) to interact with an environment and receive rewards or
punishments based on its actions. The agent’s goal is to maximize its cumulative rewards.
This type of learning is experience-based, where the agent learns from trial and error rather
than having access to pre-labelled data. In this process, the agent explores different actions to
find the most rewarding path, much like solving a puzzle without knowing the answer
beforehand. It is goal-oriented and involves making sequential decisions to reach a final
outcome.

Agent: The decision-maker (e.g., a robot, software, or a program).

Environment: The space where the agent interacts.
Actions: Choices available to the agent (e.g., move left, right, up, down in a grid
game).
Reward: Feedback signal for actions (positive or negative).
Policy: The strategy the agent uses to determine the actions.
Goal: Achieve the highest cumulative reward over time.

Dr. MADHURA GANGAIAH BGSCET

 Machine Learning (BAI602)

1.5 CHALLENGES OF MACHINE LEARNING

Include several factors that impact its effectiveness and performance. Some key challenges
are:
i. Problems: Machine learning works best with well-posed problems where specifications
are clear and complete. Ill-posed problems, where the relationship between input and
output is unclear or ambiguous, are difficult to solve. For example, if multiple functions
can fit the same data (e.g., y = x, y = x², or y = x³), it becomes challenging to find the
correct model.
Input (x1, x2) a. Output (y)
1, 1 1
2, 1 2
3, 1 3
4, 1 4
5, 1 5
ii. Huge Data: Machine learning requires large amounts of quality data to build accurate
models. Gathering this data is often a challenge, as it needs to be both large in size and
free from issues like missing or incorrect data.
iii. High Computation Power: With big data comes the need for high computational
power. Machine learning algorithms often require powerful systems, such as those with
Graphics Processing Units (GPUs) or Tensor Processing Units (TPUs), to process
large datasets efficiently.
iv. Complexity of Algorithms: Choosing, designing, and applying the right algorithms is a
significant challenge. The complexity of selecting, describing, and evaluating algorithms
adds to the difficulty of machine learning tasks, requiring expertise from data scientists.
v. Bias/Variance Trade-off: The bias/variance trade-off represents a key challenge in
machine learning. Overfitting occurs when the model fits the training data well but
performs poorly on unseen data, indicating a lack of generalization. Underfitting, on the
other hand, happens when the model fails to fit both the training and test data well.
Balancing these two extremes is crucial for developing effective machine learning
models.
vi. Interpretability and Explainability: Many machine learning models, particularly deep
learning, are considered black boxes, making it difficult to explain their predictions or
decisions.
Solution: Use simpler, interpretable models where feasible.
Employ explainability tools like SHAP or LIME.
vii. Ethical Concerns and Bias in Data : Biases present in training data can lead to biased
models, perpetuating unfair decisions (e.g., in hiring, lending, or law enforcement).
Solution: Implement fairness-aware machine learning techniques, conduct audits, and use
diverse datasets.
viii. Scalability:
Scaling machine learning models to work with massive datasets across distributed
systems is a technical challenge.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
Solution: Use scalable tools and frameworks like Apache Spark, Dask, or cloud-based
machine learning services.

1.6 MACHINE LEARNING PROCESS

The CRISP-DM (Cross-Industry Standard Process for Data Mining) methodology provides a
structured framework for building data mining and machine learning solutions.
It is widely adopted because of its versatility across industries.
Below are the six steps of the CRISP-DM process:
1. Understanding the Business
Objective: Grasp the business goals and define the problem statement.
Activities:
Identify the organization's objectives and challenges. Formulate a clear problem statement for
the machine learning solution .Choose data mining or ML algorithm that aligns with business
goals.
2. Understanding the Data
Objective: Explore and understand the data available for solving the problem.
Activities: Collect and analyze data from different sources. Study the data characteristics like
trends, anomalies etc..Formulate hypotheses based on the data and match patterns to the
hypotheses.

[Link] of Data: This step focuses on cleaning and preparing the raw data for mining.
Handling missing data is crucial since it can lead to inaccurate results during both training
and testing phases. Proper strategies must be adopted to handle such issues effectively.

Dr. MADHURA GANGAIAH BGSCET

 Machine Learning (BAI602)
[Link]: Data mining algorithms are applied to the data to create models or identify
patterns. This step uses the cleaned and prepared data to generate a model for prediction or
analysis.
[Link]: The performance of the model is evaluated through statistical analysis and
visualization techniques. Accuracy is measured to assess the model’s effectiveness. For tasks
like email classification, domain knowledge is often required to evaluate the model's
performance correctly.
[Link]: Once the model is evaluated, it is deployed to improve existing processes or
address new situations. This step ensures that the findings from the data mining process are
put into practical use.
1.7 MACHINE LEARNING APPLICATIONS
Machine learning is widely applied across various domains, making its impact significant in
daily life. Here are some common Machine Learning Applications:
1. Sentiment Analysis: Used in Natural Language Processing (NLP), this technique
converts text into sentiment classifications such as happy, sad, or angry. For example, in
movie or product reviews, sentiment analysis helps automatically attach ratings like five
stars or one star.
2. Recommendation Systems: Personalized recommendations are made based on user
preferences. For instance, Amazon suggests books or products based on your browsing
history, while Netflix recommends movies and shows similar to those you’ve watched.
3. Voice Assistants: Virtual assistants like Amazon Alexa, Microsoft Cortana, Apple Siri,
and Google Assistant use machine learning to process speech commands and perform
various tasks, acting as interactive chatbots.
4. Navigation and Location Services: Tools like Google Maps and Uber leverage machine
learning algorithms to offer optimal routes and reduce travel time by locating the shortest
paths.
These applications demonstrate how machine learning is seamlessly integrated into daily
technology, providing smarter, more personalized services to users.
Table 1.4: Applications’ Survey Table
[Link]. Problem Domain Applications
1. Business Predicting the bankruptcy of a business firm
2. Banking Prediction of bank loan defaulters and detecting credit card frauds
3. Image Processing Image search engines, object identification, image classification,
and generating synthetic images
4. Audio/Voice Chatbots like Alexa, Microsoft Cortana. Developing chatbots
forcustomer support, speech to text, and text to voice
5. Telecommuni- cation Trend analysis and identification of bogus calls, fraudulent calls
and its callers, churn analysis
6. Marketing Retail sales analysis, market basket analysis, product performance
analysis, market segmentation analysis, and study of travel patterns
of customers for marketing tours
7. Games Game programs for Chess, GO, and Atari video games
8. Natural Language Translation Google Translate, Text summarization, and sentiment analysis
Dr. MADHURA GANGAIAH BGSCET
 Machine Learning (BAI602)
9. Web Analysis and Services Identification of access patterns, detection of e-mail spams, viruses,
personalized web services, search engines like Google, detection of
promotion of user websites, and finding loyalty of users after web
page layout modification
10. Medicine Prediction of diseases, given disease symptoms as cancer or
diabetes. Prediction of effectiveness of the treatment using patient
history and Chatbots to interact with patients like IBM Watson uses
machinelearning technologies.
11. Multimedia and Security Face recognition/identification, biometric projects like
identificationof a person from a large image or video database, and
applications involving multimedia retrieval
12. Scientific Domain Discovery of new galaxies, identification of groups of houses
basedon house type/geographical location, identification of
earthquake epicenters, and identification of similar land use

Key Terms:
 Machine Learning – A branch of AI that concerns about machines to learn automatically
withoutbeing explicitly programmed.
 Data – A raw fact.
 Model – An explicit description of patterns in a data.
 Experience – A collection of knowledge and heuristics in humans and historical training data in
case of machines.
 Predictive Modelling – A technique of developing models and making a prediction of unseen data.
 Deep Learning – A branch of machine learning that deals with constructing models using
neural networks.
 Data Science – A field of study that encompasses capturing of data to its analysis covering all
stagesof data management.
 Data Analytics – A field of study that deals with analysis of data. Big Data – A study of data that
has characteristics of volume, variety, and velocity.
 Statistics – A branch of mathematics that deals with learning from data using statistical methods.
 Hypothesis – An initial assumption of an experiment.
 Learning – Adapting to the environment that happens because of interaction of an agent with
the environment.
 Label – A target attribute.
 Labelled Data – A data that is associated with a label.
 Unlabelled Data – A data without labels.
 Supervised Learning – A type of machine learning that uses labelled data and learns with the
help of a supervisor or teacher component.
 Classification Program – A supervisory learning method that takes an unknown input and
assigns a label for it. In simple words, finds the category of class of the input attributes.
 Regression Analysis – A supervisory method that predicts the continuous variables based on the
input variables.
 Unsupervised Learning – A type of machine leaning that uses unlabelled data and groups the
attributes to clusters using a trial and error approach.
 Cluster Analysis – A type of unsupervised approach that groups the objects based on attributes
so that similar objects or data points form a cluster.
 Semi-supervised Learning – A type of machine learning that uses limited labelled and large
unlabelled data. It first labels unlabelled data using labelled data and combines it for learning
purposes.
 Reinforcement Learning – A type of machine learning that uses agents and environment
interactionfor creating labelled data for learning.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
 Well-posed Problem – A problem that has well-defined specifications. Otherwise, the problem is
called ill-posed.
 Bias/Variance – The inability of the machine learning algorithm to predict correctly due to lackof
generalization is called bias. Variance is the error of the model for training data. This leads to
problems called overfitting and underfitting.
 Model Deployment – A method of deploying machine learning algorithms to improve the
existing business processes for a new situation.

Understanding The Data

2.1 WHAT IS DATA?

Data encompasses all facts that are encoded in various forms like numbers, text, images,
audio, and video. It can either be directly interpretable by humans (e.g., numbers or text) or
require a computer for interpretation (e.g., images or videos).
Data can come from various sources such as flat files, databases, or data warehouses. It is
typically categorized into:
 Operational Data: Data encountered during normal business processes, like daily sales
records.
 Non-operational Data: Data used for decision-making purposes, such as market trends
or financial reports.
However, data in its raw form is meaningless. It requires processing and labeling to transform
it into information, which reveals patterns, relationships, or insights. For example, raw sales
data becomes valuable when it reveals which products had the highest sales in a given period.

Elements of Big Data

Big Data refers to data sets significantly larger than traditional "small data," characterized by
the following:
1. Volume: The sheer amount of data generated, typically measured in petabytes (PB) or
exabytes (EB), compared to small data's gigabytes (GB) or terabytes (TB).
2. Velocity: The speed at which data is generated and processed, such as data coming from
IoT devices and the internet. This influences how quickly big data can be accessed and
acted upon.
3. Variety: Big data comes in various forms, such as text, audio, video, images, and maps,
as well as composite data types like videos with integrated audio. It also varies by
function (e.g., transaction records, social media data) and source (e.g., open data, social
media).
4. Veracity: Refers to the quality and truthfulness of data. It accounts for potential errors,
such as technical, typographical, or human mistakes, affecting the reliability of data.
5. Validity: Ensures the data's accuracy and its suitability for decision-making or addressing
a given problem.
6. Value: Measures the usefulness of the information derived from the data, influencing
decisions and outcomes.
These six characteristics, known as the "6 Vs" of big data, help in understanding and
evaluating the complexity and quality of data. Additionally, data quality can be assessed
Dr. MADHURA GANGAIAH BGSCET
Machine Learning (BAI602)
through:
 Precision: How consistent measurements are, often measured by standard deviation.
 Bias: Systematic errors due to incorrect assumptions.
 Accuracy: Closeness of measurements to the true value, indicated by significant digits in
data.

2.1.1 TYPES OF DATA

In the context of Big Data, data can be categorized into three types: structured data,
unstructured data, and semi-structured data.
1. Structured Data
Structured data is organized in a well-defined manner, typically in tables within databases. It
can be easily retrieved and manipulated using tools like SQL. Common types of structured
data encountered in machine learning include:
 Record Data: This consists of a collection of measurements from a process. Each object
in a dataset has a set of measurements organized into rows and columns, where rows
represent objects (or entities) and columns represent attributes or features.
 Data Matrix: A variation of record data, this type involves numeric attributes. The data is
often represented as vectors or points in a multidimensional space, where each attribute is
a dimension describing the object.
 Graph Data: This represents relationships between objects. For example, a web page
referencing another web page can be modeled as a graph, with web pages as nodes and
hyperlinks as edges.
 Ordered Data: In this type, data objects have an implicit order. Examples include:
o Temporal Data: Attributes associated with time, such as customer purchasing
patterns during a specific season or time series data.
o Sequence Data: Data involving a sequence of items (like DNA data, which
consists of the sequence A, T, G, C).
o Spatial Data: Attributes related to position or area, such as location data found in
maps.
2. Unstructured Data
Unstructured data does not have a predefined structure and is typically in formats like
images, audio, video, and textual documents. It is estimated that around 80% of data is
unstructured. Examples include:
 Videos
 Images
 Audio files
 Textual documents
 Programs and blogs
3. Semi-Structured Data
Semi-structured data is partially organized, meaning it doesn't follow the strict structure of
databases but still contains some level of organization. Examples include:
 XML/JSON Data: Data formatted with tags or key-value pairs, allowing some

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
structure but not as rigid as relational databases.
 RSS Feeds: Data that includes metadata but is less strictly structured.
 Hierarchical Data: Data arranged in a tree-like structure with parent-child
relationships, like file directories or organizational charts.
Each type of data presents unique challenges and requires different techniques for analysis,
storage, and retrieval in machine learning and big data contexts.

2.1.2 DATA STORAGE AND REPRESENTATION

Once a dataset is assembled, it needs to be stored in a way that allows for efficient data
analysis. The goal of data storage management is to make data accessible and usable for
analysis. Various methods exist to organize and manage data storage, from flat files to
sophisticated data warehouses. Here are some common data storage structures:
1. Flat Files
Flat files are the simplest and most commonly available form of data storage. These files
store data in a plain text format (e.g., ASCII or EBCDIC) and are a cost-effective solution for
organizing data. However, flat files have limitations, particularly when the dataset grows
larger. They are suitable for small datasets but not ideal for larger datasets due to performance
issues.
Common flat file formats include:
 CSV Files (Comma-Separated Values): Data is stored with values separated by commas.
The first row typically contains attribute names, and subsequent rows contain the actual
data. These files are widely used and compatible with spreadsheet and database
applications.
 TSV Files (Tab-Separated Values): Similar to CSV, but with values separated by tabs
instead of commas. Both CSV and TSV files are widely supported and can be processed
using tools like Google Sheets or Microsoft Excel.
2. Database Systems
A database system consists of database files and a Database Management System (DBMS).
The DBMS helps manage data and improve query performance through tools like a database
administrator, query processor, and transaction manager. A relational database typically
consists of tables, with columns representing attributes and rows representing tuples
(records).
Different types of databases include:
1. Transactional Databases: These store records of transactions. Each transaction typically
contains a time stamp, identifier, and a set of items. Transaction databases are often used
for association analysis to uncover correlations among items.
2. Time-Series Databases: These databases store time-related data, such as log files or sales
data collected at regular intervals (e.g., hourly, daily, yearly).

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
3. Spatial Databases: These contain spatial data, which can be stored in either raster or
vector formats. Raster data are typically used for images (bitmap), while vector data store
information like maps using geometric primitives (points, lines, polygons).
[Link] Wide Web (WWW)
The World Wide Web serves as a vast online resource for data. Data mining algorithms aim to
extract interesting patterns and information from this unstructured data available across the
web.

[Link] (eXtensible Markup Language)

XML is a versatile data format that is both machine- and human-readable. It is commonly
used to represent data that needs to be exchanged across different platforms and systems,
providing flexibility for data sharing.
[Link] Stream
Data streams consist of dynamic data that flows continuously in and out of an environment.
Data streams are characterized by large volumes of data, real-time processing constraints, and
the fixed order in which the data arrives. This type of data is typically used in real-time
applications.
6. RSS (Really Simple Syndication)
RSS is a standard for sharing and distributing content feeds, allowing data to be syndicated
across various platforms for immediate updates.
[Link] (JavaScript Object Notation)
JSON is a lightweight and widely used data format for data interchange, especially for
machine learning algorithms. It is easy for both humans to read and machines to parse and
generate, making it popular for web and mobile applications.
Each of these storage methods plays a role in efficiently handling different types of data and
ensuring that the data is properly organized, accessible, and ready for analysis.
2.2 BIG DATA ANALYTICS AND TYPES OF ANALYTICS
The primary goal of data analysis is to help business organizations make informed decisions.
For instance, a company might want to know which product is the fastest-selling in order to
adjust its marketing strategies. Data analysis transforms raw data into useful insights to aid
decision-making.
The terms data analysis and data analytics are often used interchangeably, but there is a
subtle distinction:
 Data Analytics refers to the entire process, which includes data collection, preprocessing,
and analysis. It covers the entire cycle of data management.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
 Data Analysis, on the other hand, is just the analytical part of the process. It focuses on
analyzing historical data to extract insights.
Data analytics, in general, is more concerned with the future and uses historical data to help
predict future outcomes. There are four main types of data analytics:
1. Descriptive Analytics : Descriptive analytics is about summarizing the main features of
the data. After collecting data, it is examined and quantified to provide a clear picture of what
happened in the past. It is essentially a form of statistics that focuses on describing data rather
than drawing inferences or making predictions.
2. Diagnostic Analytics: Diagnostic analytics addresses the question "Why?" It is also
known as causal analysis. It aims to understand the causes behind specific outcomes or
events. For example, if a product is not selling well, diagnostic analytics tries to determine
why it is happening, examining potential reasons and their effects.
3. Predictive Analytics: Predictive analytics answers the question "What will happen in the
future?" This type of analytics uses historical data and applies algorithms to identify patterns
that can help predict future events or trends. Machine learning plays a key role in predictive
analytics, as it helps to identify these patterns for forecasting.
4. Prescriptive Analytics: Prescriptive analytics focuses on answering "What should we
do?" It goes beyond prediction and provides recommendations for the best course of action
based on analysis. This type of analytics helps organizations make decisions that maximize
outcomes while minimizing risks.
2.3 Big Data Analysis Framework
Many frameworks have been proposed for performing data analytics. Although each
framework may vary, most have some common factors. Big data frameworks are typically
layered architectures that offer flexibility and scalability. A typical 4-layer big data
architecture includes:
1. Data Connection Layer: This layer deals with data ingestion mechanisms and connectors.
Data ingestion refers to importing raw data into appropriate data structures for analysis. It
includes the ETL process (Extract, Transform, Load), which is responsible for extracting data
from sources, transforming it into a usable format, and loading it into a storage system.
2. Data Management Layer: The data management layer is responsible for preprocessing
data. Its main task is to manage the data effectively to allow parallel query execution, read
and write operations, and general data management tasks. Data may be handled in various
ways in this layer, such as through data-in-place mechanisms or by creating data repositories
like data warehouses.
3. Data Analytics Layer: This layer includes the statistical tests and machine learning
algorithms needed to analyze the data. It involves building models, applying algorithms, and
validating results to extract useful insights from the data.
4. Presentation Layer: The presentation layer focuses on visualizing the results of the
analytics process. It includes tools like dashboards and applications that display insights
generated by the analytical engines and machine learning algorithms. This layer enables
decision-makers to view and interpret the results clearly.
Big Data Processing Cycle
Dr. MADHURA GANGAIAH BGSCET
Machine Learning (BAI602)
The big data processing cycle consists of several iterative steps aimed at transforming raw
data into actionable insights:
1. Data Collection: Gathering datasets from various sources.
2. Data Preprocessing: Cleaning and transforming data to ensure quality and accuracy.
3. Application of Machine Learning Algorithms: Using algorithms to analyze and
model the data.
4. Interpretation and Visualization of Results: Presenting insights through
visualization tools.
This cycle is continuous to ensure that data remains accurate, relevant, and useful for
decision-making.
2.3.1 DATA COLLECTION
The first step in the data analytics process is data collection. Gathering good-quality data is
critical, as the quality of the data directly affects the quality of the insights generated. The
characteristics of "good data" include:
1. Timeliness: Data should be up-to-date and relevant to the current context.
2. Relevancy: The data should be directly applicable to the problem at hand and free
from bias.
3. Understandability(Knowledge about the data ): The data should be clear and
interpretable, with enough context to support meaningful analysis.
Data sources can be classified into three broad categories:
 Open/Public Data: Data that is freely available for use, typically without strict
copyright restrictions. Examples include government census data, scientific data, and
healthcare databases.
 Social Media Data: Data generated from social platforms like Twitter, Facebook, and
YouTube. This data is valuable for sentiment analysis and trend analysis.
 Multimodal Data: Data that includes multiple forms, such as text, video, and audio.
Examples include image archives, web data, and video feeds.
2.3.2 DATA PREPROCESSING
In the real world, raw data is often "dirty," meaning it contains errors or inconsistencies. The
process of cleaning and preparing data is called data preprocessing. This step is crucial for
ensuring that machine learning algorithms can work effectively.
Common issues with raw data include:
 Incomplete Data: Missing values or attributes.
 Inaccurate Data: Errors in measurement or data entry.
 Outliers: Extreme values that may not be representative of the overall dataset.
 Inconsistent Data: Values that don't align with expected formats or ranges.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
 Duplicate Data: Repeated records that can skew analysis.

The process of data cleaning involves detecting and removing errors, while data wrangling
refers to transforming data into a format that is suitable for analysis. Noise (random errors)
and artifacts (systematic distortions) must also be dealt with in this process to improve the
quality of data for machine learning algorithms.
a. It can be observed that data like Salary = ’ ’ is incomplete data.
b. The DoB of patients, John, Andre, and Raju, is the missing data.
c. The age of David is recorded as ‘5’ but his DoB indicates it is 10/10/1980. This is called
inconsistent data.

• Inconsistent data occurs due to problems in conversions, inconsistent formats, and

difference in units.
• Salary for John is -1500. It cannot be less than ‘0’. It is an instance of noisy data.
• Outliers are data that exhibit the characteristics that are different from other data and have
very unusual values. The age of Raju cannot be 136. It might be a typographical error. It
is often required to distinguish between noise and outlier [Link] may be legitimate
data and sometimes are of interest to the data mining algorithms.
• These errors often come during data collection stage. These must be removed so that
machine learning algorithms yield better results as the quality of results is determined by
the quality of input data. This removal process is called data cleaning.

Missing Data Analysis

One of the most critical tasks in data cleaning is missing data analysis. The objective is to
handle the missing data in such a way that it doesn't skew or negatively impact the
performance of machine learning models. Several methods are available to handle missing
data, each with its own advantages and disadvantages. Below are some of the common
strategies for addressing missing values:
1. Ignore the Tuple
 If a tuple (row) contains missing data, especially the class label, it can simply be ignored.
This method is most effective when only a small percentage of the data is missing.
However, if the missing values are significant, ignoring the rows may lead to biased
results.
2. Fill in the Values Manually
 In this method, domain experts examine the data and fill in the missing values manually.
While this approach can be accurate, it is time-consuming and may not be feasible for
larger datasets.
Dr. MADHURA GANGAIAH BGSCET
Machine Learning (BAI602)
3. Use a Global Constant
 Missing values can be filled with a predefined global constant, such as "Unknown" or
"Infinity." This approach can be quick but might lead to biased results since the value
inserted could be artificial and not representative of the missing data.
4. Fill with the Attribute's Own Value
 A more sophisticated approach is to replace missing values with a calculated attribute
value. For example, missing income values might be replaced with the average income
for that attribute. This is a common method in many machine learning techniques.
5. Use the Mean of the Same Class
 If the data belongs to a specific class, the missing attribute value may be replaced with the
mean of all samples that belong to the same class. This helps to preserve some degree of
consistency within the class.
6. Use the Most Probable Value
 Advanced methods, such as classification algorithms or decision tree prediction, can be
used to predict the missing values. The most probable value for the missing attribute can
be derived from the patterns of other similar instances.
Note: Some of these methods may introduce bias into the dataset, especially when the filled
values are not accurate but rather approximations. The error between the estimated value and
the true value is called bias. This bias must be considered when analyzing the effectiveness
of the missing data handling techniques.
Removal of Noisy or Outlier Data
Noise refers to random errors or variance in a dataset that can distort the analysis. Noisy data
may result from incorrect measurements, faulty instruments, or random fluctuations.
In machine learning, noise can degrade model accuracy, and thus it must be removed or
minimized.
Binning: One technique used to remove noisy data is binning. In binning, the dataset is
divided into multiple "bins" or groups, where each bin contains a set of data values that fall
within a certain range. By smoothing the data within the bins, noise can be reduced.
Some of the binning techniques include:
Smoothing by Means: The mean value of each bin is calculated and used to replace the
values within that bin. This helps to smooth out random variations.
2. Smoothing by Bin Medians: Instead of using the mean, the median of each bin is used to
replace the bin's values. This method is more robust to outliers than the mean.
3. Smoothing by Bin Boundaries: In this method, the values within a bin are replaced by the
closest bin boundary (the minimum or maximum value in the bin). This technique is useful
for preserving the data's distribution while reducing the impact of extreme values.
Example: Binning Techniques
Let’s consider a dataset S={12,14,19,22,24,26,28,31,34}S = \{12, 14, 19, 22, 24, 26, 28, 31,
34\} and apply different binning techniques:
Step 1: Equal-Frequency Bin Method

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
We divide the dataset into bins with an equal number of data points. If we choose bins of size
3, the data can be distributed as:
o Bin 1: 12, 14, 19
o Bin 2: 22, 24, 26
o Bin 3: 28, 31, 34
Step 2: Smoothing by Means
Replace the values in each bin with the mean of the values in that bin:
o Bin 1: 15, 15, 15 (mean of 12, 14, 19)
o Bin 2: 24, 24, 24 (mean of 22, 24, 26)
o Bin 3: 31.0, 31.0, 31.0 (mean of 28, 31, 34)
Step 3: Smoothing by Bin Boundaries
Replace the values in each bin with the closest bin boundary (minimum or maximum value in
the bin):
o Bin 1: 12, 12, 19 (minimum value is 12, maximum is 19)
o Bin 2: 22, 22, 26 (minimum value is 22, maximum is 26)
o Bin 3: 28, 32, 32 (minimum value is 28, maximum is 34)
As we can see, each binning method has its own approach to smoothing the data, and the
choice of method depends on the context of the data and the specific requirements of the
analysis.
These techniques help in reducing noise, which is important to improve the reliability and
accuracy of data analysis and machine learning models.
Data Integration and Data Transformations
Data Integration Data integration is the process of combining data from different sources
into a cohesive dataset for analysis. Often, data is collected from various databases,
spreadsheets, or external sources, and the goal is to combine these sources to create a unified
dataset. This process is important because data from multiple sources can have redundant
information, inconsistent formats, or missing data. Therefore, the integration routine must
focus on detecting and removing redundancies to ensure the accuracy and quality of the final
dataset.
Common challenges in data integration include:
 Schema matching: Ensuring that data from different sources aligns correctly.
 Data redundancy: Detecting and eliminating duplicated records.
 Inconsistency in formats: Ensuring that all data is represented in a consistent format.
 Data quality: Ensuring the accuracy and completeness of integrated data.
Data Transformation Once data is integrated, data transformation routines are employed
to modify the data so that it can be processed efficiently by data mining algorithms. Data
transformation typically involves operations like normalization, aggregation, or feature
selection. Transformation is crucial for improving the performance of machine learning and
data mining algorithms. A commonly used data transformation technique is normalization,
Dr. MADHURA GANGAIAH BGSCET
Machine Learning (BAI602)
which scales attribute values to fall within a specified range, often to ensure uniformity
across different data dimensions and improve algorithm performance.
Normalization Techniques
Normalization is the process of adjusting the values of numerical data to a common scale,
without distorting differences in the ranges of values. This is particularly useful when data
from different sources have different units or scales, which may make the data mining model
more sensitive to certain attributes. Here are two common techniques used for normalization:
1. Min-Max Normalization
Min-Max normalization rescales the data to a fixed range, usually between 0 and 1. It is
achieved by subtracting the minimum value of the attribute from each data value, and then
dividing by the range (the difference between the maximum and minimum values). This

ensures that the transformed data falls within the desired range.

Example:
Suppose we have the following dataset with the attribute "Age":
Age={15,25,35,45,55}
To apply Min-Max normalization:
 min(Age)=15
 max(Age)=55
The normalized value for an age of 25 would be: Normalized Age=25−1555−15=1040=0.25
So, 25 is normalized to 0.25 within the range [0, 1].
2. Z-Score Normalization (Standardization)
Z-Score normalization, also known as standardization, transforms the data to have a mean
of 0 and a standard deviation of 1. This method is particularly useful when the data follows a
Gaussian distribution or when you need to eliminate the influence of extreme values or
outliers.
The formula for Z-Score normalization is:

Example:
For the same "Age" dataset:
Age={15,25,35,45,55}
The mean μ\mu and standard deviation σ\sigma are calculated as:
μ=15+25+35+45+555=35 σ=(15−35)2+(25−35)2+(35−35)2+(45−35)2+(55−35)25=15.81
Now, the Z-Score normalized value for an age of 25 would be:
Dr. MADHURA GANGAIAH BGSCET
Machine Learning (BAI602)
Z-Score Normalized Age=25−3515.81=−1015.81≈−0.63 So, 25 is transformed into a Z-score
of approximately -0.63.
Summary of Normalization Techniques
 Min-Max Normalization:
o Rescales the data to a fixed range, typically [0, 1].
o Best suited for algorithms sensitive to the scale of the data (e.g., neural
networks).
 Z-Score Normalization:
o Transforms data to have a mean of 0 and a standard deviation of 1.
o Useful for algorithms that assume normally distributed data or when you want
to handle outliers.
Both methods have their applications depending on the data distribution and the specific
needs of the analysis. Min-Max is commonly used for neural networks and deep learning,
while Z-Score normalization is more appropriate when the data needs to be treated as
standardized with respect to the distribution.
Data Reduction
Data reduction is the process of reducing the size of the dataset while retaining the essential
information and characteristics. The purpose of data reduction is to improve the efficiency of
data mining algorithms and reduce the computational cost. By reducing the size of the data,
the time for processing, memory usage, and storage requirements can all be optimized.
However, it is important to ensure that the reduced data still produces accurate and reliable
results.
2.4 DESCRIPTIVE STATISTICS
Descriptive statistics involves the summarization and description of datasets. It focuses on
providing simple summaries and insights about the data, such as measures of central
tendency, dispersion, and distribution patterns. Descriptive statistics do not involve making
predictions or inferences about the population. It is a foundational aspect of data analysis, as
it helps to understand the underlying structure and characteristics of data before proceeding
with further analysis, like statistical modeling or machine learning.
Dataset and Data Types
A dataset refers to a collection of data objects (e.g., records, points, vectors, patterns) that
contain several attributes (or properties). These attributes are the characteristics or features of
the objects.
For example, in a medical database, each patient can be considered an object, and attributes
like age, blood pressure, height, and weight are examples of the patient's characteristics.
Each attribute should be assigned a value during the measurement process.
The type of attribute determines the data type, which can be categorized into different types
based on the characteristics of the data.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)

Fig 2.1 Types of Data

Types of Data
Broadly, data can be classified into two primary categories:
1. Categorical (Qualitative) Data
2. Numerical (Quantitative) Data
1. Categorical or Qualitative Data
Categorical data can be further divided into two types:
 Nominal Data:
o Nominal data consists of labels or categories that have no intrinsic order or
ranking. Operations like equality (e.g., checking if two data points are equal)
make sense for nominal data, but other operations such as averages do not.
o Example: Patient ID, Gender (Male, Female), Blood Type (A, B, AB, O).
 Ordinal Data:
o Ordinal data has a natural order or ranking, but the distances between values are
not necessarily consistent. This type of data provides more information than
nominal data.
o Example: Fever levels (Low, Medium, High). There is an inherent order here
(Low < Medium < High), but the difference between them is not necessarily
uniform or measurable.
2. Numerical or Quantitative Data
Numerical data can be divided into two types:
 Interval Data:
o Interval data is numeric data where the differences between values are
meaningful, but the data does not have a true zero point. This means that you can
compute differences between values (e.g., the difference between 30°C and 40°C)
but cannot compute ratios (e.g., 40°C is not "twice as hot" as 20°C).
o Example: Temperature in Celsius or Fahrenheit, dates.
 Ratio Data:
o Ratio data has both meaningful differences and ratios. It includes a true zero point,
meaning that you can say something is "twice as much" or "half as much" as
another value. This type of data allows for a broader range of mathematical
operations, including division and multiplication.
Dr. MADHURA GANGAIAH BGSCET
Machine Learning (BAI602)
o Example: Height, weight, income, age.
Further Classification of Data Types
Data can also be categorized based on the type of values it contains:
1. Discrete Data:
o Discrete data consists of distinct, separate values, typically represented as
integers. This data type often results from counting things.
o Example: Number of children in a family, number of cars in a parking lot.
2. Continuous Data:
o Continuous data can take any value within a given range and can include decimal
points. This data is often the result of measuring things and can be infinitely
precise, depending on the measurement tool's accuracy.
o Example: Height, weight, temperature, distance.
Classification Based on the Number of Variables
Data can also be classified based on the number of variables used to describe each data object
in the dataset:

Fig 2.2: Types of Data Based on Variables

1. Univariate Data:
o Univariate data refers to a dataset that contains only one variable. The analysis
focuses on summarizing and describing the distribution and characteristics of that
single variable.
o Example: A dataset containing the heights of individuals.
2. Bivariate Data:
o Bivariate data involves two variables, and the analysis explores the relationship or
correlation between them.
o Example: A dataset containing both the height and weight of individuals.
3. Multivariate Data:
o Multivariate data involves more than two variables. The analysis can reveal
complex relationships between multiple variables and how they influence each
other.
o Example: A dataset containing the height, weight, and age of individuals, where
relationships between all three variables are explored.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
2.5 UNIVARIATE DATA ANALYSIS AND VISUALIZATION
Univariate analysis is the simplest form of statistical analysis. As the name indicates, the
dataset has only one variable. A variable can be called as a category. Univariate does not deal
with cause or relationships. The aim of univariate analysis is to describe data and find
patterns. Univariate data description involves finding the frequency distributions, central
tendency measures, dispersion or variation, and shape of the data.
2.5.1 Data Visualization
Let us consider some forms of graphs Bar Chart A Bar chart (or Bar graph) is used to display
the frequency distribution for variables. Bar charts are used to illustrate discrete data. The
charts can also help to explain the counts of nominal data.
It also helps in comparing the frequency of different groups.
1. Bar Chart: Univariate analysis of categorical data. Used to illustrate discrete data.
Helps in Comparing the frequency of different groups.
Ex: The bar chart for students' marks {45, 60, 60, 80, 85} with Student ID = {1, 2, 3, 4, 5} is
shown below in Figure 2.3.

[Link] Chart :helps us to visualize the percentage of the data belonging to each category.
Pie Chart These are equally helpful in illustrating the univariate data.
Ex: The percentage frequency distribution of students' marks {22, 22, 40, 40, 70, 70, 70, 85,
90, 90} is below in Figure 2.4.

It can be observed that the number of students with 22 marks are 2. The total number of
students are 10. So, 2/10 × 100 = 20% space in a pie of 100% is allotted for marks 22 in
Figure 2.4.
[Link]: A histogram displays the frequency distribution of a continuous variable by
dividing the data into bins and showing how many values fall within each bin.
Dr. MADHURA GANGAIAH BGSCET
Machine Learning (BAI602)
Ex: The histogram for students’ marks {45, 60, 60, 80, 85} in the group range of 0-25, 26-50,
51-75, 76-100 is given below in Figure 2.5. One can visually inspect from Figure 2.5 that the
number of students in the range 76-100 is 2.

Histogram conveys useful information like nature of data and its mode. Mode indicates the
peak of dataset. In other words, histograms can be used as charts to show frequency,
skewness present in the data, and shape.

[Link] Plots :
A dot plot is a simple way to visualize the distribution of a small set of data points by plotting
individual dots along an axis. Similar to bar charts but less clustered as compared , as they
illustrate the bars only with single points. The dot plot of English marks for five students with
ID as {1, 2, 3, 4, 5} and marks {45, 60, 60, 80, 85} is given in Figure 2.6. The advantage is
that by visual inspection one can find out who got more marks.

2.5.2 DESCRIPTIVE STATISTICS MEASURES

Central Tendencies in Statistics are the numerical values that are used to represent mid-
value or central value a large collection of numerical data, gives a general idea of the whole
data set.
[Link] : Mean in general terms is used for the arithmetic mean of the data, but other than the
arithmetic mean there are geometric mean and harmonic mean as well that are calculated
using different formulas.
Mean for Ungrouped Data : Arithmetic mean (xˉ) is defined as the sum of the individual
observations (xi) divided by the total number of observations N

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)

Mean for Grouped Data : Mean (xˉ) is defined for the grouped data as the sum of the
product of observations (xi) and their corresponding frequencies (fi) divided by the sum of all
the frequencies (fi).

[Link]: Median of any distribution is that value that divides the distribution into two equal
parts such that the number of observations above it is equal to the number of observations
below it.

Median of Grouped Data: Where,

• l is the lower limit of median class,
• n is the total number of observations,
• cf is the cumulative frequency of the preceding class,
• f is the frequency of each class, and
• h is the class size.

[Link]
Mode is the value of that observation which has a maximum frequency corresponding to it. In
other, that observation of the data occurs the maximum number of times in a dataset.
2.5.3 DISPERSION
Refers to the spread or variability of a dataset around a central tendency (such as the mean,
median, or mode). In other words, dispersion measures how spread out or clustered the values
in a dataset are. High dispersion means the values are spread out widely, while low dispersion
indicates that the values are closer to the central point.
Dispersion is a crucial aspect of understanding the distribution of data. Several measures are
used to quantify dispersion, including range, variance, standard deviation, and standard
error. These are second-order measures, meaning they provide additional information
beyond the central tendency.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)

Common Measures of Dispersion

[Link]:The range is the simplest measure of dispersion, calculated as the difference
between the maximum and minimum values in the dataset.
Formula: Range=Maximum Value−Minimum Value
Example:If the dataset is {10,20,30}, the range is 30−10=20 =20.
If the dataset is {10,50,0}, the range is 50−0=50 =50.
[Link] Deviation:Standard deviation measures the average distance of each data point
from the mean of the dataset. It provides a better understanding of the spread of data than the
range, as it takes into account all the data points.
Two datasets with the same mean can have very different spreads. For example, the datasets
{10,20,30} and {10,50,0}both have the same mean of 20, but the spread of values in the
second dataset is much wider.

Where:
 xi = Each data point
 xˉ = Mean of the dataset
 n = Number of data points
 s = Sample standard deviation
Explanation:
 First, calculate the mean xˉ of the dataset.
 Then, for each data point, find the squared difference from the mean.
 Sum these squared differences and divide by the number of data points minus one (for
sample data, dividing by n−1 instead of n corrects for bias).
 Finally, take the square root of this value to obtain the standard deviation.
[Link] and Interquartile Range (IQR)
In statistics, quartiles are used to subdivide a dataset into four equal parts, which can help in
understanding the spread of the data. The quartiles are based on percentiles, which divide the
data into 100 equal parts. Specifically:

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
 The 1st quartile (Q1) is the 25th percentile (i.e., 25% of the data lies below this
value).
 The 2nd quartile (Q2) is the median or 50th percentile (i.e., 50% of the data lies
below this value).
 The 3rd quartile (Q3) is the 75th percentile (i.e., 75% of the data lies below this
value).
The Interquartile Range (IQR) measures the spread of the middle 50% of the data and is
defined as the difference between the third quartile (Q3) and the first quartile (Q1):
IQR=Q3−Q1
Outliers are often defined as values that fall outside the range:
Outliers>Q3+1.5×IQR or Outliers<Q1−1.5×IQR
This method identifies data points that are significantly higher or lower than the rest of the
data.
Example of IQR Calculation
Consider the dataset: {12, 14, 19, 22, 24, 26, 28, 31, 34\}{12,14,19,22,24,26,28,31,34}
Step 1: Find the median (Q2):
The median is the middle value when the data is sorted in ascending order. The sorted data is
already given as: {12, 14, 19, 22, 24, 26, 28, 31, 34}
The median (Q2) is the middle value, which is 24.
Step 2: Find the first quartile (Q1):
The first quartile is the median of the lower half of the data (below the median 24). The lower
half is: {12,14,19,22}
The median of this subset (Q1) is the average of the second and third values:
Q1=14+192=16.5
Step 3: Find the third quartile (Q3):
The third quartile is the median of the upper half of the data (above the median 24). The
upper half is:{26,28,31,34}
The median of this subset (Q3) is the average of the second and third values:
Q3=28+312=29.5
Step 4: Calculate the Interquartile Range (IQR):
IQR=Q3−Q1=29.5−16.5=13
Thus, the Interquartile Range (IQR) is 13.

2.5.5 Five-Point Summary and Box Plot

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)

The Five-Point Summary of a dataset is a concise summary that includes the following:
1. Minimum: The smallest value in the dataset.
2. Q1: The first quartile.
3. Median (Q2): The middle value of the dataset.
4. Q3: The third quartile.
5. Maximum: The largest value in the dataset.
[Link] of Five-Point Summary
For the dataset:{13,11,2,3,4,8,9}
Step 1: Sort the dataset in ascending order:{2,3,4,8,9,11,13}
Step 2: Calculate the five-point summary:
Minimum: 2
Q1: The first quartile is the median of the lower half of the data {2,3,4}\{2, 3, 4\}{2,3,4}, so
Q1 = 3.
Median (Q2): The middle value is 8.
Q3: The third quartile is the median of the upper half of the data {9,11,13}
Maximum: 13
Thus, the five-point summary is:{2,3,8,11,13}
[Link] of Five-Point Summary
Let us take a sample data to understand how to create a box plot,100, 120, 110, 150, 110, 140,
130, 170, 120, 220, 140, 110.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)

Any value below 57.5 or above 197.5 is an [Link] in this dataset: 220 (since it's
greater than 197.5).

Box Plot
A box plot (also known as a box-and-whisker plot) is a graphical representation of the five-
point summary, which helps in visualizing the distribution of the data, identifying outliers,
and understanding the spread of the data.
 The box represents the interquartile range (IQR) from Q1 to Q3.
 The line inside the box represents the median (Q2).
 The whiskers extend from the box to the minimum and maximum values.
 Outliers, if any, are shown as points beyond the whiskers.
In this case, for the dataset {2,3,4,8,9,11,13}, the box plot would show:
 A box from Q1 = 3 to Q3 = 11.
 A line at Median = 8 inside the box.
 The whiskers extend from 2 (minimum) to 13 (maximum).

2.5.4 Shape : Skewness and Kurtosis (called moments) indicate the symmetry/asymmetry
and peak location of the dataset.
Dr. MADHURA GANGAIAH BGSCET
Machine Learning (BAI602)
Skewness : The measures of direction and degree of symmetry are called measures of third
order. Ideally, skewness should be zero as in ideal normal distribution. More often, the given
dataset may not have perfect symmetry (consider the following Figure 2.8).

Skewness refers to the asymmetry or lack of symmetry in the distribution of data. It describes
the extent to which a distribution is stretched to the left or right. In a negatively skewed
distribution (also called left-skewed), the left tail is longer, and the mean is less than the
median. On the other hand, in a positively skewed distribution (or right-skewed), the right
tail is longer, and the mean is greater than the median.
Pearson's Skewness Coefficient
To quantify skewness, we often use Pearson's skewness coefficient. The formula for
skewness is given by:

For a negatively skewed distribution, the mean will be less than the median, while for a
positively skewed distribution, the mean will be greater than the median. If the skewness
value is close to zero, the distribution is approximately symmetric.
The skewness can also be calculated using the following formula:

Where:

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)
 X1,X2,X_N are the data values
 m is the population mean
 s is the population standard deviation
 N is the number of observations
This formula helps to measure the extent of skewness in the data, and bias correction is
applied by using N - 1 instead of N in certain cases.
Kurtosis
Kurtosis is a measure of the "tailedness" of the distribution. It describes the sharpness of the
peak and the thickness of the tails in the distribution. There are two main types of kurtosis:
 Leptokurtic: Distribution with a high peak and heavy tails (high kurtosis).
 Platykurtic: Distribution with a flat peak and light tails (low kurtosis).
Kurtosis is calculated using the formula:

Where:
 Xi is the individual data points
 m is the mean
 s is the standard deviation
 N is the number of data points
For normal distribution, the kurtosis is 3. If the kurtosis is greater than 3, it is considered
leptokurtic, indicating a distribution with heavier tails. If it is less than 3, it is platykurtic,
indicating lighter tails.
Mean Absolute Deviation (MAD)
MAD is a robust measure of the spread of data that is less sensitive to outliers. It is calculated
by taking the absolute deviations from the mean (or median in some cases). The formula for
MAD is:

Where:
 Xi are the data points
 Mean is the average of the data
 N is the number of observations
MAD provides an indication of the typical deviation from the central value of the data. It is a
good alternative to standard deviation, especially when dealing with data that may contain
extreme outliers.
Coefficient of Variation (CV)
Dr. MADHURA GANGAIAH BGSCET
Machine Learning (BAI602)
The Coefficient of Variation (CV) is a relative measure of the variability of the data. It is
useful for comparing datasets with different units or scales. It is calculated by dividing the
standard deviation by the mean, and then multiplying by 100 to express it as a percentage:

This measure helps to compare the degree of variation between different datasets, even if the
units of measurement are different. A higher CV indicates more variation relative to the
mean.
2.5.5 Special Univariate plots :
Stem-and-Leaf Plot
A stem-and-leaf plot is a graphical representation that helps visualize the shape and
distribution of a dataset. Each data point is split into two parts:
 The stem represents the leading digit(s) (e.g., tens, hundreds).
 The leaf represents the trailing digit (e.g., ones).
For example, consider the dataset: {45,60,60,80,85}
The stem-and-leaf plot would look like:
4|5
6|00
8|05
This plot gives a clear view of the shape of the dataset and allows easy identification of the
range and central values. It is particularly useful for small datasets.

Dr. MADHURA GANGAIAH BGSCET

Machine Learning (BAI602)

Dr. MADHURA GANGAIAH BGSCET