Introduction To Machine Learning

• Branch of AI that concerns machine to learn automatically without being explicitly

programmed. Machine Learning (ML) is a promising and flourishing field.

• It can enable top management of an organization to extract the knowledge from the data
stored in various archives of the business organizations to facilitate decision making.
Such decisions can be useful for organizations to

1. Design New Products

2. Improve Business Processes

3. To Develop Decision Support Systems

Need For Machine Learning

• Business organizations use huge amount of data for their daily activities. Earlier this
data was not utilized due to
1. Data being scattered across different archive systems and organizations not being
able to integrate these sources fully.
2. The lack of awareness about software tools that could help to unearth the useful
information from data.

• Machine learning has become popular because of three reasons:

1. High volume of available data to manage.
 Big companies such as Facebook, Twitter, and YouTube generate huge amount
of data that grows at a phenomenal rate.

2. The cost of storage has reduced.

 The hardware cost has dropped. Therefore, it is easier now to capture, process,
store, distribute, and transmit the digital information.
3. Availability of complex algorithms.

 With the advent of deep learning, many algorithms are available for machine
learning.

 A knowledge pyramid is shown below

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
  Data can be numbers or text that can be processed by a computer.
 Data sources such as flat files, databases, or data warehouses in different storage
formats.

 Processed data is called information.

 This includes patterns, associations, or relationships among data.

 Example: Sales data can be analyzed to extract information like which is the fast-
selling product.

 Condensed information is called knowledge.

 Example: The historical patterns and future trends obtained in the above sales data
can be called knowledge.

 Applied knowledge for actions is called Intelligence.

 An actionable form of knowledge is called intelligence.

 The ultimate objective of knowledge pyramid is wisdom that represents the maturity of
mind that is, so far, exhibited only by humans.

 The objective of machine learning is

1. To process archival data for organizations to take better decisions to design new
products

2. Improve the business processes

3. To develop effective decision support systems

Machine Learning Explained

 Machine learning is a sub-branch of Artificial Intelligence (AI).

 Definition of machine learning by Arthur Samuel

“Machine learning is the field of study that gives the computers ability to learn
without being explicitly programmed.”
The key to this definition is that the systems should learn by itself without explicit
programming.

In conventional programming

 After understanding the problem, a detailed design of the program such as a flowchart
or an algorithm needs to be created and converted into programs using a suitable
programming language.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
  This approach was impractical in many domains, as programs still depended on human
expertise and hence did not truly exhibit intelligence.
Machine learning is a form of data driven systems

 The focus of AI is to develop intelligent systems by using data-driven approach, where

data is used as an input to develop intelligent models. The models can then be used to
predict new inputs.

 The aim of machine learning is to learn a model or set of rules from the given dataset
automatically so that it can predict the unknown data correctly. Humans take decisions
based on an experience.

 Computers make models based on extracted patterns in the input data and then use these
data-filled models for prediction and to take decisions.

 In statistical learning relationship between input x and output y is modeled as a function

in the form y = f(x)

 Here,
 f is the learning function that maps the input x to output y.
 Learning of function f is the crucial aspect of forming a model in statistical learning.

 In machine learning, this is simply called mapping of input to output.

 The learning program summarizes the raw data in a model.

 A model is an explicit description of patterns within the data in the form of:
1. Mathematical equation

2. Relational diagrams like trees/graphs

3. Logical if/else rules
4. Groupings called clusters

 A model can be a formula, procedure or representation that can generate data decisions.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
  The difference between pattern and model is that the former is local and applicable only
to certain attributes but the latter is global and fits the entire dataset.

 Example
 A model can be helpful to examine whether a given email is spam or not.

 Another pioneer of AI, Tom Mitchell's definition of machine learning states that,

"A computer program is said to learn from experience E, with respect to task T
and some performance measure P, if its performance on T measured by P
improves with experience E."

The important components of this definition are experience E, task T, and performance
measure P.
Example: The task T could be detecting an object in an image. The machine can gain
the knowledge of object using training dataset of thousands of images. This is called
experience E. So, the focus is to use this experience E for this task of object detection
T. The ability of the system to detect the object is measured by performance measures
like precision and recall. Based on the performance measures, course correction can be
done to improve the performance of the system.

 In systems, experience is gathered by these steps:

1. Collection of data

2. Once data is gathered, abstract concepts are formed out of that data.
 Abstraction is used to generate concepts. This is equivalent to humans' idea of
objects, Example - we have some idea about how an elephant looks like.

3. Generalization converts the abstraction into an actionable form of intelligence.

 It can be viewed as ordering of all possible concepts.
 Generalization involves ranking of concepts, inferencing from them and formation
of heuristics, an actionable aspect of intelligence.

 Heuristics are educated guesses for all tasks.

 Example, if one runs or encounters a danger, it is the resultant of human experience
or his heuristics formation. In machines, it happens the same way.

4. The course correction is done by taking evaluation measures.

 Evaluation checks the thoroughness of the models and to-do course correction, if
necessary, to generate better formulations.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
Machine Learning in Relation to Other Fields
Machine learning uses the concepts of
1. Artificial Intelligence

2. Data Science
3. Statistics primarily

1. Machine Learning and Artificial Intelligence

 Machine learning is an important branch (subbranch) of AI.

 The aim of AI is to develop intelligent agents.

 An agent can be a robot, humans, or any autonomous systems.

 The aim of Machine Learning is to extract the patterns for prediction.

 It is a broad field that includes learning from examples and other areas like
reinforcement learning.

 The relationship of AI and machine learning is shown in below figure

 Deep learning is a subbranch of machine learning. In deep learning, the models are
constructed using neural network technology.

 Neural networks are based on the human neuron models. Many neurons form a network
connected with the activation functions that trigger further neurons to perform tasks.

2. Machine Learning, Data Science, Data Mining, and Data Analytics

• Data science is an 'Umbrella' term that encompasses many fields.

• Machine learning starts with data and is a branch of data science.

• Data science deals with gathering of data for analysis.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • It is a broad field that includes:

1. Big Data
2. Data Mining

3. Data Analytics
4. Pattern Recognition

1. Big Data
• Data science concerns about collection of data.
• Big data is a field of data science that deals with data's following characteristics:

1. Volume: Huge amount of data is generated by big companies like Facebook,

YouTube, etc
2. Variety: Data is available in variety of forms like images, videos, and in different
formats.

3. Velocity: It refers to the speed at which the data is generated and processed.

• Big data is used by many machine learning algorithms for applications such as language
translation and image recognition and influences the growth of subjects like Deep
learning.

2. Data Mining
• Data mining's origin is in the business.

• It aims to extract the hidden patterns that are present in the data.

3. Data Analytics
• Branch of data science. It aims to extract useful knowledge from crude data.
• There are different types of analytics.

1. Predictive data analytics is used for making predictions.

2. Machine learning is closely related to this branch of analytics and shares almost all
algorithms.

4. Pattern Recognition
• It uses machine learning algorithms to extract the features for pattern analysis and
pattern classification. One can view pattern recognition as a specific application of
machine learning.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
3. Machine Learning and Statistics
Statistics:
 Focuses on hypotheses, experiments, and rigorous mathematical models to find
relationships in data.
 Requires a strong theoretical background and often involves complex equations and
assumptions.

Machine Learning:
 Less assumption-heavy compared to statistics and automates the learning process.
 Often viewed as an evolved version of "old statistics."
Difference:

 Statistics aims to validate hypotheses, while machine learning focuses on predictions

and automation.

Types of Machine Learning

• Machine learning involves learning through the interaction of a program with its
environment.
• The types of learning are categorized based on the data used and the supervision
provided during the learning process.

Labelled Data:
 Data with target attributes (labels) is used for supervised learning.

 Example 1: Data in the form of images or video frame

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
  Example 2: The Iris dataset includes features (e.g., sepal length, petal width) with a
target label (class of Iris flower).

Unlabelled Data:

• Data without target attributes is used for unsupervised learning.

• The algorithm identifies patterns or groups without explicit supervision.

• There are four types of machine learning.

• The types of learning are categorized based on the data used and the supervision
provided during the learning process.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
1. Supervised Learning

• Involves a teacher or supervisor providing labelled data for training and testing.

• In supervised learning algorithms, learning takes place in two stages.

1. First stage,
 The teacher communicates the information to the student that the student is
supposed to master. The student receives the information and understands it.
 During this stage, the teacher has no knowledge of whether the information is
grasped by the student.
2. Second stage,
 The teacher then asks the student a set of questions to find out how much
information has been grasped by the student.
 Based on these questions, the student is tested, and the teacher informs the student
about his assessment.

• This kind of learning is typically called supervised learning.

• Supervised learning has two methods:

1. Classification
2. Regression

1. Classification

• Classification is a supervised learning method. The input attributes of the classification

algorithms are called independent variables. The target attribute is called label or
dependent variable.
• The relationship between the input and target variable is represented in the form of a
structure which is called a classification model.

• The focus of classification is to predict the 'label' that is in a discrete form (a value from
the set of finite values).

• Example: A Classification algorithm takes a set of labelled data images such as dogs
and cats to construct a model that can later be used to classify an unknown test image
data.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • In classification, learning takes place in two stages.

1. First stage, called training stage, the learning algorithm takes a labelled dataset and
starts learning. After the training set, samples are processed and the model is
generated.
2. Second stage, the constructed model is tested with test or unknown sample and
assigned a label. This is the classification process.

• Classification models can be categorized

1. Generative Models: Focus on data generation (e.g., probabilistic models).

2. Discriminative Models: Focus on classification only.

• Key Algorithms of classification

a) Decision Trees
b) Random Forest

c) Support Vector Machines (SVM)

d) Naïve Bayes
e) Artificial Neural Networks (e.g., CNNs for deep learning)

2. Regression
• Regression predicts continuous variables, such as prices or sales.

• Example: Predicting product sales over weeks using linear regression.

• Algorithm:

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
  Linear Regression: Fits a line (e.g., y=0.66x+0.54y = 0.66x + 0.54y=0.66x+0.54)
to predict outcomes based on independent variables.

2. Unsupervised Learning
• Self-instruction-based learning with no supervisor or labelled data.

• Algorithms observe examples, recognize patterns, and group similar objects.

• Example:
a) Cluster Analysis algorithm

b) Dimensional reduction algorithm

Cluster Analysis:
• It aims to group objects into disjoint clusters or groups.
• Cluster analysis clusters objects based on its attributes.

 All the data objects of the partitions are similar in some aspect and vary from the
data objects in the other partitions significantly.

Examples
a) Segmentation of a region of interest in an image

b) Detection of abnormal growth in a medical image

c) Determining clusters of signatures in a database.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • Algorithms:

1. k-means algorithm

2. Hierarchical algorithms
Dimensionality Reduction:

• Dimensionality reduction is a type of unsupervised learning where high-dimensional

data is transformed into a lower-dimensional space while retaining the critical variance
or patterns in the dataset.

• The goal is to simplify the dataset by reducing the number of features without losing its
essential information.

Differences between Supervised and Unsupervised Learning

3. Semi-Supervised Learning
• Semi-supervised learning operates in scenarios where a large portion of the dataset is
unlabeled, and only a small portion is labelled.

• Since labelling data is expensive and time-consuming, semi-supervised algorithms

make use of unlabeled data by assigning pseudo-labels (predicted labels).

4. Reinforcement Learning
• Reinforcement learning is inspired by how humans learn through interaction and
feedback.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • An agent interacts with the environment, perceives its state, takes actions, and receives
rewards or penalties. The agent aims to maximize cumulative rewards by learning the
best actions to take over time.
• Key Concepts:

 Agent: The decision-maker (e.g., a robot, software, or a program).

 Environment: The space where the agent interacts.

 Actions: Choices available to the agent (e.g., move left, right, up, down in a grid
game).

 Reward: Feedback signal for actions (positive or negative).

 Policy: The strategy the agent uses to determine actions.

 Goal: Achieve the highest cumulative reward over time.

• Example - Grid Game

• In the grid game:

 Gray tiles represent danger.

 Black tiles are blocks.

 Diagonal-striped tile is the goal.

• The agent learns to navigate from the bottom-left grid to the goal by exploring various
paths and gaining rewards for successful actions or penalties for poor decisions.

Challenges of Machine Learning

1. Problems
• Machine learning can deal with the 'well-posed' problems where specifications are
complete and available. Computers cannot solve 'ill-posed' problems.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 2. Huge data
• This is a primary requirement of machine learning.

• A quality data means it should be large and should not have data problems such as
missing data or incorrect data.

3. High computation power

• Systems with Graphics Processing Unit (GPU) or even Tensor Processing Unit
(TPU) are required to execute machine learning algorithms.

• Machine learning tasks have become complex and hence time complexity has
increased, and that can be solved only with high computing power.

4. Complexity of the algorithms

• Algorithms have become a challenge for machine learning professionals to design,
select, and evaluate optimal algorithms.

• The selection of algorithms, describing the algorithms, application of algorithms to

solve machine learning task, and comparison of algorithms.

5. Bias/Variance
• Variance is the error of the model. This leads to a problem called bias/ variance
tradeoff. Overfitting and underfitting are great challenges for machine learning
algorithms.

• A model that fits the training data correctly but fails for test data, in general lacks
generalization, is called overfitting.

• The reverse problem is called underfitting where the model fails for training data
but has good generalization.

Machine Learning Process

• The emerging process model for the data mining solutions for business organizations
is CRISP-DM.
• CRISP-DM stands for Cross Industry Standard Process - Data Mining.

• This process involves six steps.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 1. Understanding the business
• This step involves understanding the objectives and requirements of the business
organization.

• This step also involves the formulation of the problem statement for the data mining
process.

2. Understanding the data

• It involves the steps like data collection, study of the characteristics of the data,
formulation of hypothesis, and matching of patterns to the selected hypothesis.

3. Preparation of data
• This step involves producing the final dataset by cleaning the raw data and
preparation of data for the data mining process.

• The missing values may cause problems during both training and testing phases.

• Missing data forces classifiers to produce inaccurate results. Hence, suitable

strategies should be adopted to handle the missing data.

4. Modelling
• This step plays a role in the application of data mining algorithm for the data to
obtain a model or pattern.

5. Evaluate
• This step involves the evaluation of the data mining results using statistical analysis
and visualization methods.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • The performance of the classifier is determined by evaluating the accuracy of the
classifier.
 Example - Classification of emails requires extensive domain knowledge and
requires domain experts. Hence, performance of the classifier is very crucial.

6. Deployment
• This step involves the deployment of results of the data mining algorithm to
improve the existing process or for a new situation.

Machine Learning Applications

1. Sentiment analysis

 This is an application of natural language processing (NLP) where the words of

documents are converted to sentiments like happy, sad, and angry which are
captured by emotions effectively.

 Example: For movie reviews or product reviews, five stars or one star are
automatically attached using sentiment analysis programs.

2. Recommendation systems
 These are systems that make personalized purchases possible.
 Example: Amazon recommends users to find related books or books bought by
people who have the same taste like you, and Netflix suggests shows or related
movies of your taste. The recommendation systems are based on machine learning.

3. Voice assistants
 Products like Amazon Alexa, Microsoft Cortana, Apple Siri, and Google Assistant
are all examples of voice assistants.
 They take speech commands and perform tasks. These chatbots are the result of
machine learning technologies.
4. Technologies like Google Maps and those used by Uber are all examples of machine
learning which offer to locate and navigate shortest paths to reduce time.

• Following table summarize some of the machine learning applications

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
Understanding Data
Data
• Data is available in different data sources like flat files, databases, or data warehouses.
• Data can either be an operational data or a non-operational data.

 Operational data is the one that is encountered in normal business procedures and
processes. Example - Daily sales data is operational data.

 Non-operational data is the kind of data that is used for decision making.

• Processed data is called information that includes patterns, associations, or relationships

among data.
• Example - Sales data can be analyzed to extract information like which product was
sold larger in the last quarter of the year.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
Elements of Big Data or 6V’s of Big Data
• Big data is a larger data whose volume is much larger than 'small data' and is
characterized as follows:
1. Volume
 Big Data is measured in terms of petabytes (PB) and exabytes (EB). One
exabyte is 1 million terabytes.
2. Velocity
 The fast arrival speed of data and its increase in data volume is noted as
velocity.
 Velocity helps to understand the relative growth of big data and its accessibility
by users, systems and applications.
3. Variety
 The variety of Big Data includes:
a) Form
 There are many forms of data. Data types range from text, graph, audio,
video, to maps.
 There can be composite data too, where one media can have many other
sources of data. example, a video can have an audio song.

b) Function
 These are data from various sources like human conversations, transaction
records, and old archive data.

c) Source of data
 The data source can be classified as open/public data, social media data and
multimodal data.

4. Veracity of data
 Veracity of data deals with aspects like conformity to the facts, truth- fulness,
believability, and confidence in data.
 There may be many sources of error such as technical errors, typographical
errors, and human errors. So, veracity is one of the most important aspects of
data.
5. Validity
 Validity is the accuracy of the data for taking decisions that are needed by the
given problem.
6. Value
 Value indicates the value of the information that is extracted from the data and
its influence on the decisions that are taken based on it.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
Types of Data
1. Structured Data

2. Unstructured Data

3. Semi-Structured Data

1. Structured Data
• In structured data, data is stored in an organized manner such as a database where it is
available in the form of a table.

• The data can also be retrieved in an organized manner using tools like SQL.
• The structured data in machine learning are listed below

Record Data
• A dataset is a collection of measurements taken from a process.

• We have a collection of objects in a dataset and each object has a set of measurements.
• The measurements can be arranged in the form of a matrix.

 Rows in the matrix represent an object and can be called as entities, cases, or
records.

 The columns of the dataset are called attributes, features, or fields.

 The table is filled with observed data.

 Label is the term that is used to describe the individual observations.

Data Matrix
• The data is thought of as points or vectors in the multidimensional space where every
attribute is a dimension describing the object.

Graph Data
• It involves the relationships among objects.

• Example - a web page can refer to another web page. The modes are web pages and the
hyperlink is an edge that connects the nodes.

Ordered Data
• It involves attributes that have an implicit order among them.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 The examples of ordered data are:
1. Temporal data - It is the data whose attributes are associated with time. For example,
the customer purchasing patterns during festival time is sequential data. Time series
data is a special type of sequence data where the data is a series of measurements over
time.

2. Sequence data - It is like sequential data but does not have time stamps. This data
involves the sequence of words or letters. For example, DNA data is a sequence of four
characters -ATGC.
3. Spatial data - It has attributes such as positions or areas. For example, maps are spatial
data where the points are related by location.

2 Unstructured Data
• Unstructured data includes video, image, and audio.

• It also includes textual documents, programs, and blog data.

3 Semi-Structured Data
• Semi-structured data are partially structured and partially unstructured.
• These include data like XML/JSON data, RSS feeds, and hierarchical data.

Data Storage and Representation

• The goal of data storage management is to make data available for analysis.

• There are different approaches to organize and manage data

1. Flat Files
• These are the simplest and most commonly available data source.

• These flat files are the files where data is stored in plain ASCII or EBCDIC format. Flat
file is suitable only for storing small dataset.

• Some of the popular spreadsheet formats are listed below:

1. CSV files - CSV stands for comma-separated value files where the values are
separated by commas. These are used by spreadsheet and database applications. The
first row may have attributes and the rest of the rows represent the data.

2. TSV files - TSV stands for Tab separated values files where values are separated
by Tab.

2. Database System
• Database files contain original data and metadata.
2
0
Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • DBMS aims to manage data and improve operator performance by including various
tools like database administrator, query processing, and transaction manager.

• A user can access and manipulate the data in the database using SQL.
• Different types of databases are listed below:
1. A transactional database is a collection of transactional records. Each record is a
transaction. A transaction may have a time stamp, identifier and a set of items,
which may have links to other tables.

2. Time-series database stores time related information like log files where data is
associated with a time stamp.

3. Spatial databases contain spatial information in a raster or vector format.

 Raster formats are either bitmaps or pixel maps. For example, images can be
stored as a raster data.
 Vector format can be used to store maps as maps use basic geometric primitives
like points, lines, polygons and so forth.

3. World Wide Web (WWW)

• It provides a diverse, worldwide online information source. The objective of data

mining algorithms is to mine interesting patterns of information present in WWW.

4. XML (eXtensible Markup Language)

• It is both human and machine interpretable data format that can be used to represent
data that needs to be shared across the platforms.

5. Data Stream

• It is dynamic data, which flows in and out of the observing environment. Typical
characteristics of data stream are huge volume of data, dynamic, fixed order movement,
and real-time constraints.

6. RSS (Really Simple Syndication)

• It is a format for sharing instant feeds across services.

7 JSON (JavaScript Object Notation)

• It is another useful data interchange format that is often used for many machine learning
algorithms.

Big Data Analytics and Types of Analytics

• The primary aim of data analysis is to assist business organizations to take decisions.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • Example, a business organization may want to know which is the fastest selling product,
in order for them to market activities.

• Data analysis is an activity that takes the data and generates useful information and
insights for assisting the organizations.
 Data analytics refers to the process of data collection, preprocessing and analysis.
It deals with the complete cycle of data management.
 Data analysis is just analysis and is a part of data analytics. It takes historical data
and does the analysis. Data analytics, instead, concentrates more on future and helps
in prediction.

• There are four types of data analytics:

1. Descriptive analytics
2. Diagnostic analytics

3. Predictive analytics
4. Prescriptive analytics

1. Descriptive Analytics
• Descriptive analytics deals with the collected data and quantifies it.
 It is often stated that analytics is essentially statistics.

 There are two aspects of statistics - Descriptive and Inference.

 Descriptive analytics only focuses on the description part of the data and not the
inference part.
2. Diagnostic Analytics
• It aims to find out the cause and effect of the events.

• Example, if a product is not selling diagnostic analytics aims to find out the reason.
3. Predictive Analytics
• This involves the application of algorithms to identify the patterns to predict the future.

• The entire course of machine learning is mostly about predictive analytics and forms
the core of this book.
4. Prescriptive Analytics

• It is about the finding the best course of action for the business organizations.

• Prescriptive analytics goes beyond prediction and helps in decision making by giving
a set of actions.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • It helps the organizations to plan better for the future and to mitigate the risks that are
involved.

Big Data Analysis Framework

• Big data framework is a layered architecture.

• A 4-layer architecture has the following layers

1. Data connection layer
2. Data management layer
3. Data analytics later
4. Presentation layer
1 Data Connection Layer
• It has data ingestion mechanisms and data connectors. Data ingestion means taking raw
data and importing it into appropriate data structures. It performs the tasks of ETL
process. By ETL, it means extract, transform and load operations.

2 Data Management Layer

• It performs preprocessing of data. The purpose of this layer is to allow parallel
execution of queries, and read, write and data management tasks.

3 Data Analytic Layer

• It has many functionalities such as statistical tests, machine learning algorithms to
understand, and construction of machine learning models. This layer implements many
model validation mechanisms too.

4 Presentation Layer
• It has mechanisms such as dashboards, and applications that display the results of
analytical engines and machine learning algorithms.
• The Big Data processing cycle involves data management that consists of the following
steps.

1. Data collection
2. Data preprocessing
3. Applications of machine learning algorithm
4. Interpretation of results and visualization of machine learning algorithm
• This is an iterative process and is carried out on a permanent basis to ensure that data
is suitable for data mining.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 Data Collection
• The first task of gathering datasets is the collection of data. Time is spent for collection
of good quality data. 'Good data' is one that has the following properties:

1. Timeliness -The data should be relevant and not stale or obsolete data.
2. Relevancy - The data should be relevant and ready for the machine learning or data
mining algorithms. All the necessary information should be available and there
should be no bias in the data.

3. Knowledge about the data - The data should be understandable and interpretable,
and should be self-sufficient for the required application as desired by the domain
knowledge engineer.

• The data source can be classified as

1. Open/public data
2. Social media data

3. Multimodal data.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
1. Open or public data source

• It is a data source that does not have any stringent copyright rules or restrictions.
• Its data can be primarily used for many purposes.

 Government census data are good examples of open data:

 Digital libraries that have huge amount of text data as well as document images
 Scientific domains with a huge collection of experimental data like genomic
data and biological data
 Healthcare systems that use extensive databases like patient databases, health
insurance data, doctors' information, and bioinformatics information
2. Social media

• It is the data that is generated by various social media platforms like Twitter, Facebook,
YouTube, and Instagram.
3. Multimodal data
• It includes data that involves many modes such as text, video, audio and mixed types.

• They are listed below:

1. Image archives contain larger image databases along with numeric and text data
2. The World Wide Web (www) has huge amount of data that is distributed on the
Internet.

Data Preprocessing
• Data preprocessing improves the quality of the data mining techniques. The raw data
must be preprocessed to give accurate results.
• Data cleaning is the process of detection and removal of errors in data.

• Data wrangling is the process of making the data processable for machine learning
algorithms.

• Data errors include human errors such as

 Typographical Errors or Incorrect Measurement

 Structural Errors Like Improper Data Formats

• Data errors can arise from omission and duplication of attributes.

 Noise is a random component and involves distortion of a value or introduction of

spurious objects.
• Certain deterministic distortions in the form of a streak are known as artifacts.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • Example: The 'bad' or 'dirty' data can be observed in below table.

It can be observed that

• Data is incomplete like

 Salary = '' is incomplete data

 The DoB of patients, John, Andre, and Raju, is the missing data.
 The age of David is recorded as '5' but his DoB indicates it is 10/10/1980. This is
called inconsistent data.

• Inconsistent data occurs due to

 Problems In Conversions
 Inconsistent Formats
 Difference In Units

 Salary for John is -1500. It cannot be less than '0'. It is an instance of noisy data

• Outliers are data that exhibit the characteristics that are different from other data and
have very unusual values.
 The age of Raju cannot be 136

• Outliers may be legitimate data and sometimes are of interest to the data mining
algorithms.
• These errors often come during data collection stage. These must be removed so that
machine learning algorithms yield better results as the quality of results is determined
by the quality of input data. This removal process is called data cleaning.

Missing Data Analysis

• The primary data cleaning process is missing data analysis.

• Data cleaning routines attempt to fill up the missing values, smoothen the noise while
identifying the outliers and correct the inconsistencies of the data. This enables data
mining to avoid overfitting of the models.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • The procedures that are given below can solve the problem of missing data:

1. Ignore the tuple

 A tuple with missing data, especially the class label, is ignored.

 This method is not effective when the percentage of the missing values increases.
2. Fill in the values manually
 The domain expert can analyze the data tables and carry out the analysis and fill
in the values manually.

3. A global constant can be used to fill in the missing attributes.

 The missing values may be 'Unknown' or be 'Infinity’

4. The attribute value may be filled by the attribute value

 Example: the average income can replace a missing value
5. Use the attribute mean for all samples belonging to the same class.
6. Use the most possible value to fill in the missing value.
 The most probable value can be obtained from other methods like classification
and decision tree prediction.
 The filled value may not be correct and could be just an estimated value. Hence,
the difference between the estimated and the original value is called an error or
bias.

Removal of Noisy or Outlier Data

• Noise is a random error or variance in a measured value.

• It can be removed by using binning, which is a method where the given data values are
sorted and distributed into equal frequency bins. The bins are also called as buckets.

• The binning method then uses the neighbor values to smooth the noisy data.
• Some of the techniques commonly used are

1. 'Smoothing by means' where the mean of the bin removes the values of the bins
2. 'Smoothing by bin medians' where the bin median replaces the bin values
3. 'Smoothing by bin boundaries' where the bin value is replaced by the closest bin
boundary. The maximum and minimum values are called bin boundaries.
• Binning methods can be used as a discretization technique.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
Example: Consider the following set: S = [12, 14, 19, 22, 24, 26, 28, 31, 34). Apply various
binning techniques and show the result.

Solution:
• By equal-frequency bin method, the data should be distributed across bins.

• Let us assume the bins of size 3, then the above data is distributed across the bins
as shown below:

• As per the method,

 The minimum and maximum values of the bin are determined, and it serves as
bin boundary and does not change. Rest of the values are transformed to the
nearest value.

 It can be observed in bin 1, the middle value 14 is compared with the boundary
values 12 and 19 and changed to the closest value, that is 12. This process is
repeated for all bins.

Data Integration and Data Transformations

• Data integration involves routines that merge data from multiple sources into a single
data source. The goal of data integration is to detect and remove redundancies that arise
from integration.

• Data transformation routines perform operations like normalization to improve the

performance of the data mining algorithms.

• It is necessary to transform data so that it can be processed. It is considered as a

preliminary stage of data conditioning.
• Normalization is one such technique.
 In normalization, the attribute values are scaled to fit in a range (say 0-1) to
improve the performance of the data mining algorithm.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
  Some of the normalization procedures used are:

1. Min-Max
2. z-Score

Min-Max Procedure
• It is a normalization technique where each variable V is normalized by its difference
with the minimum value divided by the range to a new range, say 0-1.
• The formula to implement this normalization is given as:

Here,
 max-min is the range. Min and max are the minimum and maximum of the given
data
 New max and new min are the minimum and maximum of the target range, say 0
and 1

Example:

• Consider the set: V= (88, 90, 92, 94). Apply Min-Max procedure and map the marks to
a new range 0-1.

• Solution: The minimum of the list V is 88 and maximum is 94. The new min and new
max are 0 and 1, respectively. The mapping can be done using above equation as:

z- Score Normalization

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • This procedure works by taking the difference between the field value and mean value,
and by scaling this difference by standard deviation of the attribute.
V = V - μ/σ
• Here,

 σ is the standard deviation of the list V

 μ is the mean of the list V

Data Reduction
• Data reduction reduces data size but produces the same results.
• There are different ways in which data reduction can be carried out such as

1. Data aggregation
2. Feature selection

3. Dimensionality reduction

Descriptive Statistics
• Descriptive statistics is a branch of statistics that does dataset summarization.

• It is used to summarize and describe data. Descriptive statistics are just descriptive and
does not bother about machine learning algorithms and its functioning.

• Data visualization is a branch of study that is useful for investigating the given data.

• Descriptive analytics and data visualization techniques help to

 To understand the nature of the data

 To determine the kinds of machine learning or data mining tasks that can be
applied to the data.

 This step is often known as Exploratory Data Analysis (EDA).

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
  The focus of EDA is to understand the given data and to prepare it for machine
learning algorithms. EDA includes descriptive statistics and data visualization.

Dataset and Data Types

• A dataset can be assumed to be a collection of data objects. The data objects may be
records, points, vectors, patterns, events, cases, samples or observations. These records
contain many attributes.

• An attribute can be defined as the property or characteristics of an object.

• Example: Consider the following database shown below.

• Every attribute should be associated with a value. The process is called measurement.
• The type of attribute determines the data types referred as measurement scale types.

Categorical or Qualitative Data

• The categorical data can be divided into two types.

1. Nominal Data

• Nominal data are symbols and cannot be processed like a number.

 Example - The average of a patient ID does not make any statistical sense.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • Nominal data type provides only information but has no ordering among data.

• Only operations like (=,) are meaningful for these data.

 Example - The patient ID can be checked for equality and nothing else.
2. Ordinal Data

• It provides enough information and has natural order.

 Example - Fever = (Low, Medium, High) is an ordinal data.
Low is less than medium and medium is less than high, irrespective of the value.
Any transformation can be applied to these data to get a new value.

Numeric or Qualitative Data

• It can be divided into two categories.
1. Interval Data
 Interval data is a numeric data for which the differences between values are
meaningful.
 Example - There is a difference between 30 degree and 40 degree. Only the
permissible operations are + and -.

2. Ratio Data
 For ratio data, both differences and ratio are meaningful.

 The difference between the ratio and interval data is the position of zero in the scale.
 Example - Take the Centigrade-Fahrenheit conversion. The zeroes of both scales
do not match. Hence, these are interval data.

Another way of classifying the data is to classify it as:

1. Discrete value data
2. Continuous data

Discrete Data
• This kind of data is recorded as integers.
• Example -

1. The responses of the survey can be discrete data.

2. Employee identification number such as 10001 is discrete data.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
Continuous Data
• It can be fitted into a range and includes decimal point.

• Example - Age is a continuous data. Though age appears to be discrete data, one may
be 12.5 years old and it makes sense. Patient height and weight are all continuous data.

• Third way of classifying the data is based on the number of variables used in the dataset.
• The data can be classified as

1. Univariate Data: Dataset has only one variable.

2. Bivariate Data: Dataset has two variables.

3. Multivariate Data: Dataset uses three or more variables.

Univariate Data Analysis and Visualization

• Univariate analysis is the simplest form of statistical analysis.

• It indicates, the dataset has only one variable. A variable can be called as a category.

• The aim of univariate analysis is to describe data and find patterns.

• Univariate data description involves

1. Finding the frequency distributions

2. Central tendency measures
3. Dispersion or variation
4. Shape of the data

Data Visualization
• Data visualization helps to understand data. It helps to present information and data to
customers.
• The graphs that are used in univariate data analysis are

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 1. Bar charts

2. Histograms
3. Frequency polygons

4. Pie charts
• The advantages of the graphs are

1. Presentation of data

2. Summarization of data
3. Description of data

4. Exploration of data
5. To make comparisons of data

1. Bar Chart:
• A Bar chart (or Bar graph) is used to display the frequency distribution for variables.

• Bar charts are used to illustrate discrete data. The charts can also help to explain the
counts of nominal data. It also helps in comparing the frequency of different groups.

• Example: The bar chart for students' marks (45, 60, 60, 80, 85) with Student ID = {1,
2, 3, 4, 5) is shown below.

2. Pie Chart
• These are helpful in illustrating the univariate data.
Example:

• The percentage frequency distribution of student’s mark {22, 22, 40, 40, 70, 70, 70, 85,
90, 90}

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • It could be observed that number of students with 22 marks is 2. Total number of
students are 10. so 2/20 * 100 = 20% is allocated for marks 22

3. Histogram
• Plays a role in Data mining showing frequency distributions.

• The histogram for student’s mark {45, 60, 60, 80, 85} in the group range of 0-25, 26-
50, 51-75, 76-100 is given below.
• Histogram conveys useful information like nature of data and its mode.

• Mode indicates the peak of dataset.

4. Dot Plots
• These are similar to bar charts. They illustrate bars only with single points.

• Example: The dot plot of English marks for five students with ID as {1, 2, 3, 4, 5} and
marks {45, 60, 60, 80, 85}

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
Central Tendency
• Central tendency can explain the characteristics of data and that further helps in
comparison.

• Mass data have tendency to concentrate at certain values, normally in the central
location. It is called measure of central tendency (or averages). This represents the first
order of measures.
• Popular measures are

1. Mean

2. Median
3. Mode

1. Mean
• Arithmetic average (or mean) is a measure of central tendency that represents the
'center' of the dataset. This is used in our daily conversation such as average income or
average traffic.

• It can be found by adding all the data and dividing the sum by the number of
observations. Mathematically, the average of all the values in the sample (population)
is denoted as x.

• Let x1, x2, ….xN be a set of 'N' values or observations, then the arithmetic mean is given
as:

Weighted Mean

• Weighted mean gives different importance to all items as the item importance varies.
Hence, different weightage can be given to items.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • In case of frequency distribution, mid values of the range are taken for computation.
This is illustrated in the following computation.

• In weighted mean, the mean is computed by adding the product of proportion and group
mean.

• It is mostly used when the sample sizes are unequal.

Geometric mean
• Let x1, x2, ….xN be a set of 'N' values or observations.

• Geometric mean is the Nth root of the product of N items.

• The formula for computing geometric mean is given as follows:

2. Median
• The middle value in the distribution is called median.

 If the total number of items in the distribution is odd, then the middle value is called
median.
 If the numbers are even, then the average value of two items in the center is the
median.

• It can be observed that the median is the value where x, is divided into two equal halves,
with half of the values being lower than the median and half higher than the median.
• A median class is that class where (N/2)th item is present.

• Median is given by formula

• Median class is that class where N/2th item is present.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • Here, i is the class interval of the median class and L1, is the lower limit of median
class, f is the frequency of the median class, and cf is the cumulative frequency of all
classes preceding median.

3. Mode
• Mode is the value that occurs more frequently in the dataset.

• It is the value that has the highest frequency. It is only for discrete data and is not
applicable for continuous data as there are no repeated values in continuous data.

• The procedure for finding the mode is to calculate the frequencies for all the values in
the data, and mode is the value (or values) with the highest frequency.
• Normally, the dataset is classified as

 unimodal, bimodal and trimodal with modes 1, 2 and 3, respectively.

Dispersion
• The spread out of a set of data around the central tendency (mean, median or mode) is
called dispersion. Dispersion is represented by various ways such as range, variance,
standard deviation, and standard error.

• These are second order measures. The most common measures of the dispersion data
are listed below:
1. Range: Range is the difference between the maximum and minimum of values of
the given list of data.

2. Standard Deviation: The mean does not convey much more than a middle point.
Example:

• The following datasets (10, 20, 30} and {10, 50, 0) both have a mean of 20.
• The difference between these two sets is the spread of data.

• Standard deviation is the average distance from the mean of the dataset to each
point.
• The formula for sample standard deviation is given by:

• Here,

 N is the size of the population

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
  xi is observation or value from the population and u is the population
mean.

 N - 1 is used instead of N in the denominator

3. Quartiles and Inter Quartile Range
Inter Quartiles Range:

• Measure that is useful to measure dispersion is Inter Quartile Range (IQR).

• The IQR is the difference between Q3 and Q1.

Interquartile percentile = Q3 – Q1

Quartiles Range
• Quartile are values falling apart at least by the amount 1.5 * IQR above third
quartile or below the first quartile.

Interquartile is defined by Q0.75 – Q0.25

Five-point Summary and Box Plots

• The median, quartiles Q1 and Qy and minimum and maximum written in the order <
Minimum, Q1, Median, Qy Maximum > is known as five-point summary.
• Box plots are suitable for continuous variables and a nominal variable.

• Box plots can be used to illustrate data distributions and summary of data.

• It is the popular way for plotting five number summaries.

• A Box plot is also known as a Box and whisker plot.

• The box contains bulk of the data.

4
0
Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • These data are between first and third quartiles. The line inside the box indicates
location - mostly median of the data. If the median is not equidistant, then the data is
skewed.

• The whiskers that project from the ends of the box indicate the spread of the tails and
the maximum and minimum of the data value.

Shape
• Skewness and Kurtosis (called moments) indicate the symmetry/asymmetry and peak
location of the dataset.

1. Skewness
• The measures of direction and degree of symmetry are called measures of third order.

• The dataset may either have very high values or extremely low values.

 If the dataset has far higher values, then it is said to be skewed to the right.
 If the dataset has far lower values, then it is said to be skewed towards left.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
  If the tail is longer on the left-hand side and hump on the right-hand side, it is called
positive skew.

 Otherwise, it is called negative skew.

• The dataset may have an equal distribution of data.

 If the data is skewed, then there is a greater chance of outliers in the dataset.

 This affects the mean and median. Hence, this may affect the performance of the
data mining algorithm.
 A perfect symmetry means the skewness is zero.
• In the case of skew, the median is greater than the mean.

• In positive skew, the mean is greater than the median.

• In negatively skewed distribution, the median is more than the mean.

• The relationship between skew and the relative size of the mean and median can be
summarized by a convenient numerical skew index known as Pearson 2 skewness
coefficient.

Kurtosis
• Kurtosis also indicates the peaks of data.
• If the data is high peak, then it indicates higher kurtosis and vice versa.

• Kurtosis is the measure of whether the data is heavy tailed or light tailed relative to
normal distribution.

• It can be observed that

 Normal distribution has bell-shaped curve with no long tails

 Low kurtosis tends to have light tails

• Let x1, X2,….., XN be a set of 'N' values or observations. Then, kurtosis is measured
using the formula given below:

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
Mean Absolute Deviation (MAD)
• MAD is another dispersion measure and is robust to outliers.

• The outlier point is detected by computing the deviation from median and by dividing
it by MAD.

• Here, the absolute deviation between the data and mean is taken.
• Thus, the absolute deviation is given as:

Coefficient of Variation (CV)

• Coefficient of variation is used to compare datasets with different units.

• CV is the ratio of standard deviation and mean, and %CV is the percentage of
coefficient of variations.

Special Univariate Plots

• The ideal way to check the shape of the dataset is a stem and leaf plot.

• A stem and leaf plot are a display that help us to know the shape and distribution of the
data.

• In this method, each value is split into a ‘stem’ and a ‘leaf’.

• The last digit is usually the leaf and digits to the left of the leaf mostly form the stem.

• Example: Marks 45 are divided into stem 4 and leaf 5.

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri
 • It can be seen from above diagram that the first column is stem and the second column
is leaf.
• For the given English marks, two students with 60 marks are shown in stem and leaf
plot as stem-6 with 2 leaves with 0.

• A Q-Q plot can be used to assess the shape of the dataset.

• The Q-Q plot is a 2D scatter plot of an univariate data against theoretical normal
distribution data or of two datasets - the quartiles of the first and second datasets.
• The normal Q-Q plot for marks x= [1311234 89] is given below

Notes: Prepared by Akshatha Dange, Dept. of CSE (Iot, CS, BT), YIT, Moodabidri