0% found this document useful (0 votes)

26 views113 pages

R Programming Temperature Conversion Guide

Q: Explain why decision trees are suitable for both classification and regression tasks in machine learning.

Decision trees are suitable for both classification and regression due to their hierarchical structure and ability to make decisions based on input data attributes. For classification, they determine the class label by traversing nodes and evaluating attribute values, effectively making decisions based on different conditions . In regression tasks, decision trees predict continuous values by averaging the output variable in the leaf nodes. Their interpretability and capability to handle both numerical and categorical data make them versatile for various tasks .

Q: How does the k-means++ initialization method improve upon the traditional k-means clustering initialization, and why might this be advantageous in practice?

The k-means++ initialization method improves the traditional k-means clustering by selecting the first centroid randomly and then each subsequent centroid with a probability proportional to the distance from the closest existing centroid . This reduces the chances of poor clustering caused by unlucky initializations where centroids are too close to each other or isolated from data points . It significantly decreases the likelihood of the algorithm getting stuck in local minima, thus often leading to better results and faster convergence .

Q: Discuss the reasons behind utilizing uniform distribution for initial centroid selection in k-means clustering and its potential limitations.

Uniform distribution for initial centroid selection in k-means clustering helps ensure that centroids are spread across the entire data space, preventing them from being concentrated in one region . However, this method doesn't account for data density and can place centroids far from any data points, leading to slow convergence or suboptimal solutions . These limitations are partially mitigated by the k-means++ initialization, which considers data distribution when selecting centroids, improving convergence speed and clustering quality .

Q: What role does pruning play in decision trees, and how can it enhance a model's performance?

Pruning in decision trees involves removing branches that have little significance and do not provide much predictive power, thus reducing model complexity. Pruning enhances model performance by preventing overfitting; it removes parts of the tree that capture noise in the training data . This reduction in complexity helps maintain generalization by focusing on the most important decision paths, thus improving predictive accuracy on unseen data .

Q: Describe how the expectation-maximization (EM) algorithm is utilized in Gaussian Mixture Models (GMMs) for clustering.

In Gaussian Mixture Models, the EM algorithm is used to iteratively refine the parameters of the mixture components. During the E-step, it computes the probability (responsibility) that each data point belongs to each Gaussian based on current parameters. In the M-step, it updates the parameters (mean, covariance, and weight of each Gaussian) using these responsibilities to maximize the likelihood of the data. This alternation continues until convergence is achieved, ensuring that the final parameters are those that best explain the observed data .

Q: Evaluate the pros and cons of using decision trees compared to other machine learning algorithms.

Decision trees offer several advantages, such as interpretability, ease of explanation, and the ability to handle both numerical and categorical data . They are versatile and require minimal preprocessing of data. However, they can be prone to overfitting, especially if not pruned adequately. Compared to more complex models like random forests or neural networks, decision trees might not capture complex relationships as effectively and typically have lower predictive power in practice. Nevertheless, their simplicity and visual appeal make them a popular choice for initial modeling and understanding data-driven decision processes .

Q: How can the decision criterion in building decision trees affect the quality of the tree's predictions?

The decision criterion, such as Gini impurity or information gain, guides how nodes are split in decision trees. The choice of criterion directly influences which attributes are used at each split, impacting the tree's depth and complexity . A criterion that better captures the variance or information in the data will result in clearer, more effective splits, enhancing predictive accuracy. Poorly chosen criteria can lead to suboptimal trees with high bias or variance, reducing both interpretability and predictive capability .

Q: What challenges arise from using continuous attributes in decision trees and how are they addressed?

Continuous attributes in decision trees can lead to complex splitting conditions and potentially deep trees. These attributes require discretization or selection of thresholds for splitting, which can increase computational complexity . The decision tree algorithm addresses this by determining optimal points to split the continuous data using criteria like information gain. Proper handling of continuous attributes is crucial for preventing overfitting and ensuring manageable tree complexity .

Q: Illustrate how the process of centroid selection differs in the traditional k-means algorithm and the k-means++ algorithm.

In the traditional k-means algorithm, centroids are initialized randomly throughout the data space without any consideration of data distribution, which can lead to suboptimal clustering if centroids are poorly initialized. In contrast, the k-means++ algorithm starts by selecting the first centroid randomly, but subsequent centroids are chosen based on a probability proportional to the squared distance from the nearest existing centroid . This method aims to position initial centroids more strategically and thus often results in better clustering performance .

Q: Analyze the impact of the stopping criterion in decision tree algorithms on model complexity and generalization.

The stopping criterion in decision tree algorithms, such as maximum depth or minimum instances per leaf, directly impacts model complexity and generalization. A stringent stopping criterion reduces tree depth, potentially underfitting by not capturing sufficient complexity in the data. Conversely, lenient criteria allow deeper trees, possibly capturing noise and leading to overfitting . Therefore, stopping criteria need careful calibration to balance complexity and generalization, ensuring the tree is neither too simplistic nor overly detailed .

The document provides a comprehensive guide on data analytics using R programming, covering temperature conversions, area calculations for various shapes, and string manipulation techniques. It includes detailed examples and functions for converting temperatures between Celsius, Fahrenheit, and Kelvin, as well as calculating areas of geometric shapes and generating sequences of even numbers. Additionally, it demonstrates how to manipulate data frames and perform string operations in R.

Uploaded by

aruna

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as DOCX, PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

26 views113 pages

R Programming Temperature Conversion Guide

Uploaded by

aruna

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as DOCX, PDF, TXT or read online on Scribd

Data Analytics using R Programming Lab

1. Program to convert the given temperature from Fahrenheit to Celsius and vice versa depending upon user’s
choice.

Discover the art of precise temperature conversion in R with our

comprehensive article. Uncover the secrets behind seamlessly converting
Celsius, Fahrenheit, and Kelvin scales using R programming. Whether
you’re a data enthusiast or a scientific researcher, this article equips you
with the tools and knowledge to master temperature conversions
effortlessly. Say goodbye to temperature confusion and hello to accuracy in
your R projects.
Concepts
Celsius (°C): The Celsius scale is a commonly used temperature scale in
which 0 degrees Celsius represents the freezing point of water and 100
degrees Celsius represents the boiling point of water at standard
atmospheric pressure.
Fahrenheit (°F): The Fahrenheit scale is mainly used in the United States.
In this scale, 32 degrees Fahrenheit represents the freezing point of water,
and 212 degrees Fahrenheit represents the boiling point of water at
standard atmospheric pressure.
Kelvin (K): The Kelvin scale is a scientific temperature scale that starts
from absolute zero, the lowest possible temperature at which all molecular
motion ceases. Absolute zero is defined as 0 Kelvin.
Conversion Formulas:
To convert from Celsius to Kelvin:
Kelvin = Celsius + 273.15
To convert from Celsius to Fahrenheit:
Fahrenheit = (Celsius * 9/5) + 32
To convert from Fahrenheit to Celsius:
Celsius = (Fahrenheit – 32) * 5/9
To convert from Kelvin to Celsius:
Celsius = Kelvin – 273.15
To convert from Fahrenheit to Kelvin:
Kelvin = (Fahrenheit – 32) * 5/9 + 273.15
Converting Celsius to Kelvin
 R

# Convert Celsius to Kelvin

celsius_to_kelvin <- function(celsius) {
return(celsius + 273.15)
}

# Example: Convert 25 degrees Celsius to Kelvin

celsius_temp <- 25
kelvin_temp <- celsius_to_kelvin(celsius_temp)
cat(celsius_temp, "degrees Celsius is equal to", kelvin_temp, "Kelvin\n")

Output:
25 degrees Celsius is equal to 298.15 Kelvin

 We start with 25 degrees Celsius.

 The celsius_to_kelvin function is used to convert this temperature
to Kelvin using the formula: Kelvin = Celsius + 273.15.
 The result is printed, stating that 25 degrees Celsius is equal to the
calculated value in Kelvin.
Celsius to Fahrenheit Conversion
 R

# Convert Celsius to Fahrenheit

celsius_to_fahrenheit <- function(celsius) {
return((celsius * 9/5) + 32)
}

# Example: Convert 25 degrees Celsius to Fahrenheit

celsius_temp <- 25
fahrenheit_temp <- celsius_to_fahrenheit(celsius_temp)
cat(celsius_temp, "degrees Celsius is equal to", fahrenheit_temp, "Fahrenheit\n")

Output:
25 degrees Celsius is equal to 77 Fahrenheit

 We start with 25 degrees Celsius.

 The celsius_to_fahrenheit function is used to convert this
temperature to Fahrenheit using the formula: Fahrenheit = (Celsius
* 9/5) + 32.
 The result is printed, stating that 25 degrees Celsius is equal to the
calculated value in Fahrenheit.
Fahrenheit to Celsius Conversion
 R

# Convert Fahrenheit to Celsius

fahrenheit_to_celsius <- function(fahrenheit) {
return((fahrenheit - 32) * 5/9)
}

# Example: Convert 77 degrees Fahrenheit to Celsius

fahrenheit_temp <- 77
celsius_temp <- fahrenheit_to_celsius(fahrenheit_temp)
cat(fahrenheit_temp, "degrees Fahrenheit is equal to", celsius_temp, "Celsius\n")

Output:
77 degrees Fahrenheit is equal to 25 Celsius

 We start with 77 degrees Fahrenheit.

 The fahrenheit_to_celsius function is used to convert this
temperature to Celsius using the formula: Celsius = (Fahrenheit –
32) * 5/9.
 The result is printed, stating that 77 degrees Fahrenheit is equal to
the calculated value in Celsius.
Kelvin to Celsius Conversion
 R

# Convert Kelvin to Celsius

kelvin_to_celsius <- function(kelvin) {
return(kelvin - 273.15)
}

# Example: Convert 298.15 Kelvin to Celsius

kelvin_temp <- 298.15
celsius_temp <- kelvin_to_celsius(kelvin_temp)
cat(kelvin_temp, "Kelvin is equal to", celsius_temp, "degrees Celsius\n")

Output:
298.15 Kelvin is equal to 25 degrees Celsius

 We start with 298.15 Kelvin.

 The kelvin_to_celsius function is used to convert this temperature
to Celsius using the formula: Celsius = Kelvin – 273.15.
 The result is printed, stating that 298.15 Kelvin is equal to the
calculated value in Celsius.
Fahrenheit to Kelvin Conversion
 R

# Convert Fahrenheit to Kelvin

fahrenheit_to_kelvin <- function(fahrenheit) {
return((fahrenheit - 32) * 5/9 + 273.15)
}

# Example: Convert 77 degrees Fahrenheit to Kelvin

fahrenheit_temp <- 77
kelvin_temp <- fahrenheit_to_kelvin(fahrenheit_temp)
cat(fahrenheit_temp, "degrees Fahrenheit is equal to", kelvin_temp, "Kelvin\n")

Output:
77 degrees Fahrenheit is equal to 298.15 Kelvin

 We start with 77 degrees Fahrenheit.

 The fahrenheit_to_kelvin function is used to convert this
temperature to Kelvin using the formula: Kelvin = (Fahrenheit – 32)
* 5/9 + 273.15.
 The result is printed, stating that 77 degrees Fahrenheit is equal to
the calculated value in Kelvin.

2. Program, to find the area of rectangle, square, circle and triangle by accepting suitable input parameters from user.

Syntax:
# Calculate the area of the triangle
area <- 0.5 * base * height
# Print the result
cat("The area of the triangle is:", area)

Example 1:
 R

# Values for base and height

base <- 10
height <- 5

# Calculate the area of the triangle

area <- 0.5 * base * height

# Print the result

cat("The area of the triangle is:", area)

Output:
The area of the triangle is: 25

 Assigns the value 10 to the variable base, representing the length

of the base of the triangle.
 Assigns the value 5 to the variable height, representing the height
of the triangle.
 Calculates the area of the triangle using the formula 0.5 * base *
height and stores the result in the variable area.
 Uses the cat() function to print the message.
Example 2:
 R

# Values for base and height

base <- 10
height <- 10

# Calculate the area of the triangle

area <- 0.5 * base * height

# Print the result

cat("The area of the triangle is:", area)

Output:
The area of the triangle is: 50

2. Write a program to find list of even numbers from 1 to n using RLoops.

R Example: How to display even numbers from 1 to 100 in R

1
2for (num in 1:100) {
3 if (num %% 2 == 0) {
4 print(paste("Even number is :", num))
5}
6}
7

3. Create a function to print squares of numbers in sequence.

# Function to calculate the square of a number using a while loop

calculateSquare <- function(num) {
i <- 1
while (i <= num) {
square <- i * i
cat(i, "squared is", square, "\n")
i <- i + 1
}
}

# Example usage
input <- 10
calculateSquare(input)

Output:
1 squared is 1
2 squared is 4
3 squared is 9
4 squared is 16
5 squared is 25
6 squared is 36
7 squared is 49
8 squared is 64
9 squared is 81
10 squared is 100

# Function to calculate the squares of numbers and store in a vector

calculateSquaresVector <- function(num) {
i <- 1
squares <- c()
while (i <= num) {
square <- i * i
squares <- c(squares, square)
i <- i + 1
}
return(squares)
}

# Example usage
input <- 10
output <- calculateSquaresVector(input)
print(output)

Output:
[1] 1 4 9 16 25 36 49 64 81 100

4. Write a program to join columns and rows in a data frame using cbind() and rbind() in R.
Let’s start by creating two vectors, having first and last names of players, by using following commands.

Download Example File

f_name <- c("Stephen","Chris","Derrick")
l_name <- c("Cury","Paul","Rose")

Using cbind() in R
The first function we are using is cbind() which essentially stands for column bind. This function is used to bind
vectors or matrices as columns to create a new matrix. So for the above vectors we just created, if we want to
combine them by column, the following command is used

full_cbind <- cbind(f_name,l_name)

The above command gives following results

It’s important to note that cbind() works element-wise, meaning it combines the elements of vectors or
matrices based on their positions in columns.
Using rbind() in R
Just like cbind() is used to combine columns of vectors or data frames, rbind() combines the rows of vectors or
data frames. It binds vectors, matrices, or data frames by rows to create a new vector, matrix, or data frame. Let’s
demonstrate it by using the above vectors created earlier. If we want the rows of above data set to be combined,
use the following command

full_rbind <- rbind(f_name,l_name)

The command combines the rows of data in following way

Combining vectors with Data frames:

Now, if we have a data frame, and we want to combine these vectors and have a data frame, this could also be
done by using the cbind() or rbind() function, depending on whether you want the data to be combined by
columns or by rows. To understand this, let’s create a data frame using following following command

basketball <- [Link](f_name = c("Stephen", "Chris" , "Derrick"),

l_name = c("Cury", "Paul" , "Rose"))
To create a vector for combining it later with the data frame, we use following command

age <- c("32" , "29" , "34")

Both data frame and vector has been created, let’s combine these by using following command

all_details <- cbind(basketball,age)

The above command combines vector with data frame in a single data set, as shown below.

Note that, in above commands, both data frame and vector are of same length, so they get combined smoothly.

Combine Multiple vectors or Data frames using cbind() and rbind()

In order to combine multiple vectors or data frames, we again take the example of the above created vectors
named “f_name, l_name, age”. To combine these vectors, use the following command

three_vectors <- cbind(f_name,l_name,age)

This generates the following output, showing a matrix of 3 by 3.

It’s important to note that the resulting object consists of three vectors side by side, forming rows and columns.
These vectors combined above by the name of “three_vectors” can be converted into data frames by the
following command
three_vectors <- [Link](three_vectors)

Combining vectors and data frames of different lengths

When you want to combine data of different lengths, you generally have a few options to use. For instance, let’s
generate data frames of different lengths by using following commands

football <- [Link](f_name_foot = c("Mathew", "Andrew" , "Brad","Joe"),

l_name_foot = c("Ryan", "Redmayne" , "Jones","Gauci")) basketball <-
[Link](f_name = c("Stephen", "Chris" , "Derrick"),
l_name = c("Cury", "Paul" , "Rose"))

For above data frames, football has 4 observations in each column, and basketball has three observations in each
column. Thus, the length of data set is different.

Now, if we combine these two data frames of different lengths; football and basketball, either by using rbind() or
cbind() function, the command wouldn’t run. Let’s try by using both commands as given below
missing_rbind <- rbind(basketball,football) missing_cbind <-
cbind(basketball,football)

The following error will be shown by the above commands. This implies that the length of data sets required to
combine should be same.

The only way to combine these data frames of different lengths is by using another function, bind_rows(). To use
this function, we first load the following library

library(dplyr)

Next, use the following command to combine these data frames

missing_rbind <- bind_rows(basketball,football)

The output generated from the above command is following

The rbind() function appends the data set in a way that any missing columns in the shorter data frame are filled
with NA values, as shown in the above image.
Another way to combine these columns of different lengths is by using the merge() function. The merge function
is used for merging data frames based on common columns.

The following command shows how we can use merge() function to combine these columns of different lengths.

football <- cbind("row_no"=[Link](football),football) basketball <-

cbind("row_no"=[Link](basketball),basketball) missing_cbind <-
merge(football,basketball, all=TRUE)

In the first two commands, row_no assigns unique row number to each row in the data frame. It creates a new
column called “row_no” in both the “football” and “basketball” data frames. The purpose of adding the
“row_no” column is to give each row a unique identifier so that the data frames can be merged based on these
identifiers. the next step is to merge these columns based on that unique identifier. The following output is
generated from the above command
Similarly, if you have vectors of different lengths, they should be assigned a same length or maximum length
of the vector, and then combine using cbind() or rbind() function. If three vectors have different length, then we
first find out the maximum length of the vector by using following command
m_len <- max(length(f_name), length(l_name),length(age))

The output shows that maximum length of above vectors is 3, so we assign same length to each of the above
vectors. Following commands should be used to serve the purpose

length(f_name) <- m_len

length(l_name) <- m_len
length(age) <- m_len

Once the same length has been assigned, next we combine these vectors by using either of the functions, as
shown in the command below

diff_len <- cbind(f_name,l_name,age)

The above command combines vectors and save them by the name of diff_len.

Tweet
Share
Share
Pin

5. Implement different String Manipulation functions in R.

String Manipulation in R
Concatenation of Strings
String Concatenation is the technique of combining two strings. String
Concatenation can be done using many ways:
 paste() function Any number of strings can be concatenated
together using the paste() function to form a larger string. This
function takes separator as argument which is used between the
individual string elements and another argument ‘collapse’ which
reflects if we wish to print the strings together as a single larger
string. By default, the value of collapse is NULL. Syntax:
paste(..., sep=" ", collapse = NULL)
 Example:
 Python3

# R program for String concatenation

# Concatenation using paste() function

str <- paste("Learn", "Code")
print (str)

 Output:
"Learn Code"
 In case no separator is specified the default separator ” ” is inserted
between individual strings. Example:
 Python3

str <- paste(c(1:3), "4", sep = ":")

print (str)

 Output:
"1:4" "2:4" "3:4"
 Since, the objects to be concatenated are of different lengths, a
repetition of the string of smaller length is applied with the other
input strings. The first string is a sequence of 1, 2, 3 which is then
individually concatenated with the other string “4” using separator
‘:’.
 Python3

str <- paste(c(1:4), c(5:8), sep = "--")

print (str)

 Output:
"1--5" "2--6" "3--7" "4--8"
 Since, both the strings are of the same length, the corresponding
elements of both are concatenated, that is the first element of the
first string is concatenated with the first element of second-string
using the sep ‘–‘.
 cat() function Different types of strings can be concatenated
together using the cat()) function in R, where sep specifies the
separator to give between the strings and file name, in case we wish
to write the contents onto a file. Syntax:
cat(..., sep=" ", file)
 Example:
 Python3
# R program for string concatenation

# Concatenation using cat() function

str <- cat("learn", "code", "tech", sep = ":")
print (str)

 Output:
learn:code:techNULL
 The output string is printed without any quotes and the default
separator is ‘:’.NULL value is appended at the end. Example:
 Python3

cat(c(1:5), file ='[Link]')

 Output:
1 2 3 4 5
The output is written to a text file [Link] in the same working directory.
Calculating Length of strings
 length() function The length() function determines the number of
strings specified in the function. Example:
 Python3

# R program to calculate length

print (length(c("Learn to", "Code")))

 Output:
2
 There are two strings specified in the function.
 nchar() function nchar() counts the number of characters in each
of the strings specified as arguments to the function
individually. Example:
 Python3

print (nchar(c("Learn", "Code")))

 Output:
5 4
 The output indicates the length of Learn and then Code separated
by ” ” .
Case Conversion of strings
 Conversion to upper case All the characters of the strings
specified are converted to upper case. Example:
 Python3
print (toupper(c("Learn Code", "hI")))

 Output :
"LEARN CODE" "HI"
 Conversion to lower case All the characters of the strings
specified are converted to lower case. Example:
 Python3

print (tolower(c("Learn Code", "hI")))

 Output :
"learn code" "hi"
 casefold() function All the characters of the strings specified are
converted to lowercase or uppercase according to the arguments in
casefold(…, upper=TRUE). Examples:
 Python3

print (casefold(c("Learn Code", "hI")))

 Output:
"learn code" "hi"
 By default, the strings get converted to lower case.
 Python3

print (casefold(c("Learn Code", "hI"), upper = TRUE))

 Output:
"LEARN CODE" "HI"
Character replacement
Characters can be translated using the chartr(oldchar, newchar, …) function
in R, where every instance of old character is replaced by the new character
in the specified set of strings. Example 1:
 Python3

chartr("a", "A", "An honest man gave that")

Output:
"An honest mAn gAve thAt"
Every instance of ‘a’ is replaced by ‘A’. Example 2:
 Python3

chartr("is", "#@", c("This is it", "It is great"))

Output:
"Th#@ #@ #t" "It #@ great"
Every instance of old string is replaced by new specified string. “i” is
replaced by “#” by “s” by “@”, that is the corresponding positions of old
string is replaced by new string. Example 3:
 Python3

chartr("ate", "#@", "I hate ate")

Output:
Error in chartr("ate", "#@", "I hate ate") : 'old' is longer than 'new'
Execution halted
The length of the old string should be less than the new string.
Splitting the string
A string can be split into corresponding individual strings using ” ” the
default separator. Example:
 Python3

strsplit("Learn Code Teach !", " ")

Output:
[1] "Learn" "Code" "Teach" "!"
Working with substrings
substr(…, start, end) or substring(…, start, end) function in R extracts
substrings out of a string beginning with the start index and ending with the
end index. It also replaces the specified substring with a new set of
characters. Example:
 Python3

substr("Learn Code Tech", 1, 4)

Output:
"Lear"
Extracts the first four characters from the string.

 Python3

str & lt
- c(& quot
program", & quot
with"
, & quot
new"
, & quot
language"
)
substr(str, 3, 3) & lt
- & quot
% & quot
print(str)

Output:
"pr%gram" "wi%h" "ne%" "la%guage"
Replaces the third character of every string with % sign.

 Python3

str <- c("program", "with", "new", "language")

substr(str, 3, 3) <- c("%", "@")
print(str)

Output:
"pr%gram" "wi@h" "ne%" "la@guage"
6. Implement different data structures in R (Vectors, Lists, Data Frames)

#############################################
##########
# LESSON 3: VECTORS, LISTS, MATRICES, AND
DATA FRAMES #
# Christopher Jeruzal
#
# 09/09/2018
#
#############################################
##########

# PART 1: WORKING WITH VECTORS, SCALARS, AND

FACTORS
# Vectors are able to hold one kind of data
type or mode (e.g. numeric, character).
# Create a numeric vector using the combine
function 'c()'.
my_first_vector <- c(1, 2, 3, 4, 5)

# Now, create a vector containing strings of

fruit names
my_second_vector <- c("Orange", "Apple",
"Banana", "Pear", "Peach")

# To reference the first element in the

vector use [1]
my_second_vector[1]

# Create a new vector that is a subset of the

original vector's last three items using the
'[3:5]'
my_third_vector <- my_second_vector[3:5]

# Create a new vector that is a subset of the

original vector's first and last items using
the '[c(1,5)]'
my_fouth_vector <- my_second_vector[c(1,5)]

# Add another fruit to the vector

my_fifth_vector <- c(my_second_vector,
"Apricot")

# Scalars are vectors with only one element

# Pi is a scalar. You can test this by
referencing it's "first" element
pi[1]

# Factors are similar to vectors except is

stores each unique value in the vector as a
'level' or label
# Factors can used to label or categorize
your data.
my_first_factors <- factor(my_fifth_vector)

# PART 2: WORKING WITH LISTS

# Unlike vectors, lists are able to hold more
than one kind of data type or mode (e.g.
numeric, character).
# Create a list with numeric and character
data modes using the 'list()' function.
my_first_list <- list(1, 2, 3, "car",
"truck", "van")

# To reference the fourth element in the list

use [[4]]
my_first_list[[4]]
# Create a new list that is a subset of the
original list's last three items using the
'[3:5]'
my_second_list <- my_first_list[3:5]

# Create a new list that is a subset of the

original list's first and last items using
the '[c(1,5)]'
my_third_list <- my_first_list[c(1,5)]

# Add another vehicle to the list

my_fourth_list <- c(my_first_list, "suv")

# PART 3: WORKING WITH MATRICES & ARRAYS

# Matrices are vectors with one or more
dimensions.
# A matrix can be produced from multiple
vectors.
a <- c(1,2,3)
b <- c(4,5,6)
c <- c(7,8,9)

# Create a matrix using the 'matrix' function

and the three vectors above.
# There are three parameters used by the
'matrix' function that define what the matrix
looks like.
# The 'data' parameter allow you to specify
what vectors to use for data using the 'c()'
function.
# The 'nrow' and 'ncol' allow you tell R how
many rows and columns define the matrix.
my_first_matrix <- matrix(data = c(a,b,c),
nrow = 3, ncol = 3)

# To reference the item in the first row of

the third column in a matrix use the '[1,3]'
my_first_matrix[1,3]

# Create a new vector

d <- c(10,11,12)
# Add the new vector to 'my_first_matrix' as
an additional column
my_second_matrix <- cbind(my_first_matrix,d)

# Add the new vector to 'my_first_matrix' as

an additional row
my_third_matrix <- rbind(my_first_matrix,d)

# Arrays are just matrices with 3 or more

dimensions.
# First lets make a vector with 27 items
my_first_array <- c(1:27)

# Now, let's divide the vector up into three

3 x 3 matrices and stack them... your first
array
dim(my_first_array) <- c(3,3,3)

# Check out your first array

my_first_array

# PART 4: WORKING WITH DATA FRAMES

# Data frames are designed to hold tabular
data. They are similar to spreadsheets.
# They have columns and rows and can store
all data types or modes.
# Data frames can be built from vectors,
lists, and matrices.
# Let's build our first data frame from
'my_first_vector' and 'my_second_vector'
using the '[Link]'
# function. For this example, we're going
also going to set a parameter called
'stringsAsFactors' to FALSE.
# This will make sure that the string data is
saved as 'character' data type rather than
'factor' data type.
my_first_df <- [Link](my_first_vector,
my_second_vector, stringsAsFactors = FALSE)
# We can also create data frames from lists,
so we'll do that next
# But, before we do that we have to create a
list of vectors
my_vector_list <- list(my_first_vector,
my_second_vector)

# When using lists to create a data frame,

you need to use the '[Link]' function
rather than the '[Link]' function
my_other_df <- [Link](my_vector_list,
stringsAsFactors = FALSE)

# Since the default column names are the

vector names, let's rename the columns
names(my_first_df) <- c("Quantity", "Fruit")

# Using matrix notation, let see what's in

row 3
my_first_df[3,]

# Now, Let see what's in column 2

my_first_df[,2]

# OR use the name of column 2 to see what's

in it
my_first_df$Fruit

# Next, Let's see what's in row 4, column 2

my_first_df[4,2]

# Now, Let's change the value in row 4,

column 2 from 'Pear' to 'Plum'
my_first_df[4,2] <- "Plum"

8 Write a program to read a csv file and analyze the data in the file in R.
BLACK FRIDAY SALE
Get Programiz PRO for LIFE at 60% off!
Claim My Discount
Sale ends in 00d : 02hrs : 08mins : 30s

TutorialsExamples Courses
Login to PRO

 R Introduction

o
o
o
o
o
o
 R Flow Control

o
o
o
o
o
o
o
o
 R Data Structure

o
o
o
o
o
o
o
 R Data Visualization

o
o
o
o
o
o
o
o
 R Data Manipulation

o R Read and Write CSV

o R Read and Write xlsx
o R Dataset
o R min() and max()
o R mean, median and mode
o R Percentile

 R Additional Topics

o
o
o
o

R Tutorials

 R Read and Write xlsx Files

 R Data Frame
 R Data Frame
 R Save Plots to File
 R dataset
 R Save Plot

R Read and Write CSV Files

The CSV (Comma Separated Value) file is a plain text file that uses a comma to
separate values.

R has a built-in functionality that makes it easy to read and write a CSV file.

Sample CSV File

To demonstrate how we read CSV files in R, let's suppose we have a CSV file
named [Link] with following data:

Month, 1958, 1959, 1960

JAN, 340, 360, 417
FEB, 318, 342, 391
MAR, 362, 406, 419
APR, 348, 396, 461
MAY, 363, 420, 472
JUN, 435, 472, 535
JUL, 491, 548, 622
AUG, 505, 559, 606
SEP, 404, 463, 508
OCT, 359, 407, 461
NOV, 310, 362, 390
DEC, 337, 405, 432

The CSV file above is a sample data of monthly air travel, in thousands of passengers,
for 1958-1960.
Now, let's try to read data from this CSV File using R's built-in functions.

Read a CSV File in R

In R, we use the [Link]() function to read a CSV file available in our current
directory. For example,

# read [Link] file from our current directory

read_data <- [Link]("[Link]")

# display csv file

print(read_data)

Output

Month, 1958, 1959, 1960

1 JAN 340 360 417
2 FEB 318 342 391
3 MAR 362 406 419
4 APR 348 396 461
5 MAY 363 420 472
6 JUN 435 472 535
7 JUL 491 548 622
8 AUG 505 559 606
9 SEP 404 463 508
10 OCT 359 407 461
11 NOV 310 362 390
12 DEC 337 405 432

In the above example, we have read the [Link] file that is available in our
current directory. Notice the code,
read_data <- [Link]("[Link]")

Here, [Link]() reads the csv file [Link] and creates a dataframe which is
stored in the read_data variable.
Finally, the csv file is displayed using print() .

Note: If the file is in some other location, we have to specify the path along with the file
name as: [Link]("D:/folder1/[Link]") .

Number of Rows and Columns of CSV File in R

We use the ncol() and nrow() function to get the total number of rows and columns
present in the CSV file in R. For example,

# read [Link] file from our directory

read_data <- [Link]("[Link]")

# print total number of columns

cat("Total Columns: ", ncol(read_data))

# print total number of rows

cat("Total Rows:", nrow(read_data))

Output

Total Columns: 4
Total Rows: 12

In the above example, we have used the ncol() and nrow() function to find the total
number of columns and rows in the [Link] file.
Here,

 ncol(read_data) - returns total number of columns i.e. 4

 nrow(read_data) - returns total number of rows i.e. 12
Using min() and max() With CSV Files
In R, we can also find minimum and maximum data in a certain column of a CSV file
using the min() and max() function. For example,

# read [Link] file from our directory

read_data <- [Link]("[Link]")

# return minimum value of 1960 column of [Link]

min_data <- min(read_data$1960) # 390

# return maximum value of 1958 column of [Link]

min_data <- max(read_data$1958) # 505

Output

[1] 390
[1] 505

Here, we have used the min() and max() function to find the minimum and maximum
value of the 1960 and 1958 column of the [Link] file respectively.
 min(read_data$1960) - returns the minimum value from the 1960 column i.e. 390
 max(read_data$1958) - returns the maximum value from the 1958 column i.e. 505

Subset of a CSV File in R

In R, we use the subset() function to return all the datas from a CSV file that satisfies
the specified condition. For example,

# read [Link] file from our directory

read_data <- [Link]("[Link]")

# return subset of csv where number of air

# traveler in 1958 should be greater than 400
sub_data <- subset(read_data, 1958 > 400)

print(sub_data)

Output
Month, 1958, 1959, 1960
6 JUN 435 472 535
7 JUL 491 548 622
8 AUG 505 559 606
9 SEP 404 463 508

In the above example, we have specified a certain condition inside

the subset() function to extract data from a CSV file.

subset(read_data, 1958 > 400)

Here, subset() creates a subset of [Link] with data column 1958 having data
greater than 400 and stored it in the sub_data data frame.
Since column 1958 has data greater than 400 in 6th, 7th, 8th, and 9th row, only these
rows are displayed.

Write Into CSV File in R

In R, we use the [Link]() function to write into a CSV file. We pass the data in the
form of dataframe. For example,

# Create a data frame

dataframe1 <- [Link] (
Name = c("Juan", "Alcaraz", "Simantha"),
Age = c(22, 15, 19),
Vote = c(TRUE, FALSE, TRUE))

# write dataframe1 into file1 csv file

[Link](dataframe1, "[Link]")

In the above example, we have used the [Link]() function to export a data frame
named dataframe1 to a CSV file. Notice the arguments passed inside [Link]() ,

[Link](dataframe1, "[Link]")

Here,

 dataframe1 - name of the data frame we want to export

 [Link] - name of the csv file
Finally, the [Link] file would look like this in our directory:

CSV FIle System Output

If we pass "quote = FALSE" to [Link]() as:

[Link](dataframe1, "[Link]",
quote = FALSE
)

Our [Link] would look like this:

9 Create pie chart and bar chart using R.

R – Pie Charts
R Programming Language uses the function pie() to create pie charts. It
takes positive numbers as a vector input.
Syntax: pie(x, labels, radius, main, col, clockwise)
Parameters:
 x: This parameter is a vector that contains the numeric values
which are used in the pie chart.
 labels: This parameter gives the description to the slices in pie
chart.
 radius: This parameter is used to indicate the radius of the circle
of the pie chart.(value between -1 and +1).
 main: This parameter is represents title of the pie chart.
 clockwise: This parameter contains the logical value which
indicates whether the slices are drawn clockwise or in anti
clockwise direction.
 col: This parameter give colors to the pie in the graph.

Creating a simple pie chart

To create a simple R pie chart:
 By using the above parameters, we can draw a pie chart.
 It can be described by giving simple labels.
Example:
R
1
# Create data for the graph.
2
geeks<- c(23, 56, 20, 63)
3
labels <- c("Mumbai", "Pune", "Chennai", "Bangalore")
4

5
# Plot the chart.
6
pie(geeks, labels)
Output:
R – Pie Charts

Pie chart including the title and colors

To create a color and title pie chart.
 Take all parameters which are required to make a R pie chart by
giving a title to the chart and adding labels.
 We can add more features by adding more parameters with more
colors to the points.
Example:
R
1
# Create data for the graph.
2
geeks<- c(23, 56, 20, 63)
3
labels <- c("Mumbai", "Pune", "Chennai", "Bangalore")
4

5
# Plot the chart with title and rainbow
6
# color pallet.
7
pie(geeks, labels, main = "City pie chart",
8
col = rainbow(length(geeks)))
Output:
R – Pie Charts

Slice Percentage & Chart Legend

To create chart legend and slice percentage, we can plot by doing the
below methods.
 There are two more properties of the pie chart:

slice percentage
o
o chart legend.
 We can show the chart in the form of percentages as well as add
legends.
Example:
R
1
# Create data for the graph.
2
geeks <- c(23, 56, 20, 63)
3
labels <- c("Mumbai", "Pune", "Chennai", "Bangalore")
4

5
piepercent<- round(100 * geeks / sum(geeks), 1)
6

7
# Plot the chart.
8
pie(geeks, labels = piepercent,
9
main = "City pie chart", col = rainbow(length(geeks)))
10
legend("topright", c("Mumbai", "Pune", "Chennai", "Bangalore"),
11
cex = 0.5, fill = rainbow(length(geeks)))
Output:

R – Pie Charts

Add pie chart color palettes

With the help [Link] function of the RColorBrewer package in R.
R
1
Get the library.
2
library(RColorBrewer)
3

4
# Create data for the graph.
5
geeks <- c(23, 56, 20, 63)
6
labelss <- c("Mumbai", "Pune", "Chennai", "Bangalore")
7

8
labels<- [Link](length(geeks), "Set2")
9
10
pie(geeks, labels = labelss)
Output:

R – Pie Charts

modify the line type of the borders of the plot we can make use of
the lty argument:
R
1
Get the library.
2
library(RColorBrewer)
3

4
# Create data for the graph.
5
geeks <- c(23, 56, 20, 63)
6
labelss <- c("Mumbai", "Pune", "Chennai", "Bangalore")
7

8
labels<- [Link](length(geeks), "Set2")
9

10
pie(geeks, labels = labelss, col = color, lty = 2)
Output:
R – Pie Charts

Add shading lines with the density argument.

R
1
#Get the library.
2
library(RColorBrewer)
3

4
# Create data for the graph.
5
geeks <- c(23, 56, 20, 63)
6
labelss <- c("Mumbai", "Pune", "Chennai", "Bangalore")
7

8
labels<- [Link](length(geeks), "Set2")
9

10
pie(geeks, labels = labelss,col = color, density = 50, angle = 45)
Output:
R – Pie Charts

3D Pie Chart
Here we are going to create a 3D Pie chart using plotrix package and then
we will use pie3D() function to plot 3D plot.
R
1
# Get the library.
2
library(plotrix)
3

4
# Create data for the graph.
5
geeks <- c(23, 56, 20, 63)
6
labels <- c("Mumbai", "Pune", "Chennai", "Bangalore")
7

8
piepercent<- round(100 * geeks / sum(geeks), 1)
9

10
# Plot the chart.
11
pie3D(geeks, labels = piepercent,
12
main = "City pie chart", col = rainbow(length(geeks)))
13
legend("topright", c("Mumbai", "Pune", "Chennai", "Bangalore"),
14
cex = 0.5, fill = rainbow(length(geeks)))
Output:

R – Pie Charts

Looking to dive into the world of programming or sharpen your Python

skills? Our Master Python: Complete Beginner to Advanced Course is
your ultimate guide to becoming proficient in Python. This course covers
everything you need to build a solid foundation from fundamental
programming concepts to advanced techniques. With hands-on projects,
real-world examples, and expert guidance, you'll gain the confidence to
tackle complex coding challenges. Whether you're starting from scratch
or aiming to enhance your skills, this course is the perfect fit. Enroll now
and master Python, the language of the future!
10. Create a data set and do statistical analysis on the data using R.

 Courses
 Tutorials
 Data Science
 Practice



 Sign In
 Winter Tickets Sale!
 Data Visualization
 Statistics in R
 Machine Learning in R
 Data Science in R
 Packages in R
 Data Types
 String
 Array
 Vector
 Lists
 Matrices
 Oops in R
▲
 160 Days of DSA
 Share Your Experiences
 R Tutorial | Learn R Programming Language
Introduction
Fundamentals of R
Variables
Input/Output
Control Flow
Functions
Data Structures
Object Oriented Programming
Error Handling
File Handling
Packages in R
Data Interfaces
Data Visualization
Statistics
o R - Statistics
o Mean, Median and Mode in R Programming
o Exploring Statistical Measures in R: Average, Variance, and Standard
Deviation Explained
o Descriptive Analysis in R Programming
o Normal Distribution in R
o Binomial Distribution in R Programming
o ANOVA (Analysis of Variance) Test in R Programming
o Covariance and Correlation in R Programming
o Skewness in R Programming
o Hypothesis Testing in R Programming
o Bootstrapping in R Programming
o Time Series Analysis in R
Machine Learning
 DSA to DevelopmentCourse
R – Statistics
Last Updated : 12 Jul, 2024




Statistics is a form of mathematical analysis that concerns the collection,

organization, analysis, interpretation, and presentation of data. Statistical
analysis helps to make the best use of the vast data available and improves
the efficiency of solutions.
R – Statistics
R Programming Language is used for environment statistical computing and
graphics. The following is an introduction to basic R Statistics concepts
like normal distribution (bell curve), central tendency (the mean, median,
and mode), variability (25%, 50%, 75% quartiles), variance, standard
deviation, modality, and skewness.
Data Concepts
In R Statistics Data can be formed in different structures and different
formats, before starting the concepts of R Statistics we need to know the
data formats.
These are some formats:
 Vector
 Dataframe
 Variable
 Continuous Data
 Discrete Data
 Normal Data
 Categorical Data
 Normal Distribution
 Skewed Distribution
Statistics in R
 Average, Variance and Standard Deviation in R
 Mean, Median and Mode in R Programming
 Probability in R
o Discrete distributions
o Benford Distribution
o Bernoulli
o Binomial
o Hypergeometric distribution
o Geometric distribution
o Multinomial
o Negative binomial distribution
o Poisson distribution
o Zipf’s law
o Continuous distributions
o Beta distributions
o Dirichlet distributions
o Cauchy
o Chi-Square distribution
o Exponential
o Fisher-Snedecor
o Gamma
o Levy
o Log-normal distribution
o Normal and related distributions
o Pareto Distributions
o Student’s t distribution
o Uniform distribution
o Weibull
o Calculate Conditional Probability
o Binomial Distribution
o Normal Distribution in R
o Beta Distribution in R
 Hypothesis in R
 Types of Hypothesis
o Null Hypothesis
o Alternative Hypothesis
o One Sample T-Testing
o Two Sample T-Testing
o Paired Sample T-test
 Decision Errors in R
o Type I Error
o Type II Error
 Confidence Intervals
 Correlation and Covariance
 Covariance Matrix
 Pearson Correlation
 Normal Probability Plot
 Quantile Quantile plots
 Residuals Leverage Plot
 Spearman’s Rank Correlation Measure
 Kendall Rank Correlation Measure
 Evaluation Metrics – Accuracy, Precision, Recall, F1-Score, MAE, MSE
 Root-Mean-Square Error
 ROC and AUC curve
Plotting graphs in Statistics in R Programming
Language
Following is a list of functions that are required to plot graphs for the
representation of R Statistics data:
 plot() Function: This function is used to Draw a scatter plot with
axes and titles.
Syntax:
plot(x, y = NULL, ylim = NULL, xlim = NULL, type = “b”….)
data() function: This function is used to load specified data sets.
Syntax:
data(list = character(), [Link] = NULL, package = NULL…..)
 table() Function: The table function is used to build a contingency
table of the counts at each combination of factor levels in R
Statistics.
table(x, [Link] = NULL, ...)
barplot() Function: It creates a bar plot with vertical/horizontal

bars.
Syntax:
barplot(height, width = 1, [Link] = NULL, space = NULL…)
pie() Function: This function is used to create a pie chart.
Syntax:
pie(x, labels = names(x), radius = 0.6, edges = 100, clockwise = TRUE …)
hist() Function: The function hist() creates a histogram of the
given data values.
Syntax:
hist(x, breaks = “Sturges”, probability = !freq, freq = NULL,…)
Note: You can find the information about each function using the “?”
symbol before the beginning of each function.
R built-in datasets are very useful to start with and develop skills, So we will
be using a few Built-in datasets. Let’s start by creating a simple bar chart by
using chickwts dataset and learn how to use datasets and few functions of
RStudio for R Statistics.
Bar charts
A Bar chart represents categorical data with rectangular bars where the bars
can be plotted vertically or horizontally in R Statistics.
R
1
# ? is used before a function
2
# to get help on that function
3
?plot
4
?chickwts
5
data(chickwts) #loading data into workspace
6
plot(chickwts$feed) # plot feed from chickwts
Output:

R – Statistics

In the above code ‘?’ in front of a particular function means that it gives
information about that function with its syntax. In R ‘#’ is used for
commenting single line and there is no multiline comment in R Statistics.
Here we are using chickwts as the dataset and feed is the attribute in the
dataset.
Plots graph in decreasing order
Now we will plot graph in decreasing order in R Statistics.
R
1
feeds=table(chickwts$feed)
2

3
# plots graph in decreasing order
4
barplot(feeds[order(feeds, decreasing=TRUE)])
Output:
R – Statistics

Plots Horizontal bars

Now we will plot horizontal bars for visualization in R Statistics.
R
1
feeds = table(chickwts$feed)
2

3
# Set outside margins (bottom, left, top, right).
4
par(oma=c(1, 1, 1, 1))
5
par(mar=c(4, 5, 2, 1))
6

7
# Use las for the orientation of axis labels.
8
barplot(feeds[order(feeds, decreasing=TRUE)],
9
xlab="Number of chicks", las=1, col="yellow")
10

11
# Use horiz for bars to be shown as horizontal.
12
barplot(feeds[order(feeds)], horiz=TRUE,
13
xlab="Number of chicks", las=1, col="yellow")
Output:

R – Statistics

Pie charts
A pie chart is a circular statistical graph that is divided into slices to show
the different sizes of the data.
R
1
data("chickwts")
2

3
# main is used to create
4
# an heading for the chart
5
d = table(chickwts$feed)
6

7
pie(d[order(d, decreasing=TRUE)],
8
clockwise=TRUE,
9
main="Pie Chart of feeds from chichwits", )
Output:
R – Statistics

Histograms
Histograms are the representation of the distribution of data(numerical or
categorical). in R Statistics It is similar to a bar chart but it groups data in
terms of ranges.
R
1
# break is used for number of bins.
2
data(lynx)
3

4
# lynx is a built-in dataset.
5
lynx
6

7
# hist function is used to plot histogram.
8
hist(lynx)
9
hist(lynx, col="green",
10
main="Histogram of Annual Canadian Lynx Trappings")
Output :
Time Series:
Start = 1821
End = 1934
Frequency = 1
[1] 269 321 585 871 1475 2821 3928 5943 4950 2577 523 98 184
[14] 279 409 2285 2685 3409 1824 409 151 45 68 213 546 1033
[27] 2129 2536 957 361 377 225 360 731 1638 2725 2871 2119 684
[40] 299 236 245 552 1623 3311 6721 4254 687 255 473 358 784
[53] 1594 1676 2251 1426 756 299 201 229 469 736 2042 2811 4431
[66] 2511 389 73 39 49 59 188 377 1292 4031 3495 587 105
[79] 153 387 758 1307 3465 6991 6313 3794 1836 345 382 808 1388
[92] 2713 3800 3091 2985 3790 674 81 80 108 229 399 1132 2432
[105] 3574 2935 1537 529 485 662 1000 1590 2657 3396

R – Statistics

Plot The Distribution

Now we will plot R Statistics visualization distribution.
R
1
data(lynx)
2

3
# if freq=FALSE this will draw normal distribution
4
hist(lynx)
5
hist(lynx,col="green",
6
freq=FALSE ,main="Histogram of Annual Canadian Lynx Trappings")
7

8
curve(dnorm(x, mean=mean(lynx),
9
sd=sd(lynx)), col="red",
10
lwd=2, add=TRUE)
Output:

R – Statistics

Box Plots
Box Plot is a function for graphically depicting groups of numerical data
using quartiles. In R Statistics It represents the distribution of data and
understanding mean, median, and variance.
R
1
# USJudgeRatings is Built-in Dataset.
2
?USJudgeRatings
3

4
# ylim is used to specify the range.
5
boxplot(USJudgeRatings$RTEN, horizontal=TRUE,
6
xlab="Lawyers Rating", notch=TRUE,
7
ylim=c(0, 10), col="pink")
Output:

R – Statistics

11 Program to find factorial of the given number using recursive function

# Define the recursive function to calculate factorial

factorial_recursive <- function(n) {
if (n <= 1) {
return(1)
} else {
return(n * factorial_recursive(n - 1))
}
}

# Test the function with an example

result <- factorial_recursive(5)
print(result) # Output: 120
12 Write a R program to count the number of even and odd numbers from array of N numbers.

# Example usage
numbers <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
result <- count_even_odd(numbers)
cat("Even numbers:", result$even, "\n")
cat("Odd numbers:", result$odd, "\n")
MACHINE LEARNING LAB

1. Solving Regression & Classification using Decision Trees

 Courses
 Tutorials
 Data Science
 Practice



 Sign In
 Winter Tickets Sale!
 AI ML DS
 Data Science
 Data Analysis
 Data Visualization
 Machine Learning
 Deep Learning
 NLP
 Computer Vision
 Artificial Intelligence
 AI ML DS Interview Series
 AI ML DS Projects series
 Data Engineering
 Web Scrapping
▲
 160 Days of DSA
 Share Your Experiences
 Machine Learning Algorithms
 Top 15 Machine Learning Algorithms Every Data Scientist Should Know in 2024
Linear Model Regression
Linear Model Classification
Regularization
K-Nearest Neighbors (KNN)
Support Vector Machines
 ML | Stochastic Gradient Descent (SGD)
Decision Tree
o Major Kernel Functions in Support Vector Machine (SVM)
o CART (Classification And Regression Tree) in Machine Learning
o Decision Tree Classifiers in R Programming
o Python | Decision Tree Regression using sklearn
Ensemble Learning
Generative Model
Time Series Forecasting
Supervised Dimensionality Reduction Technique
Metrics for Classification & Regression Algorithms
Cross Validation Technique
Optimization Technique
Clustering
Association Rule Mining
Anomaly Detection
Dimensionality Reduction Technique
Model-Based Methods
Model-Free Methods
 Asynchronous Advantage Actor Critic (A3C) algorithm
 Machine Learning & Data ScienceCourse
Python | Decision Tree Regression using
sklearn
Last Updated : 11 Jan, 2023




Decision Tree is a decision-making tool that uses a flowchart-like tree

structure or is a model of decisions and all of their possible results, including
outcomes, input costs, and utility.
Decision-tree algorithm falls under the category of supervised learning
algorithms. It works for both continuous as well as categorical output
variables.
The branches/edges represent the result of the node and the nodes have
either:
1. Conditions [Decision Nodes]
2. Result [End Nodes]
The branches/edges represent the truth/falsity of the statement and take
makes a decision based on that in the example below which shows a
decision tree that evaluates the smallest of three numbers:
Decision Tree Regression:
Decision tree regression observes features of an object and trains a model in
the structure of a tree to predict data in the future to produce meaningful
continuous output. Continuous output means that the output/result is not
discrete, i.e., it is not represented just by a discrete, known set of numbers
or values.
Discrete output example: A weather prediction model that predicts
whether or not there’ll be rain on a particular day.
Continuous output example: A profit prediction model that states the
probable profit that can be generated from the sale of a product.
Here, continuous values are predicted with the help of a decision tree
regression model.
Let’s see the Step-by-Step implementation –
 Step 1: Import the required libraries.
 Python3

# import numpy package for arrays and stuff

import numpy as np
# import [Link] for plotting our result
import [Link] as plt

# import pandas for importing csv files

import pandas as pd

 Step 2: Initialize and print the Dataset.

 Python3

# import dataset
# dataset = pd.read_csv('[Link]')
# alternatively open up .csv file to read data

dataset = [Link](
[['Asset Flip', 100, 1000],
['Text Based', 500, 3000],
['Visual Novel', 1500, 5000],
['2D Pixel Art', 3500, 8000],
['2D Vector Art', 5000, 6500],
['Strategy', 6000, 7000],
['First Person Shooter', 8000, 15000],
['Simulator', 9500, 20000],
['Racing', 12000, 21000],
['RPG', 14000, 25000],
['Sandbox', 15500, 27000],
['Open-World', 16500, 30000],
['MMOFPS', 25000, 52000],
['MMORPG', 30000, 80000]
])

# print the dataset

print(dataset)

Output:
[['Asset Flip' '100' '1000']
['Text Based' '500' '3000']
['Visual Novel' '1500' '5000']
['2D Pixel Art' '3500' '8000']
['2D Vector Art' '5000' '6500']
['Strategy' '6000' '7000']
['First Person Shooter' '8000' '15000']
['Simulator' '9500' '20000']
['Racing' '12000' '21000']
['RPG' '14000' '25000']
['Sandbox' '15500' '27000']
['Open-World' '16500' '30000']
['MMOFPS' '25000' '52000']
['MMORPG' '30000' '80000']]
 Step 3: Select all the rows and column 1 from the dataset to “X”.
 Python3

# select all rows by : and column 1

# by 1:2 representing features
X = dataset[:, 1:2].astype(int)

# print X
print(X)

Output:
[[ 100]
[ 500]
[ 1500]
[ 3500]
[ 5000]
[ 6000]
[ 8000]
[ 9500]
[12000]
[14000]
[15500]
[16500]
[25000]
[30000]]
 Step 4: Select all of the rows and column 2 from the dataset to “y”.
 Python3

# select all rows by : and column 2

# by 2 to Y representing labels
y = dataset[:, 2].astype(int)

# print y
print(y)

Output:
[ 1000 3000 5000 8000 6500 7000 15000 20000 21000 25000 27000 30000
52000 80000]
 Step 5: Fit decision tree regressor to the dataset
 Python3
# import the regressor
from [Link] import DecisionTreeRegressor

# create a regressor object

regressor = DecisionTreeRegressor(random_state = 0)

# fit the regressor with X and Y data

[Link](X, y)

Output:
DecisionTreeRegressor(ccp_alpha=0.0, criterion='mse', max_depth=None,
max_features=None, max_leaf_nodes=None,
min_impurity_decrease=0.0,
min_impurity_split=None,
min_samples_leaf=1, min_samples_split=2,
min_weight_fraction_leaf=0.0,
presort='deprecated',
random_state=0, splitter='best')
 Step 6: Predicting a new value
 Python3

# predicting a new value

# test the output by changing values, like 3750

y_pred = [Link]([[3750]])

# print the predicted price

print("Predicted price: % d\n"% y_pred)

Output:
Predicted price: 8000
 Step 7: Visualising the result
 Python3

# arange for creating a range of values

# from min value of X to max value of X
# with a difference of 0.01 between two
# consecutive values
X_grid = [Link](min(X), max(X), 0.01)

# reshape for reshaping the data into

# a len(X_grid)*1 array, i.e. to make
# a column out of the X_grid values
X_grid = X_grid.reshape((len(X_grid), 1))

# scatter plot for original data

[Link](X, y, color = 'red')

# plot predicted data

[Link](X_grid, [Link](X_grid), color = 'blue')

# specify title
[Link]('Profit to Production Cost (Decision Tree Regression)')

# specify X axis label

[Link]('Production Cost')

# specify Y axis label

[Link]('Profit')

# show the plot

[Link]()

 Step 8: The tree is finally exported and shown in the TREE

STRUCTURE below, visualized using [Link]
by copying the data from the ‘[Link]’ file.
 Python3

# import export_graphviz
from [Link] import export_graphviz

# export the decision tree to a [Link] file

# for visualizing the plot easily anywhere
export_graphviz(regressor, out_file ='[Link]',
feature_names =['Production Cost'])

Output (Decision Tree):

3. Root Node Attribute Selection for Decision Trees using Information Gain

Decision Tree in Machine Learning

Last Updated : 15 Mar, 2024




A decision tree in machine learning is a versatile, interpretable algorithm

used for predictive modelling. It structures decisions based on input data,
making it suitable for both classification and regression tasks. This article
delves into the components, terminologies, construction, and advantages of
decision trees, exploring their applications and learning algorithms.
Decision Tree in Machine Learning
A decision tree is a type of supervised learning algorithm that is commonly
used in machine learning to model and predict outcomes based on input
data. It is a tree-like structure where each internal node tests on attribute,
each branch corresponds to attribute value and each leaf node represents
the final decision or prediction. The decision tree algorithm falls under the
category of supervised learning. They can be used to solve
both regression and classification problems.
Decision Tree Terminologies
There are specialized terms associated with decision trees that denote
various components and facets of the tree structure and decision-making
procedure. :
 Root Node: A decision tree’s root node, which represents the
original choice or feature from which the tree branches, is the
highest node.
 Internal Nodes (Decision Nodes): Nodes in the tree whose
choices are determined by the values of particular attributes. There
are branches on these nodes that go to other nodes.
 Leaf Nodes (Terminal Nodes): The branches’ termini, when
choices or forecasts are decided upon. There are no more branches
on leaf nodes.
 Branches (Edges): Links between nodes that show how decisions
are made in response to particular circumstances.
 Splitting: The process of dividing a node into two or more sub-
nodes based on a decision criterion. It involves selecting a feature
and a threshold to create subsets of data.
 Parent Node: A node that is split into child nodes. The original node
from which a split originates.
 Child Node: Nodes created as a result of a split from a parent node.
 Decision Criterion: The rule or condition used to determine how
the data should be split at a decision node. It involves comparing
feature values against a threshold.
 Pruning: The process of removing branches or nodes from a
decision tree to improve its generalisation and prevent overfitting.
Understanding these terminologies is crucial for interpreting and working
with decision trees in machine learning applications.
How Decision Tree is formed?
The process of forming a decision tree involves recursively partitioning the
data based on the values of different attributes. The algorithm selects the
best attribute to split the data at each internal node, based on certain
criteria such as information gain or Gini impurity. This splitting process
continues until a stopping criterion is met, such as reaching a maximum
depth or having a minimum number of instances in a leaf node.
Why Decision Tree?
Decision trees are widely used in machine learning for a number of reasons:
 Decision trees are so versatile in simulating intricate decision-
making processes, because of their interpretability and versatility.
 Their portrayal of complex choice scenarios that take into account a
variety of causes and outcomes is made possible by their
hierarchical structure.
 They provide comprehensible insights into the decision logic,
decision trees are especially helpful for tasks involving
categorisation and regression.
 They are proficient with both numerical and categorical data, and
they can easily adapt to a variety of datasets thanks to their
autonomous feature selection capability.
 Decision trees also provide simple visualization, which helps to
comprehend and elucidate the underlying decision processes in a
model.
Decision Tree Approach
Decision tree uses the tree representation to solve the problem in which
each leaf node corresponds to a class label and attributes are represented
on the internal node of the tree. We can represent any boolean function on
discrete attributes using the decision tree.

Below are some assumptions that we made while using the decision tree:
At the beginning, we consider the whole training set as the root.
 Feature values are preferred to be categorical. If the values are
continuous then they are discretized prior to building the model.
 On the basis of attribute values, records are distributed recursively.
 We use statistical methods for ordering attributes as root or the
internal node.
As you can see from the above image the Decision Tree works on the Sum
of Product form which is also known as Disjunctive Normal Form. In the
above image, we are predicting the use of computer in the daily life of
people. In the Decision Tree, the major challenge is the identification of the
attribute for the root node at each level. This process is known as attribute
selection. We have two popular attribute selection measures:
1. Information Gain
2. Gini Index
1. Information Gain:
When we use a node in a decision tree to partition the training instances into
smaller subsets the entropy changes. Information gain is a measure of this
change in entropy.
 Suppose S is a set of instances,
 A is an attribute
 Sv is the subset of S
 v represents an individual value that the attribute A can take and
Values (A) is the set of all possible values of A, then
Gain(S,A)=Entropy(S)–∑vA∣Sv∣∣S∣.Entropy(Sv)Gain(S,A)=Entropy(S)–∑vA∣S∣∣Sv∣
.Entropy(Sv)
Entropy: is the measure of uncertainty of a random variable, it
characterizes the impurity of an arbitrary collection of examples. The higher
the entropy more the information content.
Suppose S is a set of instances, A is an attribute, S v is the subset of S with A
= v, and Values (A) is the set of all possible values of A, then
Gain(S,A)=Entropy(S)–∑vϵValues(A)∣Sv∣∣S∣.Entropy(Sv) Gain(S,A)=Entropy(S)–∑vϵValues(A)∣S∣∣Sv∣
.Entropy(Sv)
Example:
For the set X = {a,a,a,b,b,b,b,b}
Total instances: 8
Instances of b: 5
Instances of a: 3

Entropy H(X)=[(38)log⁡238+(58)log⁡258]=−[0.375(−1.415)+0.625(−0.678)]=−
(−0.53−0.424)=0.954Entropy H(X)=[(83)log283+(85)log285]=−[0.375(−1.415)+0.625(−0.678)]=−
(−0.53−0.424)=0.954
Building Decision Tree using Information Gain The essentials:
 Start with all training instances associated with the root node
 Use info gain to choose which attribute to label each node with
 Note: No root-to-leaf path should contain the same discrete attribute
twice
 Recursively construct each subtree on the subset of training
instances that would be classified down that path in the tree.
 If all positive or all negative training instances remain, the label that
node “yes” or “no” accordingly
 If no attributes remain, label with a majority vote of training
instances left at that node
 If no instances remain, label with a majority vote of the parent’s
training instances.
Example: Now, let us draw a Decision Tree for the following data using
Information gain. Training set: 3 features and 2 classes
X Y Z C

1 1 1 I

1 1 0 I

0 0 1 II

1 0 0 II

Here, we have 3 features and 2 output classes. To build a decision tree using
Information gain. We will take each of the features and calculate the
information for each feature.

Split on feature X

Split on feature Y
Split on feature Z
From the above images, we can see that the information gain is maximum
when we make a split on feature Y. So, for the root node best-suited feature
is feature Y. Now we can see that while splitting the dataset by feature Y,
the child contains a pure subset of the target variable. So we don’t need to
further split the dataset. The final tree for the above dataset would look like
this:

2. Gini Index
 Gini Index is a metric to measure how often a randomly chosen
element would be incorrectly identified.
 It means an attribute with a lower Gini index should be preferred.
 Sklearn supports “Gini” criteria for Gini Index and by default, it takes
“gini” value.
 The Formula for the calculation of the Gini Index is given below.
The Formula for Gini Index is given by :
Gini Impurity

The Gini Index is a measure of the inequality or impurity of a distribution,

commonly used in decision trees and other machine learning algorithms. It
ranges from 0 to 0.5, where 0 indicates a pure set (all instances belong to
the same class), and 0.5 indicates a maximally impure set (instances are
evenly distributed across classes).
Some additional features and characteristics of the Gini Index are:
 It is calculated by summing the squared probabilities of each
outcome in a distribution and subtracting the result from 1.
 A lower Gini Index indicates a more homogeneous or pure
distribution, while a higher Gini Index indicates a more
heterogeneous or impure distribution.
 In decision trees, the Gini Index is used to evaluate the quality of a
split by measuring the difference between the impurity of the parent
node and the weighted impurity of the child nodes.
 Compared to other impurity measures like entropy, the Gini Index is
faster to compute and more sensitive to changes in class
probabilities.
 One disadvantage of the Gini Index is that it tends to favour splits
that create equally sized child nodes, even if they are not optimal for
classification accuracy.
 In practice, the choice between using the Gini Index or other
impurity measures depends on the specific problem and dataset,
and often requires experimentation and tuning.
Example of a Decision Tree Algorithm
Forecasting Activities Using Weather Information
 Root node: Whole dataset
 Attribute : “Outlook” (sunny, cloudy, rainy).
 Subsets: Overcast, Rainy, and Sunny.
 Recursive Splitting: Divide the sunny subset even more according
to humidity, for example.
 Leaf Nodes: Activities include “swimming,” “hiking,” and “staying
inside.”
Beginning with the entire dataset as the root node of the decision
tree:
 Determine the best attribute to split the dataset based on
information gain, which is calculated by the formula: Information
gain = Entropy(parent) – [Weighted average] * Entropy(children),
where entropy is a measure of impurity or disorder of a set of
examples, and the weighted average is based on the number of
examples in each child node.
 Create a new internal node that corresponds to the best attribute
and connects it to the root node. For example, if the best attribute is
“outlook” (which can have values “sunny”, “overcast”, or “rainy”),
we create a new node labeled “outlook” and connect it to the root
node.
 Partition the dataset into subsets based on the values of the best
attribute. For example, we create three subsets: one for instances
where the outlook is “sunny”, one for instances where the outlook is
“overcast”, and one for instances where the outlook is “rainy”.
 Recursively repeat steps 1-4 for each subset until all instances in a
given subset belong to the same class or no further splitting is
possible. For example, if the subset of instances where the outlook is
“overcast” contains only instances where the activity is “hiking”, we
assign a leaf node labeled “hiking” to this subset. If the subset of
instances where the outlook is “sunny” is further split based on the
humidity attribute, we repeat steps 2-4 for this subset.
 Assign a leaf node to each subset that contains instances that
belong to the same class. For example, if the subset of instances
where the outlook is “rainy” contains only instances where the
activity is “stay inside”, we assign a leaf node labeled “stay inside”
to this subset.
 Make predictions based on the decision tree by traversing it from
the root node to a leaf node that corresponds to the instance being
classified. For example, if the outlook is “sunny” and the humidity is
“high”, we traverse the decision tree by following the “sunny”
branch and then the “high humidity” branch, and we end up at a
leaf node labeled “swimming”, which is our predicted activity.
Advantages of Decision Tree
 Easy to understand and interpret, making them accessible to non-
experts.
 Handle both numerical and categorical data without requiring
extensive preprocessing.
 Provides insights into feature importance for decision-making.
 Handle missing values and outliers without significant impact.
 Applicable to both classification and regression tasks.
Disadvantages of Decision Tree
 Disadvantages include the potential for overfitting
 Sensitivity to small changes in data, limited generalization if training
data is not representative
 Potential bias in the presence of imbalanced data.
Conclusion
Decision trees, a key tool in machine learning, model and predict outcomes
based on input data through a tree-like structure. They offer interpretability,
versatility, and simple visualization, making them valuable for both
categorization and regression tasks. While decision trees have advantages
like ease of understanding, they may face challenges such as overfitting.
Understanding their terminologies and formation process is essential for
effective application in diverse scenarios.
Frequently Asked Questions (FAQs)
1. What are the major issues in decision tree learning?
Major issues in decision tree learning include overfitting, sensitivity to small
data changes, and limited generalization. Ensuring proper pruning, tuning,
and handling imbalanced data can help mitigate these challenges for more
robust decision tree models.
2. How does decision tree help in decision making?
Decision trees aid decision-making by representing complex choices in a
hierarchical structure. Each node tests specific attributes, guiding decisions
based on data values. Leaf nodes provide final outcomes, offering a clear
and interpretable path for decision analysis in machine learning.
3. What is the maximum depth of a decision tree?
The maximum depth of a decision tree is a hyperparameter that determines
the maximum number of levels or nodes from the root to any leaf. It
controls the complexity of the tree and helps prevent overfitting.
4. What is the concept of decision tree?
A decision tree is a supervised learning algorithm that models decisions
based on input features. It forms a tree-like structure where each internal
node represents a decision based on an attribute, leading to leaf nodes
representing outcomes.
5. What is entropy in decision tree?
In decision trees, entropy is a measure of impurity or disorder within a
dataset. It quantifies the uncertainty associated with classifying instances,
guiding the algorithm to make informative splits for effective decision-
making.
6. What are the Hyperparameters of decision tree?
1. Max Depth: Maximum depth of the tree.
2. Min Samples Split: Minimum samples required to split an internal
node.
3. Min Samples Leaf: Minimum samples required in a leaf node.
4. Criterion: The function used to measure the quality of a split

3. Bayesian Inference in Gene Expression Analysis

[Link]
ts as sts
import numpy as np
import [Link] as plt

mu = [Link](1.65, 1.8, num = 50)

test = [Link](0, 2)
uniform_dist = [Link](mu) + 1 #sneaky advanced note: I'm using the
uniform distribution for clarity, but we can also make the beta
distribution look completely flat by tweaking alpha and beta!
uniform_dist = uniform_dist/uniform_dist.sum() #Normalizing the
distribution to make the probability densities sum into 1
beta_dist = [Link](mu, 2, 5, loc = 1.65, scale = 0.2)
beta_dist = beta_dist/beta_dist.sum()
[Link](mu, beta_dist, label = 'Beta Dist')
[Link](mu, uniform_dist, label = 'Uniform Dist')
[Link]("Value of $\mu$ in meters")
[Link]("Probability density")
[Link]()

4. Pattern Recognition Application using Bayesian Inference

import numpy as np
from sklearn.linear_model import BayesianRidge

# Generate some data

X = [Link](100, 10)
y = [Link](100)

# Initialize the Bayesian Ridge model

br = BayesianRidge()

# Fit the model to the data

[Link](X, y)

# Make predictions on new data

new_X = [Link](50, 10)
predictions = [Link](new_X)

print(predictions)

5. Bagging in Classification

import numpy as np
from sklearn import datasets
from sklearn.model_selection import train_test_split
from [Link] import StandardScaler
from sklearn.linear_model import LogisticRegression
from [Link] import make_pipeline
from [Link] import BaggingClassifier
from sklearn.model_selection import GridSearchCV
#
# Load the breast cancer dataset
#
bc = datasets.load_breast_cancer()
X = [Link]
y = [Link]
#
# Create training and test split
#
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25,
random_state=1, stratify=y)
#
# Pipeline Estimator
#
pipeline = make_pipeline(StandardScaler(),
LogisticRegression(random_state=1))
#
# Fit the model
#
[Link](X_train, y_train)
#
# Model scores on test and training data
#
print('Model test Score: %.3f, ' %[Link](X_test, y_test),
'Model training Score: %.3f' %[Link](X_train, y_train))

6. Bagging, Boosting applications using Regression Trees

Explore Number of Trees

An important hyperparameter for the Bagging algorithm is the number of decision trees used in the
ensemble.

Typically, the number of trees is increased until the model performance stabilizes. Intuition might
suggest that more trees will lead to overfitting, although this is not the case. Bagging and related
ensemble of decision trees algorithms (like random forest) appear to be somewhat immune to
overfitting the training dataset given the stochastic nature of the learning algorithm.
The number of trees can be set via the “n_estimators” argument and defaults to 100.
The example below explores the effect of the number of trees with values between 10 to 5,000.

1 # explore bagging ensemble number of trees effect on performance

2 from numpy import mean
3 from numpy import std
4 from [Link] import make_classification
5 from sklearn.model_selection import cross_val_score
6 from sklearn.model_selection import RepeatedStratifiedKFold
7 from [Link] import BaggingClassifier
8 from matplotlib import pyplot
9
1 # get the dataset
0 def get_dataset():
1 X, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5,
1 random_state=5)
1 return X, y
2
1 # get a list of models to evaluate
3 def get_models():
1 models = dict()
4 # define number of trees to consider
1 n_trees = [10, 50, 100, 500, 500, 1000, 5000]
5 for n in n_trees:
1 models[str(n)] = BaggingClassifier(n_estimators=n)
6 return models
1
7 # evaluate a given model using cross-validation
1 def evaluate_model(model, X, y):
8 # define the evaluation procedure
1 cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
9 # evaluate the model and collect the results
2 scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)
0 return scores
2
1 # define dataset
2 X, y = get_dataset()
2 # get the models to evaluate
2 models = get_models()
3 # evaluate the models and store results
2 results, names = list(), list()
4 for name, model in [Link]():
2 # evaluate the model
5 scores = evaluate_model(model, X, y)
2 # store the results
6 [Link](scores)
2 [Link](name)
7 # summarize the performance along the way
2 print('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))
8 # plot model performance for comparison
2 [Link](results, labels=names, showmeans=True)
9 [Link]()
3
0
3
1
3
2
3
3
3
4
3
5
3
6
3
7
3
8
3
9
4
0
4
1
4
2
4
3
4
4
4
5
4
6
4
7
4
8
Running the example first reports the mean accuracy for each configured number of decision trees.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or
differences in numerical precision. Consider running the example a few times and compare the
average outcome.
In this case, we can see that that performance improves on this dataset until about 100 trees and
remains flat after that.

1>10 0.855 (0.037)

2>50 0.876 (0.035)
3>100 0.882 (0.037)
4>500 0.885 (0.041)
5>1000 0.885 (0.037)
6>5000 0.885 (0.038)
A box and whisker plot is created for the distribution of accuracy scores for each configured number of
trees.

We can see the general trend of no further improvement beyond about 100 trees.
7. Data & Text Classification using Neural Networks

# importing the necessary libraries

import numpy as np

from [Link] import Sequential

from [Link] import Embedding, Conv1D, GlobalMaxPooling1D, Dense

from [Link] import pad_sequences

from [Link] import imdb

from [Link] import accuracy_score, precision_score, recall_score, f1_score

# Setting up the parameters

maximum_features = 5000 # Maximum number of words to consider as features

maximum_length = 100 # Maximum length of input sequences

word_embedding_dims = 50 # Dimension of word embeddings

no_of_filters = 250 # Number of filters in the convolutional layer

kernel_size = 3 # Size of the convolutional filters

hidden_dims = 250 # Number of neurons in the hidden layer

batch_size = 32 # Batch size for training

epochs = 2 # Number of training epochs

threshold = 0.5 # Threshold for binary classification

# Loading the IMDB dataset

(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=maximum_features)

# Padding the sequences to ensure uniform length

x_train = pad_sequences(x_train, maxlen=maximum_length)

x_test = pad_sequences(x_test, maxlen=maximum_length)

# Building the model

model = Sequential()

# Adding the embedding layer to convert input sequences to dense vectors

[Link](Embedding(maximum_features, word_embedding_dims,

input_length=maximum_length))
# Adding the 1D convolutional layer with ReLU activation

[Link](Conv1D(no_of_filters, kernel_size, padding='valid',

activation='relu', strides=1))

# Adding the global max pooling layer to reduce dimensionality

[Link](GlobalMaxPooling1D())

# Adding the dense hidden layer with ReLU activation

[Link](Dense(hidden_dims, activation='relu'))

# Adding the output layer with sigmoid activation for binary classification

[Link](Dense(1, activation='sigmoid'))

# Compiling the model with binary cross-entropy loss and Adam optimizer

[Link](loss='binary_crossentropy',

optimizer='adam', metrics=['accuracy'])

# Training the model

[Link](x_train, y_train, batch_size=batch_size,

epochs=epochs, validation_data=(x_test, y_test))

# Predicting the probabilities for test data

y_pred_prob = [Link](x_test)

# Converting the probabilities to binary classes based on threshold

y_pred = (y_pred_prob > threshold).astype(int)

# Calculating the evaluation metrics

accuracy = accuracy_score(y_test, y_pred)

precision = precision_score(y_test, y_pred)

recall = recall_score(y_test, y_pred)

f1 = f1_score(y_test, y_pred)
# Printing the evaluation metrics

print('Accuracy:', accuracy)

print('Precision:', precision)

print('Recall:', recall)

print('F1-score:', f1)

Epoch 1/2
782/782 [==============================] - 7s 8ms/step - loss: 0.4245 -
accuracy: 0.7927 - val_loss: 0.3713 - val_accuracy: 0.8320
Epoch 2/2
782/782 [==============================] - 7s 9ms/step - loss: 0.2521 -
accuracy: 0.8971 - val_loss: 0.3251 - val_accuracy: 0.8583
782/782 [==============================] - 2s 2ms/step
Accuracy: 0.85832
Precision: 0.8426931905126244
Recall: 0.88112
F1-score: 0.8614782948768088

8. Using Weka tool for SVM classification for chosen domain application

Classifying data using Support Vector

Machines(SVMs) in Python
Last Updated : 01 Sep, 2023




Introduction to SVMs: In machine learning, support vector machines

(SVMs, also support vector networks) are supervised learning models with
associated learning algorithms that analyze data used for classification and
regression analysis. A Support Vector Machine (SVM) is a discriminative
classifier formally defined by a separating hyperplane. In other words, given
labeled training data (supervised learning), the algorithm outputs an optimal
hyperplane which categorizes new examples.
What is Support Vector Machine?
An SVM model is a representation of the examples as points in space,
mapped so that the examples of the separate categories are divided by a
clear gap that is as wide as possible. In addition to performing linear
classification, SVMs can efficiently perform a non-linear classification,
implicitly mapping their inputs into high-dimensional feature spaces.
What does SVM do?
Given a set of training examples, each marked as belonging to one or the
other of two categories, an SVM training algorithm builds a model that
assigns new examples to one category or the other, making it a non-
probabilistic binary linear classifier. Let you have basic understandings from
this article before you proceed further. Here I’ll discuss an example about
SVM classification of cancer UCI datasets using machine learning tools i.e.
scikit-learn compatible with Python. Pre-
requisites: Numpy, Pandas, matplot-lib, scikit-learn Let’s have a quick
example of support vector classification. First we need to create a dataset:
 python3

# importing scikit learn with make_blobs

from [Link] import make_blobs

# creating datasets X containing n_samples

# Y containing two classes
X, Y = make_blobs(n_samples=500, centers=2,
random_state=0, cluster_std=0.40)
import [Link] as plt
# plotting scatters
[Link](X[:, 0], X[:, 1], c=Y, s=50, cmap='spring');
[Link]()

Output: What Support vector machines do,

is to not only draw a line between two classes here, but consider a region
about the line of some given width. Here’s an example of what it can look
like:
 python3

# creating linspace between -1 to 3.5

xfit = [Link](-1, 3.5)

# plotting scatter
[Link](X[:, 0], X[:, 1], c=Y, s=50, cmap='spring')

# plot a line between the different sets of data

for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:
yfit = m * xfit + b
[Link](xfit, yfit, '-k')
plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='none',
color='#AAAAAA', alpha=0.4)

[Link](-1, 3.5);
[Link]()

Importing datasets
This is the intuition of support vector machines, which optimize a linear
discriminant model representing the perpendicular distance between the
datasets. Now let’s train the classifier using our training data. Before
training, we need to import cancer datasets as csv file where we will train
two features out of all features.
 python3

# importing required libraries

import numpy as np
import pandas as pd
import [Link] as plt

# reading csv file and extracting class column to y.

x = pd.read_csv("C:\...\[Link]")
a = [Link](x)
y = a[:,30] # classes having 0 and 1

# extracting two features

x = np.column_stack(([Link],[Link]))

# 569 samples and 2 features

[Link]

print (x),(y)

[[ 122.8 1001. ]
[ 132.9 1326. ]
[ 130. 1203. ]
...,
[ 108.3 858.1 ]
[ 140.1 1265. ]
[ 47.92 181. ]]

array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 1., 1., 1., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.,
0., 0., 0., 0., 0., 0., 0., 1., 0., 1., 1., 1., 1.,
1., 0., 0., 1., 0., 0., 1., 1., 1., 1., 0.,
1., ....,
1.])
Fitting a Support Vector Machine
Now we’ll fit a Support Vector Machine Classifier to these points. While the
mathematical details of the likelihood model are interesting, we’ll let read
about those elsewhere. Instead, we’ll just treat the scikit-learn algorithm as
a black box which accomplishes the above task.
 python3

# import support vector classifier

# "Support Vector Classifier"
from [Link] import SVC
clf = SVC(kernel='linear')

# fitting x samples and y classes

[Link](x, y)

After being fitted, the model can then be used to predict new values:
 python3

[Link]([[120, 990]])

[Link]([[85, 550]])

array([ 0.])
array([ 1.])
Let’s have a look on the graph how does this show.

This is obtained by analyzing the data

taken and pre-processing methods to make optimal hyperplanes using
matplotlib func If you like GeeksforGeeks and would like to contribute, you
can also write an article using [Link] or mail your article to
review-team@[Link]. See your article appearing on the
GeeksforGeeks main page and help other Geeks.

9. Data & Text Clustering using K-means algorithm

Create a K-Means Clustering

Algorithm from Scratch in Python
Introduction
k-means clustering is an unsupervised machine learning algorithm
that seeks to segment a dataset into groups based on the similarity
of datapoints. An unsupervised model has independent
variables and no dependent variables.

Suppose you have a dataset of 2-dimensional scalar attributes:

Image by author.

If the points in this dataset belong to distinct groups with attributes

significantly varying between groups but not within, the points
should form clusters when plotted.
Image by author.

Figure 1: A dataset of points with groups of distinct attributes.

This dataset clearly displays 3 distinct classes of data. If we seek to

assign a new data point to one of these three groups, it can be done
by finding the midpoint of each group (centroid) and selecting the
nearest centroid as the group of the unassigned data point.
Image by author.

Figure 2: The data points are segmented into groups denoted with
differing colors.

Algorithm
For a given dataset, k is specified to be the number of distinct
groups the points belong to. These k centroids are first randomly
initialized, then iterations are performed to optimize the locations of
these k centroids as follows:

1. The distance from each point to each centroid is

calculated.

2. Points are assigned to their nearest centroid.

3. Centroids are shifted to be the average value of the points
belonging to it. If the centroids did not move, the
algorithm is finished, else repeat.

Data
To evaluate our algorithm, we’ll first generate a dataset of groups in
2-dimensional space. The [Link] function make_blobs
creates groupings of 2-dimensional normal distributions, and
assigns a label corresponding to the group said point belongs to.
import seaborn as sns
from [Link] import make_blobs
import [Link] as plt
from [Link] import StandardScalercenters = 5
X_train, true_labels = make_blobs(n_samples=100, centers=centers,
random_state=42)
X_train = StandardScaler().fit_transform(X_train)[Link](x=[X[0]
for X in X_train],
y=[X[1] for X in X_train],
hue=true_labels,
palette="deep",
legend=None
)[Link]("x")
[Link]("y")
[Link]()
Image by author.

Figure 3: The dataset we will use to evaluate our k means

clustering model.

This dataset provides a unique demonstration of the k-means

algorithm. Observe the orange point uncharacteristically far from its
center, and directly in the cluster of purple data points. This point
cannot be accurately classified as belonging to the right group, thus
even if our algorithm works well it should incorrectly characterize it
as a member of the purple group.

Model Creation

Helper Functions
We’ll need to calculate the distances between a point and a dataset
of points multiple times in this algorithm. To do so, lets define a
function that calculates Euclidean distances.
def euclidean(point, data):
"""
Euclidean distance between point & data.
Point has dimensions (m,), data has dimensions (n,m), and output will
be of size (n,).
"""
return [Link]([Link]((point - data)**2, axis=1))

Implementation
First, the k-means clustering algorithm is initialized with a value for
k and a maximum number of iterations for finding the optimal
centroid locations. If a maximum number of iterations is not
considered when optimizing centroid locations, there is a risk of
running an infinite loop.
class KMeans: def __init__(self, n_clusters=8, max_iter=300):
self.n_clusters = n_clusters
self.max_iter = max_iter

Now, the bulk of the algorithm is performed when fitting the model
to a training dataset.

First we’ll initialize the centroids randomly in the domain of the test
dataset, with a uniform distribution.
# Randomly select centroid start points, uniformly distributed across the
domain of the dataset
min_, max_ = [Link](X_train, axis=0), [Link](X_train, axis=0)
[Link] = [uniform(min_, max_) for _ in range(self.n_clusters)]

Next, we perform the iterative process of optimizing the centroid

locations.

The optimization process is to readjust the centroid locations to be

the means of the points belonging to it. This process is to repeat
until the centroids stop moving, or the maximum number of
iterations is passed. We’ll use a while loop to account for the fact
that this process does not have a fixed number of iterations.
Additionally, you could also use a for loop that repeats max_iter
times and breaks when the centroids stop changing.

Before beginning the while loop, we’ll initialize the variables used in
the exit conditions.
iteration = 0
prev_centroids = None

Now, we begin the loop. We’ll iterate through the data points in the
training set, assigning them to an initialized empty list of lists. The
sorted_points list contains one empty list for each centroid, where
data points are appended once they’ve been assigned.
while np.not_equal([Link], prev_centroids).any() and iteration <
self.max_iter:
# Sort each data point, assigning to nearest centroid
sorted_points = [[] for _ in range(self.n_clusters)]
for x in X_train:
dists = euclidean(x, [Link])
centroid_idx = [Link](dists)
sorted_points[centroid_idx].append(x)

Now that we’ve assigned the whole training dataset to their closest
centroids, we can update the location of the centroids and finish the
iteration.
# Push current centroids to previous, reassign centroids as mean of the
points belonging to them
prev_centroids = [Link]
[Link] = [[Link](cluster, axis=0) for cluster in
sorted_points]
for i, centroid in enumerate([Link]):
if [Link](centroid).any(): # Catch any [Link], resulting from a
centroid having no points
[Link][i] = prev_centroids[i]
iteration += 1

After the completion of the iteration, the while conditions are

checked again, and the algorithm will continue until the centroids
are optimized or the max iterations are passed. The full fit method is
included below.
class KMeans: def __init__(self, n_clusters=8, max_iter=300):
self.n_clusters = n_clusters
self.max_iter = max_iter def fit(self, X_train): #
Randomly select centroid start points, uniformly distributed across the
domain of the dataset
min_, max_ = [Link](X_train, axis=0), [Link](X_train, axis=0)
[Link] = [uniform(min_, max_) for _ in
range(self.n_clusters)] # Iterate, adjusting centroids until
converged or until passed max_iter
iteration = 0
prev_centroids = None
while np.not_equal([Link], prev_centroids).any() and
iteration < self.max_iter:
# Sort each datapoint, assigning to nearest centroid
sorted_points = [[] for _ in range(self.n_clusters)]
for x in X_train:
dists = euclidean(x, [Link])
centroid_idx = [Link](dists)
sorted_points[centroid_idx].append(x) # Push
current centroids to previous, reassign centroids as mean of the points
belonging to them
prev_centroids = [Link]
[Link] = [[Link](cluster, axis=0) for cluster in
sorted_points]
for i, centroid in enumerate([Link]):
if [Link](centroid).any(): # Catch any [Link],
resulting from a centroid having no points
[Link][i] = prev_centroids[i]
iteration += 1

Lastly, lets make a method to evaluate a set of points to the

centroids we’ve optimized to our training set. This method returns
the centroid and the index of said centroid for each point.
def evaluate(self, X):
centroids = []
centroid_idxs = []
for x in X:
dists = euclidean(x, [Link])
centroid_idx = [Link](dists)
[Link]([Link][centroid_idx])
centroid_idxs.append(centroid_idx) return centroids,
centroid_idx

First Model Evaluation

Now we can finally deploy our model. Lets train and test it on our
original dataset and see the results. We’ll keep our original method
of plotting our data, by separating the true labels by color, but now
we’ll additionally separate the predicted labels by marker style, to
see how the model performs.
kmeans = KMeans(n_clusters=centers)
[Link](X_train)# View results
class_centers, classification = [Link](X_train)
[Link](x=[X[0] for X in X_train],
y=[X[1] for X in X_train],
hue=true_labels,
style=classification,
palette="deep",
legend=None
)
[Link]([x for x, _ in [Link]],
[y for _, y in [Link]],
'+',
markersize=10,
)[Link]()

Image by author.

Figure 4: A failed example where one centroid has no points, and

one contains two clusters.
Image by author.

Figure 5: A failed example where one centroid has no points, two

contains two clusters, and two split one cluster.
Image by author.

Figure 6: A failed example where two centroids contain one and a

half clusters, and two centroids split a cluster.

Re-evaluating Centroid Initialization

Looks like our model isn’t performing very well. We can infer two
primary problems from these three failed examples.

1. If a centroid is initialized far from any groups, it is unlikely

to move. (Example: the bottom right centroid in Figure
4.)

2. If centroids are initialized too close, they’re unlikely to

diverge from one another. (Example: the two centroids in
the green group in Figure 6.)
We’ll begin to remedy these problems with a new process of
initializing the centroid locations. This new method is referred to as
the k-means++ algorithm.

1. Initialize the first centroid as a random selection of one of

the data points.

2. Calculate the sum of the distances between each data

point and all the centroids.

3. Select the next centroid randomly, with a probability

proportional to the total distance to the centroids.

4. Return to step 2. Repeat until all centroids have been

initialized.

This code is included below.

# Initialize the centroids, using the "k-means++" method, where a random
datapoint is selected as the first,
# then the rest are initialized w/ probabilities proportional to their
distances to the first
# Pick a random point from train data for first centroid
[Link] = [[Link](X_train)]for _ in range(self.n_clusters-1):
# Calculate distances from points to the centroids
dists = [Link]([euclidean(centroid, X_train) for centroid in
[Link]], axis=0)
# Normalize the distances
dists /= [Link](dists)
# Choose remaining points based on their distances
new_centroid_idx, = [Link](range(len(X_train)), size=1,
p=dists)
[Link] += [X_train[new_centroid_idx]]

If we run this new model a few times we’ll see it performs much
better, but still not always perfect.
Image by author.

Figure 7: An ideal convergence, after implementing the k-means++

initialization method.

Conclusion
And with that, we’re finished. We learned a simple, yet elegant
implementation of an unsupervised machine learning model. The
complete project code is included below.
import numpy as np
import [Link] as plt
from [Link] import StandardScaler
from [Link] import uniform
from [Link] import make_blobs
import seaborn as sns
import random
def euclidean(point, data):
"""
Euclidean distance between point & data.
Point has dimensions (m,), data has dimensions (n,m), and output will
be of size (n,).
"""
return [Link]([Link]((point - data)**2, axis=1))
class KMeans: def __init__(self, n_clusters=8, max_iter=300):
self.n_clusters = n_clusters
self.max_iter = max_iter def fit(self, X_train): #
Initialize the centroids, using the "k-means++" method, where a random
datapoint is selected as the first,
# then the rest are initialized w/ probabilities proportional to
their distances to the first
# Pick a random point from train data for first centroid
[Link] = [[Link](X_train)] for _ in
range(self.n_clusters-1):
# Calculate distances from points to the centroids
dists = [Link]([euclidean(centroid, X_train) for centroid in
[Link]], axis=0)
# Normalize the distances
dists /= [Link](dists)
# Choose remaining points based on their distances
new_centroid_idx, = [Link](range(len(X_train)),
size=1, p=dists)
[Link] += [X_train[new_centroid_idx]] # This
initial method of randomly selecting centroid starts is less effective
# min_, max_ = [Link](X_train, axis=0), [Link](X_train, axis=0)
# [Link] = [uniform(min_, max_) for _ in
range(self.n_clusters)] # Iterate, adjusting centroids until
converged or until passed max_iter
iteration = 0
prev_centroids = None
while np.not_equal([Link], prev_centroids).any() and
iteration < self.max_iter:
# Sort each datapoint, assigning to nearest centroid
sorted_points = [[] for _ in range(self.n_clusters)]
for x in X_train:
dists = euclidean(x, [Link])
centroid_idx = [Link](dists)
sorted_points[centroid_idx].append(x) # Push
current centroids to previous, reassign centroids as mean of the points
belonging to them
prev_centroids = [Link]
[Link] = [[Link](cluster, axis=0) for cluster in
sorted_points]
for i, centroid in enumerate([Link]):
if [Link](centroid).any(): # Catch any [Link],
resulting from a centroid having no points
[Link][i] = prev_centroids[i]
iteration += 1 def evaluate(self, X):
centroids = []
centroid_idxs = []
for x in X:
dists = euclidean(x, [Link])
centroid_idx = [Link](dists)
[Link]([Link][centroid_idx])
centroid_idxs.append(centroid_idx) return centroids,
centroid_idxs
# Create a dataset of 2D distributions
centers = 5
X_train, true_labels = make_blobs(n_samples=100, centers=centers,
random_state=42)
X_train = StandardScaler().fit_transform(X_train)# Fit centroids to dataset
kmeans = KMeans(n_clusters=centers)
[Link](X_train)# View results
class_centers, classification = [Link](X_train)
[Link](x=[X[0] for X in X_train],
y=[X[1] for X in X_train],
hue=true_labels,
style=classification,
palette="deep",
legend=None
)
[Link]([x for x, _ in [Link]],
[y for _, y in [Link]],
'k+',
markersize=10,
)[Link]()

Create Your Own k-Nearest Neighbors Algorithm in Python

Introduction

Apr 9, 2022

29
In

Towards Data Science

Benjamin Etienne

Build your Personal Assistant with Agents and Tools

Learn how to build your personal assistant using LangChain agents and Gemini by grounding it in
external sources

1d ago

341
4

Towards Data Science

W Brett Kennedy

Perform Outlier Detection More Effectively Using Subsets of Features

Identify relevant subspaces: subsets of features that allow you to most effectively perform outlier
detection on tabular data

1d ago

406
9
In

Towards Data Science

Turner Luke

Create a Gradient Descent Algorithm with Regularization from Scratch in

Python
Cement your knowledge of gradient descent by implementing it yourself

May 2, 2022

99
1

See all from Turner Luke

See all from Towards Data Science

Recommended from Medium

Stackademic

Abdur Rahman

Python is No More The King of Data Science

5 Reasons Why Python is Losing Its Crown

Oct 23

8.2K
32
Raphael Schols

Understanding Binary Logistic Regression: A Comprehensive Guide to

Classification and Parameter…
Have you ever wondered how your Outlook knows an e-mail is spam? How does a bank know that
a certain transaction is fraudulent? How do…

May 31

8
In

Artificial Intelligence in Plain English

Dr. Roi Yehoshua

DBSCAN: Density-Based Clustering

In-depth explanation of the algorithm including examples in Python

Oct 17, 2023

89
1
Amit Yadav

K-Means Clustering Pseudocode and Implementation

Hey, is this you?

Jul 18
10. Data & Text Clustering using Gaussian Mixture Models

Master Generative AI: Your step-by-step guide to become a Certified GenAI expert
Download Roadmap

 Free Courses
 Learning Paths
 GenAI Pinnacle Program
 Agentic AI Pioneer ProgramNew



 Login

Interview Prep
Career
GenAI
Prompt Engg
ChatGPT
LLM
Langchain
RAG
AI Agents
Machine Learning
Deep Learning
GenAI Tools
LLMOps
Python
NLP
SQL
AIML Projects
READING LIST
Basics of Machine Learning
Machine Learning Lifecycle
Importance of Stats and EDA
Understanding Data
Probability
Exploring Continuous Variable
Exploring Categorical Variables
Missing Values and Outliers
Central Limit theorem
Bivariate Analysis Introduction
Continuous - Continuous Variables
Continuous Categorical
Categorical Categorical
Multivariate Analysis
Different tasks in Machine Learning
Build Your First Predictive Model
Evaluation Metrics
Preprocessing Data
Linear Models
KNN
Selecting the Right Model
Feature Selection Techniques
Decision Tree
Feature Engineering
Naive Bayes
Multiclass and Multilabel
Basics of Ensemble Techniques
Advance Ensemble Techniques
Hyperparameter Tuning
Support Vector Machine
Advance Dimensionality Reduction
Unsupervised Machine Learning Methods
Introduction to ClusteringApplications of ClusteringEvaluation Metrics for ClusteringUnderstanding K-
MeansImplementation of K-Means in PythonImplementation of K-Means in RChoosing Right Value for
KProfiling Market Segments using K-Means ClusteringHierarchical ClusteringImplementation of
Hierarchial ClusteringDBSCANDefining Similarity between clustersBuild Better and Accurate Clusters
with Gaussian Mixture Models
Recommendation Engines
Improving ML models
Working with Large Datasets
Interpretability of Machine Learning Models
Interpretability of Machine Learning Models
Automated Machine Learning
Model Deployment
Deploying ML Models
Embedded Devices

1. Home

2. Algorithm

3. Build Better and Accurate Clusters with Gaussian Mixture Models

Build Better and Accurate Clusters with Gaussian

Mixture Models

Aishwarya SinghLast Updated : 15 Oct, 2024

11 min read
14

Overview

 Gaussian Mixture Models are a powerful clustering algorithm

 Understand how Gaussian Mixture Models work and how to implement them in Python
 We’ll also cover the k-means clustering algorithm and see how Gaussian Mixture Models improve

on it

Introduction

I really like working on unsupervised learning problems. They offer a completely different challenge to a

supervised learning problem – there’s much more room for experimenting with the data that I have. It’s no

wonder that the majority of developments and breakthroughs in the machine learning space are happening in

the unsupervised learning domain.

And one of the most popular techniques in unsupervised learning is clustering. It’s a concept we typically

learn early on in our machine learning journey and it’s simple enough to grasp. I’m sure you’ve come across

or even worked on projects like customer segmentation, market basket analysis, etc.
But here’s the thing – clustering has many layers. It isn’t limited to the basic algorithms we learned earlier.

It is a powerful unsupervised learning technique that we can use in the real-world with unerring accuracy.

Gaussian Mixture Models are one such clustering algorithm that I want to talk about in this article.

Want to forecast the sales of your favorite product? Or perhaps you want to understand customer churn

through the lens of different groups of customers. Whatever the use case, you’ll find Gaussian Mixture

Models really helpful.

We’ll take a bottom-top approach in this article. So, we’ll first look at the basics of clustering including a

quick recap of the k-means algorithm. Then, we’ll dive into the concept of Gaussian Mixture Models and

implement them in Python.

Table of contents

1. Introduction

2. Introduction to Clustering

3. Introduction to k-means Clustering

4. Drawbacks of k-means Clustering

5. Introduction to Gaussian Mixture Models (GMMs)

6. The Gaussian Distribution

7. Characteristics of the Normal or Gaussian Distribution

8. What is Expectation-Maximization?

9. Expectation-Maximization in Gaussian Mixture Models

o E-step

o M-step

10. Implementing Gaussian Mixture Models in Python

11. End Notes

12. Frequently Asked Questions

Introduction to Clustering

Before we kick things off and get into the nitty-gritty of Gaussian Mixture Models, let’s quickly refresh

some basic concepts.

Note: If you are already familiar with the idea behind clustering and how the k-means clustering algorithm

works, you can directly skip to the fourth section, ‘Introduction to Gaussian Mixture Models’.

So, let’s start by formally defining the core idea:

Clustering refers to grouping similar data points together, based on their attributes or features.

For example, if we have the income and expenditure for a set of people, we can divide them into the

following groups:

 First – Earn high, spend high

 Second – Earn high, spend low

 Third – Earn low, spend low

 Fourth – Earn low, spend high

Each of these groups would hold a population with similar features and can be useful in pitching the relevant

scheme/product to the group. Think of credit cards, car/property loans, and so on. In simple words:

The idea behind clustering is grouping data points together, such that each individual cluster holds the most

similar points.

There are various clustering algorithms out there. One of the most popular clustering algorithms is k-means.

Let us understand how the k-means algorithm works and what are the possible scenarios where this

algorithm might come up short of expectations.

If you’re new to the world of clustering and data science, I recommend checking out the below

comprehensive course: Applied Machine Learning

Introduction to k-means Clustering

k-means clustering is a distance-based algorithm. This means that it tries to group the closest points to form

a cluster.

Let’s take a closer look at how this algorithm works. This will lay the foundational blocks to help you

understand where Gaussian Mixture Models will come into play later in this article.

So, we first define the number of groups that we want to divide the population into – that’s the value of k.

Based on the number of clusters or groups we want, we then randomly initialize k centroids.

The data points are then assigned to the closest centroid and a cluster is formed. The centroids are then

updated and the data points are reassigned. This process goes on iteratively until the location of centroids no

longer changes.

Note: This was a brief overview of k-means clustering and is good enough for this article. If you want to go

deeper into the working of the k-means algorithm, here is an in-depth guide: The Most Comprehensive

Guide to k-means you’ll Ever Need!

Drawbacks of k-means Clustering

The k-means clustering concept sounds pretty great, right? It’s simple to understand, relatively easy to

implement, and can be applied in quite several use cases. But there are certain drawbacks and limitations

that we need to be aware of.

Let’s take the same income-expenditure example we saw above. The K-means algorithm seems to be

working pretty well, right? Hold on – if you look closely, you will notice that all the clusters created are

circular. This is because the centroids of the clusters are updated iteratively using the mean value.

Now, consider the following example where the distribution of points is not circular. What do you think will

happen if we use k-means clustering on this data? It would still attempt to group the data points circularly.

That’s not great! k-means fails to identify the right clusters:

Hence, we need a different way to assign clusters to the data points. So instead of using a distance-based

model, we will now use a distribution-based model. And that is where Gaussian Mixture Models come

into this article!

Introduction to Gaussian Mixture Models (GMMs)

The Gaussian Mixture Model (GMM) is a probabilistic model used for clustering and density estimation. It

assumes that the data is generated from a mixture of several Gaussian components, each representing a
distinct cluster. GMM assigns probabilities to data points, allowing them to belong to multiple clusters

simultaneously. The model is widely used in machine learning and pattern recognition applications.

Gaussian Mixture Models (GMMs) assume that there are a certain number of components, where each

component is a Gaussian distribution. Hence, a Gaussian Mixture Model tends to group the data points

belonging to a single Gaussian component together. The parameters of the mixture components, such as the

means and covariances, are typically estimated using the Expectation-Maximization (EM) algorithm or

maximum likelihood estimation techniques.

Let’s say we have three Gaussian components (more on that in the next section) – GD1, GD2, and GD3.

These have a certain mean (μ1, μ2, μ3) and variance (σ1, σ2, σ3) value respectively. For a given set of data

points, our GMM would identify the probability of each data point belonging to each of these mixture

components. The EM algorithm iteratively updates these parameters to maximize the likelihood of the data,

without requiring the derivative to be calculated explicitly.

Wait, probability?

You read that right! Gaussian Mixture Models are probabilistic models and use the soft clustering

approach for distributing the points in different clusters. I’ll take another example that will make it

easier to understand.

Here, we have three clusters that are denoted by three colors – Blue, Green, and Cyan. Let’s take the data

point highlighted in red. The probability of this point being a part of the blue cluster is 1, while the

probability of it being a part of the green or cyan clusters is 0.

These probabilities are computed using Bayes’ theorem, which relates the prior and posterior probabilities of

the cluster assignments given the data. An important decision in GMMs is choosing the appropriate number

of components, which can be done using techniques like the Bayesian Information Criterion (BIC) or cross-

validation.

Now, consider another point – somewhere in between the blue and cyan (highlighted in the below figure).

The probability that this point is a part of cluster green is 0, right? The probability that this belongs to blue

and cyan is 0.2 and 0.8 respectively. These coefficients represent the responsibilities or soft assignments of

the data point to the different Gaussian components in the mixture.

Gaussian Mixture Models use the soft clustering technique for assigning data points to Gaussian

distributions, leveraging Bayes’ theorem to compute the posterior probabilities. I’m sure you’re wondering

what these distributions are so let me explain that in the next section.

The Gaussian Distribution

I’m sure you’re familiar with Gaussian Distributions (or the Normal Distribution). It has a bell-shaped

curve, with the data points symmetrically distributed around the mean value.

The below image has a few Gaussian distributions with a difference in mean (μ) and variance (σ 2 ).

Remember that the higher the σ value more the spread:

Source: Wikipedia

In a one dimensional space, the probability density function of a Gaussian distribution is given by:

where μ is the mean and σ2 is the variance.

But this would only be true for a single variable. In the case of two variables, instead of a 2D bell-shaped

curve, we will have a 3D bell curve as shown below:

The probability density function would be given by:

where x is the input vector, μ is the 2D mean vector, and Σ is the 2×2 covariance matrix. The covariance

would now define the shape of this curve. We can generalize the same for d-dimensions.

Thus, this multivariate Gaussian model would have x and μ as vectors of length d, and Σ would be a d x

d covariance matrix.

Hence, for a dataset with d features, we would have a mixture of k Gaussian distributions (where k is

equivalent to the number of clusters), each having a certain mean vector and variance matrix. But wait –

how is the mean and variance value for each Gaussian assigned?
These values are determined using a technique called expectation maximization (EM). We need to

understand this technique before we dive deeper into the working of Gaussian Mixture Models.

Characteristics of the Normal or Gaussian Distribution

Characteristics of the normal or Gaussian distribution:

 It’s bell-shaped with most values around the average.

 It has only one peak or mode.

 It stretches out forever in both directions.

 Its mean, median, and mode are the same.

 Its spread is measured by its standard deviation.

 The total area under its curve equals 1.

What is Expectation-Maximization?

Excellent question!

Expectation-Maximization (EM) is a statistical algorithm for finding the right model parameters. We

typically use EM when the data has missing values, or in other words, when the data is incomplete.

These missing variables are called latent variables. We consider the target (or cluster number) to be

unknown when we’re working on an unsupervised learning problem.

It’s difficult to determine the right model parameters due to these missing variables. Think of it this way – if

you knew which data point belongs to which cluster, you would easily be able to determine the mean vector

and covariance matrix.

Since we do not have the values for the latent variables, expectation-maximization tries to use the

existing data to determine the optimum values for these variables and then finds the model
parameters. Based on these model parameters, we go back and update the values for the latent variable, and

so on.

Broadly, the Expectation-Maximization algorithm has two steps:

 E-step: In this step, the available data is used to estimate (guess) the values of the missing variables

 M-step: Based on the estimated values generated in the E-step, the complete data is used to update

the parameters

Expectation-Maximization is the base of many algorithms, including Gaussian Mixture Models. So how

does GMM use the concept of EM and how can we apply it for a given set of points? Let’s find out!

Expectation-Maximization in Gaussian Mixture Models

Let’s understand this using another example. I want you to visualize the idea in your mind as you read

along. This will help you better understand what we’re talking about.

Let’s say we need to assign k number of clusters. This means that there are k Gaussian distributions, with

the mean and covariance values to be μ1, μ2, .. μk and Σ1, Σ2, .. Σk. Additionally, there is another parameter

for the distribution that defines the number of points for the distribution. In other words, the density of the

distribution is represented with Πi, capturing the relative sizes of different subpopulations.

Now, we need to find the values for these parameters to define the Gaussian distributions. We already

decided on the number of clusters and randomly assigned the values for the mean, covariance, and density.

Next, we’ll perform the expectation step (E-step) and the maximization step (M-step) iteratively!

In the E-step, we compute the probability of each data point belonging to each of the k Gaussian

components, given the current parameter values. Then, in the M-step, we re-estimate the parameters (means,

covariances, and component weights) to maximize the likelihood of the data, using the responsibilities

computed in the E-step. This optimization process continues until convergence or a maximum number of

iterations is reached. Advanced techniques like variational inference can also be used for parameter

estimation in complex GMM scenarios.

E-step

For each point x i, calculate the probability that it belongs to cluster/distribution c 1, c 2, … c k. This is done

using the below formula:

This value will be high when the point is assigned to the right cluster and lower otherwise.

M-step

Post the E-step, we go back and update the Π, μ and Σ values. These are updated in the following manner:

1. The new density is defined by the ratio of the number of points in the cluster and the total number of

points:

2. The mean and the covariance matrix are updated based on the values assigned to the distribution, in

proportion with the probability values for the data point. Hence, a data point that has a higher

probability of being a part of that distribution will contribute a larger portion:

Based on the updated values generated from this step, we calculate the new probabilities for each data point

and update the values iteratively. This process is repeated in order to maximize the log-likelihood function.

Effectively we can say that the

k-means only considers the mean to update the centroid while GMM takes into account the mean as well as

the variance of the data!

Implementing Gaussian Mixture Models in Python

It’s time to dive into the code! This is one of my favorite parts of any article so let’s get going straightaway.

We’ll start by loading the data. This is a temporary file that I have created – you can download the data

from this link.

Python Code:

import pandas as pd
import [Link] as plt
data = pd.read_csv('Clustering_gmm.csv')

[Link](figsize=(7,7))
[Link](data["Weight"],data["Height"])
[Link]('Weight')
[Link]('Height')
[Link]('Data Distribution')
[Link]()Copy Code

That’s what our data looks like. Let’s build a k-means model on this data first:
#training k-means model
from [Link] import KMeans
kmeans = KMeans(n_clusters=4)
[Link](data)

#predictions from kmeans

pred = [Link](data)
frame = [Link](data)
frame['cluster'] = pred
[Link] = ['Weight', 'Height', 'cluster']

#plotting results
color=['blue','green','cyan', 'black']
for k in range(0,4):
data = frame[frame["cluster"]==k]

[Link](data["Weight"],data["Height"],c=color[k])
[Link]()
view rawbuilding_kmeans.py hosted with ❤ by GitHub

That’s not quite right. The k-means model failed to identify the right clusters. Look closely at the clusters in

the center – k-means has tried to build a circular cluster even though the data distribution is elliptical

(remember the drawbacks we discussed earlier?).

Let’s now build a Gaussian Mixture Model on the same data and see if we can improve on k-means:

import pandas as pd
data = pd.read_csv('Clustering_gmm.csv')

# training gaussian mixture model

from [Link] import GaussianMixture
gmm = GaussianMixture(n_components=4)
[Link](data)

#predictions from gmm

labels = [Link](data)
frame = [Link](data)
frame['cluster'] = labels
[Link] = ['Weight', 'Height', 'cluster']

color=['blue','green','cyan', 'black']
for k in range(0,4):
data = frame[frame["cluster"]==k]

[Link](data["Weight"],data["Height"],c=color[k])
[Link]()
view rawgaussian_mixture_model.py hosted with ❤ by GitHub

Excellent! Those are exactly the clusters we were hoping for. Gaussian Mixture Models have blown k-

means out of the water here.

End Notes

This was a beginner’s guide to Gaussian Mixture Models. My aim here was to introduce you to this

powerful clustering technique and showcase how effective and efficient it can be as compared to your

traditional algorithms.

Common questions