Ecological Informatics 69 (2022) 101585

Contents lists available at ScienceDirect

Ecological Informatics
journal homepage: [Link]/locate/ecolinf

BLeafNet: A Bonferroni mean operator based fusion of CNN models for

plant identification using leaf image classification
Shreyan Ganguly a, Pratik Bhowal b, Diego Oliva c, e, *, Ram Sarkar d
a
Department of Construction Engineering, Jadavpur University, India
b
Department of Instrumentation and Electronics Engineering, Jadavpur University, India
c
Depto. Innovación Basada en la Información y el Conocimiento, Universidad de Guadalajara, CUCEI, Mexico
d
Department of Computer Science and Engineering, Jadavpur University, India
e
School of Computer Science and Robotics, Tomsk Polytechnic University, Tomsk, Russia

A R T I C L E I N F O A B S T R A C T

Keywords: Plants, the only natural source of oxygen, are the most important resources for every species in the world. A
Plant identification proper identification of plants is important for different fields. The observation of leaf characteristics is a popular
Leaf image method as leaves are easily available for examination. Researchers are increasingly applying image processing
Deep learning
techniques for the identification of plants based on leaf images. In this paper, we have proposed a leaf image
Ensemble learning
classification model, called BLeafNet, for plant identification, where the concept of deep learning is combined
Bonferroni operator
with Bonferroni fusion learning. Initially, we have designed five classification models, using ResNet-50 archi­
tecture, where five different inputs are separately used in the models. The inputs are the five variants of the leaf
grayscale images, RGB, and three individual channels of RGB - red, green, and blue. For fusion of the five ResNet-
50 outputs, we have used the Bonferroni mean operator as it expresses better connectivity among the confidence
scores, and it also obtains better results than the individual models. We have also proposed a two-tier training
method for properly training the end-to-end model. To evaluate the proposed model, we have used the
Malayakew dataset, collected at the Royal Botanic Gardens in New England, which is a very challenging dataset
as many leaves from different species have a very similar appearance. Besides, the proposed method is evaluated
using the Leafsnap and the Flavia datasets. The obtained results on both the datasets confirm the superiority of
the model as it outperforms the results achieved by many state-of-the-art models.

1. Introduction method as leaves are widely available for observation and examination
for deciduous, annual plants or year-round in evergreen perennials. Due
Plants are an integral part of nature and are one of the most to this, computer vision scientists are increasingly applying techniques
important resources for every species on Earth. It is the only natural for the identification of plants using leaf images as the distinguishing
source of oxygen and is one of the most important sources of food for the parameter.
entire animal kingdom. Plants help in controlling global warming. There Plant leaves in real life are often very similar in color and shape as is
are an estimated of about 400,000 plant species (Christenhusz and Byng, evident from Fig. 2, and so it is a very challenging task for computer
2016) and distinguishing among them is important not only for agri­ scientists to classify plants based on leaves. In the earlier days, mainly
culture and industry but also for the conservation process and the different handcrafted features were used for the classification process.
analysis of their population. Shape, texture, venation are some of the features that have been used
In agronomy, an important task is to identify and classify the plants. extensively in the past for leaf classification. However, there are two
Botanists usually recognize plants from the shape and color of their main problems concerning these methods. (a) The best subset of features
leaves and flowers. Text-based taxonomic keys that use leaf characters has to be used and it needs to be justified why that particular subset is
have been used in the identification of plants since the advent of used. However, the subset that performs best changes from dataset to
botanical science. Observation of leaf characteristics is a more popular dataset, thus making these models highly unreliable in a practical

* Corresponding author.
E-mail address: [Link]@[Link] (D. Oliva).

[Link]
Received 28 September 2021; Received in revised form 23 November 2021; Accepted 23 January 2022
Available online 17 February 2022
1574-9541/© 2022 Elsevier B.V. All rights reserved.
S. Ganguly et al. Ecological Informatics 69 (2022) 101585

Fig. 1. Subset of various types of leaves from Malayakew dataset, used in training BLeafNet.

Fig. 2. Each row shows leaf species with similar shape. Samples are taken from Malayakew dataset.

scenario. (b) These features rely heavily on how accurately one can state-of-the-art deep learning model, used is the same for all the five
encode morphological characteristics predefined by botanists. On the classification processes, the inputs to the models are different. The five
other hand, deep learning based models automatically learn features different inputs are the RGB image, grayscale image, red, blue and green
and are not dependent on any predefined set of morphological charac­ channels of the RGB image. ResNet-50 captures various information
teristics, as mentioned earlier. Moreover, since deep learning models from each of the five inputs and hence makes different classification
learn its own representations, it works universally on various datasets. errors. As a result, we get five decision81 makers with diverse errors,
Fusion learning further helps in increasing the accuracy of the which are then combined strategically to increase the overall accuracy
learning models. To be more specific, fusion learning helps when the of the classification process.
models make diverse errors and the strategical combination of the re­ In this paper, we use the Bonferroni mean operator for aggregating
sults removes the errors made by the individual models, thus increasing the decisions of the five ResNet-50 models. Though Bonferroni operator
the overall performance of the learning model. It has been widely used is explained in more detail later, we explain our choice of operator
in a number of varied fields like carcinoma classification (Sanyal et al., briefly here. Bonferroni operator is selected for the following two rea­
2021), signature verification (Bhowal et al., 2021), human pose esti­ sons: (a) The Bonferroni mean operator can be thought of as an anding
mation (Banerjee et al., 2020), sentiment analysis (Ray et al., 2021) etc. and averaging operator. The product operation in the Bonferroni mean
In this paper, we have combined the decisions obtained from five can be interpreted as the anding operation. (b) The Bonferroni mean
different classification models. Though the model, ResNet-50 which is a operator introduces the concept of inverse ordering. Inverse ordering is

2
S. Ganguly et al. Ecological Informatics 69 (2022) 101585

Fig. 3. Leaf images captured in different lighting conditions. Samples are taken from Leafsnap dataset.

Fig. 4. An illustration of the ResNet-50 model (He et al., 2016).

described in detail later, but here we give an intuitive idea. While • We have proposed a novel model, called BLeafNet, for the identifi­
aggregating the confidence scores of classifiers, we multiply weights cation of plants using leaf images classification. We have used pre-
with the classifier's confidence score, which are proportional to the trained ResNet- 50 model five times with different color channel
confidence scores of the other classifiers. This inverse ordering is helpful inputs of the same image and finally fused the outputs of these
because when one classifier makes a wrong decision with a high confi­ models using Bonferroni operator which we have realized using a
dence, its effect on the overall decision reduces. This is not achievable neural network structure.
using commonly used mean or weighted arithmetic mean operations • We have proposed a two-tire method to train the model. In the first
because this inverse ordering is not present and when a classifier with tire, we train each of the five CNN models individually, while in the
high accuracy makes a wrong decision with high confidence, these second tire, we train the entire BLeafNet consisting of the five CNNs
methods are unable to make the right decision even after fusion. and the realization of the BLeafNet at the end.
We have implemented the Bonferroni mean operator using a neural • We have verified our result on three datasets, namely, the Malaya­
network architecture. This has the added benefit that during training, kew dataset, the Leafsnap dataset, and the Flavia dataset. We have
the weights of the convolutional neural network (CNN) models adjust achieved an accuracy of 98.54%, an accuracy of 92.22%, and an
themselves in such a way that the final result after fusion has a higher accuracy of 98.7% on Malayakew dataset, Leafsnap dataset, and
accuracy. In traditional fusion approaches, this is not the case and the Flavia dataset respectively.
models are trained in a way to maximize their individual accuracy and
not the accuracy of the entire model, which often results in a decrease in 2. Literature survey
the overall accuracy and sub-optimal results.
In a nutshell, the main contributions of this work are as follows: The classification of plant species is an important task, and the use of

3
S. Ganguly et al. Ecological Informatics 69 (2022) 101585

Fig. 5. Phase 1 training - five deep learning based classification models trained on RGB, Grayscale, R, G, B individually are shown.

plant leaves for this purpose is one of the best ways to classify plant and texture features, and finally an SVM network for leaf image classi­
species due to the availability and unique features of the leaves. Many fication. Metrippun (Mettripun, 2020) used eight geometric properties
researchers used various handcrafted features extracted from the plant of leaf image - majority axis, minority axis, aspect ratio, eccentricity,
leaf images in the past. They have mostly used three types of features: circularity, extent, Perimeter and RPMA, followed by an artificial neural
shape, texture, and venation. network (ANN) for Thai leaf image classification. Chaudhury et al.
For describing leaf shape as features, Meyer et al. (Camargo Neto (Chaudhury and Barron, 2020) used discrete countour evolution (DCE)
et al., 2006) used elliptic Fourier and discriminant analysis methods to algorithm by plotting countour information as B-spline curve to detect
classify different plant leaves. Zhao et al. (Zhao et al., 2017) used plant species from occluded leaf images. Arribas et al. (Arribas et al.,
Centroid Contour Distance Curve (CCDC) which is based on edge point 2011) proposed a leaf classification technique on RGB color space.
detection. They converted the complex shape of the leaves into a graph However, it was only restricted to sunflower leaves. The handcrafted
structure to describe the topological skeleton of the leaves. Xiao et al. feature extraction techniques, however, may perform really well on
(Xiao et al., 2010) used the Histogram of Oriented Gradients (HOG) for training data, but the performance of these models is highly dependent
representing leaf shape and Maximum Margin Criterion (MMC) for on the chosen set of features which are mostly dataset dependent.
dimension reduction on Swedish leaf dataset and ICL dataset. Kumar Hence, the mentioned methods suffer from dataset bias problems.
et al. (Kumar et al., 2012) proposed the Histogram of Curvature (HoCS) With the rise of computation power and volume of data, deep
to analyze leaves on the LeafSnap dataset. Texture is widely used as learning models have shown impressive success in large-scale image
feature for leaf classification. Herdiyeni et al. (Herdiyeni and Kusmana, recognition. CNN based models overcome the limitations of the hand­
2013) used a Local Binary Pattern (LBP) which is a texture descriptor, crafted features, by its ability to automatically learn features from im­
and then Probabilistic Neural Network classifier was used for classifying ages/data in the training phase, and such models are yielding good
leaf images. Wang et al. (Wang et al., 2013) proposed a leaf image results in a various complex image of pattern classification tasks.
segmentation technique which was used in further works of leaf image Liu et al. used a basic CNN model for features extraction, and
classification. Sujith et al. (Sujith and Aji, 2020) used combination of appended a support vector machine (SVM) for leaf image classification
three feature extraction methods - LBP, Gray level co-occurrence matrix and was trained on Flavia dataset. Lee et al. (Lee et al., 2015) proposed
(GLCM) and HOG to classify leaf images. Kan et al. (Kan et al., 2017) the DeepPlant network trained on Malayakew dataset to classify leaf
used geometric features (like eccentricity, rectangularity, circularity) images, they also used de-convolution network (DN) to gain insight on

4
S. Ganguly et al. Ecological Informatics 69 (2022) 101585

Fig. 6. Five individual models integrated by Bonferroni mean operator after fusion stage, forming the proposed BLeafNet. (Phase 2 training)

the automatically selected features by the CNN model. It has been found which takes different color channels of the same leaf image as inputs.
that venation is a good representation of the leaf features. Lee et al. Hence, the classifiers classify the images based on features that are
further extended their work and proposed a two-stream CNN model, different from one another and as mentioned above strategic combina­
where two feature learning streams are learned on whole and path leaf tion of the results of the classifier gives a higher overall accuracy.
images on Malayakew D1 and D2 datasets. Du et al. and Rasti et al. (Du
and Gao, 2017) proposed multi-scale convolution neural network 3. Proposed work
(MSCNN) to learn features on multiple scales of the leaf images. Hu et al.
(Hu et al., 2018) further extended this work to multiscale fusion CNN In this section, we describe the proposed model, called BLeafNet,
which also learns features at different scales of a leaf image, but unlike used for the classification of the leaf images. An image in RGB color
the former, the fusion occurs at different steps. Malayakew and LeafSnap space consists of three color channels Red, Green, and Blue. In this
datasets were used for testing the model. Esgario et al. (Esgario et al., method, we utilize the RGB image, the grayscale image, and also each of
2020) used ResNet-50 model for classification and severity of coffee leaf the three color channel images of the RGB image separately.
biotic stress. Akter et al. (Akter and Hosen, 2020) used a three-layer CNN
to classify leaf images, and tested the model on a dataset comprising leaf 3.1. Data pre-processing
images belonging to 10 classes of medicinal plants. Liu et al. (Liu et al.,
2018) proposed a 10-layer CNN for leaf recognition, and tested the BLeafNet has been evaluated on the Malayakew dataset, Leafsnap
network on the Flavia dataset. Carneiro et al. (Carneiro et al., 2021) used dataset, and Flavia dataset, which comprises 44, 185, and 33 classes of
a transfer learning approach on the pre-trained Xception model for leaf images respectively. The datasets have been explained in detail later
Grapevine detection by Grapevine leaf image, and tested the model on a under the experimentation section. The Malayakew dataset consists of
Grapevine leaf image dataset. Figueroa-Mata et al. (Figueroa-Mata and segmented leaf images and image dimension in the Malayakew dataset is
Mata-Montero, 2020) used a convolutional Siamese network to learn 256 × 256. In BLeafNet, we use five ResNet-50 models, with different
similarity metric between leaf images, and used the same to discriminate inputs, pre-trained on the ImageNet dataset. So to train the proposed
between leaf images. Zhang et al. (Zhang et al., 2019) proposed a global model on the Malayakew dataset, we change the dimension of the data
pooling dilated convolutional neural network (GPDCNN) for plant dis­ to 224 × 224.
ease identification by combining dilated convolution with global pool­ In Leafsnap field dataset, firstly we crop the background rulers from
ing. Beikmohammadi et al. (Beikmohammadi and Faez, 2019) proposed the images. After cropping we rotate every image in the dataset to
SWP - Leaf Net which used transfer learning on pretrained architecture various angles. Finally, we resize the image to 224 × 224 for fitting the
MobileNetV2. Malaykew and Flavia datasets were used for performance image in ResNet-50 models. Also in Flavia dataset, we resize every single
evaluation. Song et al. used Attention-based CNN (Song et al., 2019) to leaf image to 224 × 224 for passing them as input to our model.
classify leaf images. After necessary changes, we make five copies of every image present
However, in most of these models, a single classifier or model clas­ in the dataset, each belonging to five different color domains. The five
sifies the images of the leaves based on the original RGB image. On the copies are the original RGB image(I1), the grayscale format of the image
contrary, we combine the results of five different classifiers, each of (I2), the red channel information of the image (I3), the green channel

5
S. Ganguly et al. Ecological Informatics 69 (2022) 101585

Fig. 7. The deep learning based implementation of the Bonferroni mean operator. Here, F1-F5 are the outputs of the Fully Connected Layers of the deep learning
models which have RGB, Grayscale, R, G and B channel images as their inputs respectively.

information of the image(I4), and the blue channel information of the ar, 5) be the five confidence scores of the r-th class (r ∈ [1; 44]) of the five
image(I5). This is achieved using the OpenCV library. models. Then, the Bonferroni mean B1 of the five confidence scores of
the first class is defined as.
3.2. Model architecture ⎛ ⎛ ⎞ ⎞12
⎜1 ∑ ⎜ 1 ∑ ⎟ ⎟
n n
( )
B1 a1,1 , a1,2 , a1,3 , a1,4 , …a1,n =⎝ a1,i ⎝ a1,j ⎠ ⎠ (1)
BLeafNet uses ResNet-50 as the basic building block of the model. As n i=1 n − 1 j=1
shown in Fig. 4, ResNet-50 (He et al., 2016) is a CNN model comprising j∕
=i

48 convolution layers, 1 max-pool layer, and 1 final classification layer, In this way, we calculate Bonferroni mean of confidence scores of 44
a total of 50 layers. Despite its very deep architecture, ResNet-50 per­ classes from five models. We implement this mathematical formulation
forms really well due to its residual blocks which solve the vanishing in our model in the form of neural networks. So, the final output layer is
gradient problem in such a deep model and that property distinguishes of shape 44 × 1 and is defined as (B1, B2, …, B43, B44). In the end, it
ResNet-50 from other CNN models. returns the class having the maximum confidence score in the final
We customize the ResNet-50 models, each of the custom models layer. BLeafNet is basically an fusion neural network model consisting of
replaces the final classification layer in the original ResNet model with a five ResNet-50 models, taking five images as input and returning a
fully connected dense layer with Rectified Linear Unit (ReLU) activation predicted class label as classification output as shown in Fig. 6.
function and a dropout for avoiding the overfitting issue. Finally, we Eq. (1) can be written more concisely as in Eq. (2)
append a softmax classification layer consisting of 44 nodes to each of
( )12
our custom models as the final layer as shown in Fig. 5. ( ) 1∑ n

Now, we have five models that are, in total, capable of taking five B1 a1,1 , a1,2 , a1,3 , a1,4 , …a1,n = u1,i a1,i (2)
n i=1
images of shape 224 × 224 as input and return five sets (i.e., five arrays)
( )
of confidence scores, each of shape 44 × 1. We aggregate the five sets of ∑n
confidence scores to one final output layer comprising aggregated con­ where ui = 1
n− 1 j=1 a1,j
j∕
=i
fidence scores of 44 classes from the five models and return the class
This aggregation operator can be thought of as an anding and aver­
with maximum confidence score as the output of BLeafNet. For
aging operator. The product operation can be interpreted as the anding
combining these, we use an aggregation operator, called Bonferroni
operation. Thus a multiplication between a1, i and a1, j tells us how much
mean operator, which expresses better inter-connectivity between the
both the criteria indicated by a1, i and a1, j are satisfied. Moreover, it can
confidence scores of the classes. We introduce layers of neural network
be seen that if a1, i > a1, j, u1, j > u1, i. Thus, with respect to a1, i, u1, is are
for implementing the Bonferroni mean operator. Let (ar, 1, ar, 2, ar, 3, ar, 4,

6
S. Ganguly et al. Ecological Informatics 69 (2022) 101585

Fig. 8. Confusion matrix of BLeafNet on Malayakew dataset.

3.3. Training
Table 1
Test Accuracy (in percentage), Precision, Recall, F1 score, Log Loss of different We train the model BLeafNet on the Malaykew dataset consisting of
models used in BLeafNet on Malayakew dataset. 2288 training images. The training phase is broadly divided into two
Model Test Precision Recall F1 Log steps, as follows:
Accuracy Score Loss

Resnet50 (RGB 95.32 0.955 0.9583 0.9566 0.1534 3.3.1. Before fusion
input) Initially, we train the five different ResNet-50 models individually.
Resnet50 92.50 0.9305 0.9242 0.9273 0.2556 These fives models are trained with five color variants i.e., five different
(Grayscale)
Resnet50 (R channel 92.29 0.926 0.9317 0.9288 0.287
copies of an input image, namely original RGB image, grayscale image,
input) red channel image, blue channel image, and green channel image. For
Resnet50 (G channel 92.71 0.944 0.9412 0.9428 0.1998 training on Malayakew dataset, We use Adam optimizer with dynamic
input) learning rate and log loss as the loss metric. We use a batch size of 32
Resnet50 (B channel 92.29 0.9189 0.9063 0.9126 0.2916
with a step size of 40 and an epoch of 60. For training on Leafsnap
input)
BLeafNet 98.54 0.9698 0.9659 0.9678 0.1293 dataset, We use Adam optimizer with dynamic learning rate and log loss
as the loss metric. We use a batch size of 120 with a step size of 40 and an
epoch of 80. For training on Flavia dataset, we use Adam optimizer with
inversely ordered. The importance and advantage of this inverse dynamic learning rate and Cross-entropy loss as the loss metric. We use a
ordering have been explained before. batch size of 32 with a step size of 40 and an epoch of 70.
The Bonferroni mean operator layer, as seen in Fig. 6, is elaborated as
can be visible in Fig. 7. As can be visible in Fig. 7, it consists of a number 3.3.2. After fusion
of layers and basically implements Eq. 1 using neural network layers. After we train the five individual models, we combine the models
The first layer contains five parallel channels, each of which adds the using Bonferroni mean operator layer shown in Fig. 6. We then train the
confidence score of all the 44 classes for the other four models. The fusion model with five copies of the same input and one ground-truth
second layer also consists of five parallel channels and in each of the label as output. For training on Malayakew dataset, We use Adam
channels, the values obtained from the last layer are multiplied by 0.25 optimizer with dynamic learning rate and log loss as the loss metric. We
(basically divided by (N-1) and in our case N = 5). In the third layer, we use a batch size of 32 with a step size of 40 and an epoch of 60. For
multiply the output of the second layer to that output vector of the training on Leafsnap dataset, We use Adam optimizer with dynamic
respective ResNet-50 model. In the fourth layer, we add all the tensors learning rate and log loss as the loss metric. We use a batch size of 120
that we get from the third layer, and hence from this layer onward we do with a step size of 40 and an epoch of 80. For training on Flavia dataset,
not have five parallel channels, rather, we have a single channel. The we use Adam optimizer with dynamic learning rate and Cross-entropy
output of the fourth layer is multiplied by 0.2 in the fifth layer and loss as the loss metric. We use a batch size of 32 with a step size of 40
finally, the square root of the output of the fifth layer is taken in the sixth and an epoch of 70. This proves to be highly effective to shift the weights
layer and it is the final layer and the class with the maximum confidence in the right direction of our fusion model, BLeafNet.
score in this layer is taken as the output.

7
S. Ganguly et al. Ecological Informatics 69 (2022) 101585

Table 2
Feature representation of different layers in BLeafNeton Malayakew dataset.

Stream Layer 10 Layer 15 Layer 20 Layer 25

RGB

Grayscale

Red

Green

Blue

Table 3 Table 4
Test Accuracy (in percentage), Precision, Recall, F1 score, Log Loss of different Test Accuracy (in percentage), Precision, Recall, F1 score, Log Loss of different
models used in BLeafNet on Leafsnap dataset. models used in BLeafNet on Flavia dataset.
Model Test Precision Recall F1 Log Model Test Precision Recall F1 Log
Accuracy Score Loss Accuracy Score Loss

ResNet50 (RGB 88.89 0.8465 0.7834 0.8137 0.5471 ResNet50 (RGB 95.00 0.9497 0.932 0.9407 0.2798
input) input)
ResNet50 85.96 0.8172 0.77 0.7928 0.594 ResNet50 94.58 0.9215 0.914 0.9177 0.3767
(Grayscale) (Grayscale)
ResNet50 (R channel 86.44 0.7953 0.7578 0.7761 0.672 ResNet50 (R channel 94.58 0.92 0.9178 0.9188 0.3801
input) input)
ResNet50 (G channel 87.14 0.8245 0.7783 0.8007 0.5627 ResNet50 (G channel 94.71 0.9328 0.9279 0.9302 0.3071
input) input)
ResNet50 (B channel 84.33 0.7662 0.6971 0.73 0.7965 ResNet50 (B channel 93.33 0.9139 0.9156 0.9147 0.3842
input) input)
BLeafNet 92.22 0.888 0.853 0.8701 0.4046 BLeafNet 98.70 0.9557 0.9421 0.9489 0.2502

8
S. Ganguly et al. Ecological Informatics 69 (2022) 101585

Table 5 a) Numbers of images in this dataset is 30,866, belonging to field image

Performance comparison of different methods on Malayakew dataset. and lab image.
Method Accuracy(%) b) Number of classes is 185.
LeafSnap+SVM(RBF) (Kumar et al., 2012) 42.00
LeafSnap+NN (Kumar et al., 2012) 58.90 4.1.3. Flavia dataset
S-LeafNet (Beikmohammadi and Faez, 2019) 96.51 Flavia dataset contains controlled images with truncated stem and
W-LeafNet (Beikmohammadi and Faez, 2019) 97.35 scanned with a white background. The specifications of this dataset are
P-LeafNet (Beikmohammadi and Faez, 2019) 85.35 as follows:
DeepPlant + MLP (Lee et al., 2015) 97.70
DeepPlant + SVM(Linear) (Lee et al., 2015) 98.10
MSF-CNN (Hu et al., 2018) 99.05 (a) Numbers of images in this dataset is 1907, each having an image
BLeafNet 98.54 resolution of 1200 × 1600.
(b) Number of classes is 32.

Table 6 4.2. Results

Performance comparison of different methods on Leafsnap dataset.
In this section, we discuss the performance of BLeafNet evaluated on
Method Accuracy(%)
Malayakew D1 dataset and Leafsnap dataset.
LeafSnap (Kumar et al., 2012) 75.00
MSF-CNN (Hu et al., 2018) 85.28
4.2.1. Malayakew D1 dataset
Deep-CNN (Beikmohammadi and Faez, 2018) 90.54
ABCNN (Song et al., 2019) 91.43 Table 1 shows test accuracy, precision, recall, F1 score, log loss of
BLeafNet 92.22 different models used in BLeafNet on the Malayakew dataset. In the
before-fusion stage, it is observed that the model with RGB performs the
best with a test accuracy of 95.32%. However, in the after-fusion stage,
Table 7 BLeafNet outperforms every previous model with an accuracy of
Performance comparison of different methods on Flavia dataset. 98.54%. The confusion matrix of BLeafNet on Malayakew dataset is
shown in Fig. 8
Technique Accuracy(%)

Wu et al. (Wu et al., 2007) 90.3 4.2.2. Leafsnap dataset

Krishna et al. (Singh et al., 2010) 91
Table 3 shows the test accuracy, precision, recall, F1 score, log loss of
Arun Priya et al. (Priya et al., 2012) 94.5
Gwo and Wei et al. (Gwo and Wei, 2013) 92.7 different models used in BLeafNet on the Leafsnap [Link] is observed,
Satti et al. (Satti et al., 2013) 93.3 BLeafNet outperforms each of the individual models also in this dataset.
Lavania and Matey et al. (Lavania and Matey, 2014) 87.5 BLeafNet reaches a test accuracy of 92.11% on a batch size of 120 and a
Tsolakidis et al. (Tsolakidis et al., 2014) 97.18
dynamic learning rate. A considerable improvement in performance is
Kadir (Kadir, 2014) 97.17
Hsiao et al. (Hsiao et al., 2014) 95.47
observed due to the fusion of the individual models and retraining the
Saleen et al. (Saleem et al., 2019) 98.75 hybrid fused model.
BLeafNet 98.70
4.2.3. Flavia dataset
Table 4 shows the test accuracy, precision, recall, F1 score, log loss of
4. Experimental results and discussion
different models used in BLeafNet on Flavia dataset. It is observed,
BLeafNet outperforms each of the individual models also in this dataset.
4.1. Dataset description
BLeafNet reaches a test accuracy of 98.7% on a batch size of 32 and a
dynamic learning rate. A considerable improvement in performance is
In this study, we have used three publicly accessible leaf image
observed due to the fusion of the individual models and retraining the
datasets, namely Malayakew, Leafsnap and Flavia..
hybrid fused model.

4.1.1. Malayakew D1 dataset

4.3. Discussion
MalayaKew (MK) Leaf dataset, collected at the Royal Botanic Gar­
dens, Kew, England, consists of segmented images of leaves from 44
In this section, we provide some insights on the results obtained by
species classes. This dataset is very challenging as leaves from different
testing our proposed model on three standard leaf image datasets -
species classes have a very similar appearance. Fig. 1 shows some sample
Malayakew, Leafsnap, and Flavia datasets.
of leaves from this dataset. The specifications of this dataset are as
Table 1 shows the result of individual model and the proposed fused
follows:
model tested on the Malayakew dataset. It is clear that the fused model
outperforms the individual model, reaching a testing accuracy of
a) It consists of segmented leaf images with size 256 * 256 pixels.
98.54%, so it can be confirmed that the Bonferroni-mean fusion tech­
b) Numbers of training and testing images are 2288 and 528
nique is effective and further training of BLeafNet (in the after-fusion
respectively.
stage) is proved to be highly useful to shift the weights in the right di­
c) Number of classes is 44.
rection and it helps to increase the performance of the overall model.
The confusion matrix in Fig. 8 shows the extent of false positives, true
4.1.2. Leafsnap dataset
positives, and overall performance of the model on unseen test leaf
Leafsnap dataset was obtained from two sources - the first source
images. It is observed that BLeafNet struggles mostly with class 0 and
contains leaf image captured in lab, and the second source contains field
class 1 of the Malayakew dataset, with class 0 having the maximum false
image image mostly captured in mobile devices. This field image
positive count. It can also be observed that the proposed model mis­
directory varies in sharpness, noise and illumination patterns, etc., as
classifies some images of class 1 and class 9, due to the high similarity in
shown in Fig. 3 whereas the former section consists of leaf images in
spacial structures of the stated classes. Classes 14, 22, 23, 39, and 42 also
ideal condition. The specifications of this dataset are as follows:
get misclassified to a small extent. However, most of the images
belonging to the other classes are classified correctly.

9
S. Ganguly et al. Ecological Informatics 69 (2022) 101585

Table 2 shows different feature representations as the five forms of a predicted by the Neural network itself and may result in vanishing
leaf image flow through the five streams of the BLeafNet. It is observed gradient during the second tire of the training.
the five ResNet-50 models learn distinct features from a leaf image,
making different classification errors, and it help to improve the overall 5. Conclusion and future directions
accuracy of the model. It is also observed the deeper we go in the model,
the model learns more complicated leaf image patterns like protruding The identification of plants is an important task not only for
leaf shapes and vein structures. exploiting it for the agro and medicine industries but also for their
conservation, analysis, and research. Identification of plants based on
4.4. Comparison with the state-of-the-art the color and shape of their leaves is a well-tested method and has been
applied consistently over time. Computer vision scientists have applied
In Table 5 and Table 6, performances of various state-of-the-art many image processing and machine learning techniques to classify
methods used for leaf image classification on Malayakew, Leafsnap, leaves in an attempt to classify plants. This is however a rather difficult
and Flavia datasets have been compared with the proposed model. procedure as leaves are often similar in color and shape.
BLeafNet outperforms most of the methods. We have proposed a novel network for the classification of leaves.
In Table 5, performance comparison of different method on We have realized the Bonferroni mean operator in it and hence have
Malayakew dataset has been shown. LeafSnap (Kumar et al., 2012) named it BLeafNet. We have used five different pre-trained CNN net­
proposed by Kumar et al. used histogram of curvature to encode leaf works, namely ResNet-50 models, and have combined the results using a
image features, and used SVM and neural network for decoding the number of layers that resemble Bonferroni mean operator. We have
features. This method relied heavily on texture features for the classi­ evaluated the proposed model on the Malayakew D1 dataset, Leafsnap
fication. However, this method used handcrafted features for the clas­ and Flavia dataset. We have achieved a test accuracy of 98:54%,
sification, which makes this approach highly dataset-dependent, and it 92:22%, and 98:70% on Malayakew, Leafsnap and Flavia dataset
is evident from its performance on the different datasets. S-LeafNet respectively.
(Kumar et al., 2012) used binary segmented leaf image as input to five The main limitation of the proposed method is that while combining
CBR layers (Convolutional, Batch normalization, ReLU activation func­ the confidence scores, the accuracies of the individual classifiers are not
tional layers) as well as five pooling layers for the leaf image classifi­ taken into consideration. This gives undue advantage to classifiers that
cation. W-LeafNet (Kumar et al., 2012) used whole colored leaf image as have lower accuracy, and hence, may be disadvantageous to the more
input to seven CBR layers and a pooling layer. P-LeafNet (Kumar et al., accurate classifiers. This results in some classification errors. Alterna­
2012) used patched leaf images to classify the leaf images, this approach tively, instead of the simple arithmetic mean used to calculate the value
relies heavily on the vein structure of leaves. DeepPlant (Lee et al., 2015) of u1, i, we use a weighted mean where the weights take into consider­
used two feature learning streams, taking both the whole leaf image and ation the accuracy of the respective classifiers. However, these weights
patched leaf image as input to the network. This model basically used a need to be optimised properly as they may interfere with the inversion
two-stream CNN model. criterion itself. Further, in our case, these weights need to be predicted
In Table 6, the performance comparison of different methods on by the neural network itself which may result in vanishing gradient
Leafsnap dataset has been shown. It is observed that LeafSnap proposed during the second tire of the training.
by kumar et al., which was designed for this dataset, performs better in -Besides trying to rectify the limitations mentioned above, we plan to
this dataset than the former. MSF-CNN proposed by Hu et al., encode apply this method on other leaf image datasets to prove its robustness.
leaf features by fusion of leaf image at multiple scales. This model per­
forms descent on the LeafSnap dataset. Song et al. used Attention based
Declaration of Competing Interest
CNN (Song et al., 2019) to classify leaf images.
Table 7 provides classification accuracy of famous existing methods
None.
and of the proposed method to give clear picture of how our method
outperforms other major leaf identification techniques on the Flavia
dataset. Acknowledgement
The proposed BLeafNet uses a five-stream model, which enables it to
learn from diverse errors, and in turn, improves the performance of the We would like to thank the Center for Microprocessor Applications
model. Most multi-stream networks used traditional aggregation oper­ for Training Education and Research (CMATER) research laboratory of
ators like weighted mean or normal mean. BLeafNet uses Bonferroni the Computer Science and Engineering Department, Jadavpur Univer­
mean operator which finds better interconnection among the confidence sity, Kolkata, India for providing us the infrastructural support.
scores of different models and enables the proposed model to perform
better. It is evident BLeafNet delivers decent performance on the References
Malayakew, Leafsnap and Flavia datasets, hence we can conclude this is
Akter, R., Hosen, M.I., 2020. Cnn-based leaf image classification for bangladeshi
a robust method for the leaf image classification. medicinal plant recognition. In: 2020 Emerging Technology in Computing,
Communication and Electronics (ETCCE), pp. 1–6.
4.5. Error analysis Arribas, J.I., Sánchez-Ferrero, G.V., Ruiz-Ruiz, G., Gómez-Gil, J., 2011. Leaf classification
in sunflower crops by computer vision and neural networks. Comput. Electron.
Agric. 78 (1), 9–18.
The main limitation of the Bonferroni mean operator is that while Banerjee, A., Singh, P.K., Sarkar, R., 2020. Fuzzy integral based cnn classifier fusion for
combining the confidence scores, we do not take into consideration the 3d skeleton action recognition. In: IEEE Transactions on Circuits and Systems for
Video Technology.
accuracy of the individual classifiers which are part of the combination Beikmohammadi, A., Faez, K., 2018. Leaf classification for plant recognition with deep
process. This gives rise to a problem that we give undue advantage to transfer learning. In: 2018 4th Iranian Conference on Signal Processing and
classifiers that have lower accuracy, and do not give advantage to more Intelligent Systems (ICSPIS), pp. 21–26.
Beikmohammadi, A., Faez, K., 2019. Swp-leaf net: a novel multistage approach for plant
accurate classifiers. This would result in classification errors. In other
leaf identification based on deep learning. Comput. Electron. Agric. Submitted, 10.
words, instead of the simple arithmetic mean used to calculate the value Bhowal, P., Banerjee, D., Malakar, S., Sarkar, R., 2021. A two-tier ensemble approach for
of u1, i, we can use a weighted mean where the weights take into writer dependent online signature verification. J. Ambient. Intell. Humaniz. Comput.
consideration the accuracy of the respective classifiers. However, these 1–20.
Camargo Neto, J., Meyer, G., Jones, D., Samal, A., 2006. Plant species identification
weights need to be optimised properly as they may interfere with the using elliptic fourier leaf shape analysis. In: Computers and Electronics in
inversion criterion itself. Moreover, in our case, these weights need to be Agriculture, 50, pp. 121–134.

10
S. Ganguly et al. Ecological Informatics 69 (2022) 101585

Carneiro, G., Pádua, L., Sousa, J.J., Peres, E., Morais, R., Cunha, A., 2021. Grapevine Priya, C.A., Balasaravanan, T., Thanamani, A.S., 2012. An efficient leaf recognition
variety identification through grapevine leaf images acquired in natural algorithm for plant classification using support vector machine. In: International
environment. In: 2021 IEEE International Geoscience and Remote Sensing Conference on Pattern Recognition, Informatics and Medical Engineering (PRIME-
Symposium IGARSS, pp. 7055–7058. 2012), pp. 428–432.
Chaudhury, A., Barron, J.L., 2020. Plant species identification from occluded leaf images. Ray, B., Garain, A., Sarkar, R., 2021. An ensemble-based hotel recommender system
IEEE/ACM Transact. Comput. Biol. Bioinformat. 17 (3), 1042–1055. using sentiment analysis and aspect categorization of hotel reviews. Appl. Soft
Christenhusz, M., Byng, J., 2016. The number of known plant species in the world and its Comput. 98, 106935.
annual increase. Phytotaxa 261, 201–217. Saleem, G., Akhtar, M., Ahmed, N., Qureshi, W., 2019. Automated analysis of visual leaf
Du, C., Gao, S., 2017. Image segmentation-based multi-focus image fusion through multi- shape features for plant classification. Comput. Electron. Agric. 157, 270–280.
scale convolutional neural network. IEEE Access PP, 1. Sanyal, R., Kar, D., Sarkar, R., 2021. Carcinoma type classification from high-resolution
Esgario, J.G., Krohling, R.A., Ventura, J.A., 2020. Deep learning for classification and breast microscopy images using a hybrid ensemble of deep convolutional features
severity estimation of coffee leaf biotic stress. Comput. Electron. Agric. 169, 105162. and gradient boosting trees classifiers. IEEE/ACM Transact. Comput. Biol.
Figueroa-Mata, G., Mata-Montero, E., 2020. Using a convolutional Siamese network for Bioinformatics, 2021.
image-based plant species identification with small datasets. Biomimetics 5 (1). Satti, V., Satya, A., Sharma, S., 2013. An automatic leaf recognition system for plant
Gwo, C.-Y., Wei, C.-H., 2013. Plant identification through images: using feature identification using machine vision technology. Int. J. Eng. Sci. Technol. (IJEST) 5,
extraction of key points on leaf contours. In: Applications in Plant Sciences, 1, p. 11. 874–879.
He, K., Zhang, X., Ren, S., Sun, J., June 2016. Deep residual learning for image Singh, K., Gupta, I., Gupta, S., 2010. Svm-bdt pnn and fourier moment technique for
recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern classification of leaf shape. Int. J. Signal Process. Image Process. Pattern Recognit. 3,
Recognition (CVPR). 67–78.
Herdiyeni, Y., Kusmana, I., 2013. Fusion of Local Binary Patterns Features for Tropical Song, Y., He, F., Zhang, X., 2019. To identify tree species with highly similar leaves based
Medicinal Plants Identification, pp. 353–357. on a novel attention mechanism for cnn. IEEE Access 7, 163277–163286.
Hsiao, J.-K., Kang, L.-W., Chang, C.-L., Lin, C.-Y., 2014. Comparative Study of Leaf Image Sujith, A., Aji, S., 2020. An optimal feature set with lbp for leaf image classification. In:
Recognition with a Novel Learning-Based Approach, pp. 389–393. 2020 Fourth International Conference on Computing Methodologies and
Hu, J., Chen, Z., Yang, M., Zhang, R., Cui, Y., 2018. A multiscale fusion convolutional Communication (ICCMC), pp. 220–225.
neural network for plant leaf recognition. IEEE Signal Process. Lett. 25 (6), 853–857. Tsolakidis, D.G., Kosmopoulos, D.I., Papadourakis, G., 2014. Plant leaf recognition using
Kadir, A., 2014. A Model of Plant Identification System Using Glcm, Lacunarity and Shen zernike moments and histogram of oriented gradients. In: Likas, A., Blekas, K.,
Features. Kalles, D. (Eds.), Artificial Intelligence: Methods and Applications. Springer
Kan, H., Jin, L., Zhou, F., 2017. Classification of medicinal plant leaf image based on International Publishing, Cham, pp. 406–417.
multi-feature extraction. Pattern Recog. Image Anal. 27 (3), 581–587. Wang, J., He, J., Han, Y., Ouyang, C., Li, D., 2013. An adaptive thresholding algorithm of
Kumar, N., Belhumeur, P.N., Biswas, A., Jacobs, D.W., Kress, W.J., Lopez, I.C., Soares, J. field leaf image. Comput. Electron. Agric. 96, 23–39.
V.B., 2012. Leafsnap: A computer vision system for automatic plant species Wu, S.G., Bao, F.S., Xu, E.Y., Wang, Y., Chang, Y., Xiang, Q., 2007. A leaf recognition
identification. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (Eds.), algorithm for plant classification using probabilistic neural network. CoRR abs/
Computer Vision – ECCV 2012. Springer Berlin Heidelberg, Berlin, Heidelberg, 0707.4289.
pp. 502–516. Xiao, X.-Y., Hu, R., Zhang, S.-W., Wang, X.-F., 2010. Hog-based approach for leaf
Lavania, Shubham, Matey, Palash Sushil, 2014. Leaf Recognition using Contour Based classification. In: Huang, D.-S., Zhang, X., García, C.A. Reyes, Zhang, L. (Eds.),
Edge Detection and Sift Algorithm. Advanced Intelligent Computing Theories and Applications. With Aspects of
Lee, S.H., Chan, C.S., Wilkin, P., Remagnino, P., 2015. Deep-Plant: Plant Identification Artificial Intelligence. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 149–155.
with Convolutional Neural Networks. Zhang, J., Xie, Y., Wu, Q., Xia, Y., 2019. Medical image classification using synergic deep
Liu, J., Yang, S., Cheng, Y., Song, Z., 2018. Plant leaf classification based on deep learning. Med. Image Anal. 54, 10–19.
learning. In: 2018 Chinese Automation Congress (CAC), pp. 3165–3169. Zhao, A., Tsygankov, D., Qiu, P., 2017. Graph-Based Extraction of Shape Features for
Mettripun, N., 2020. Thai herb leaves classification based on properties of image regions. Leaf Classification, pp. 663–666.
In: 2020 59th Annual Conference of the Society of Instrument and Control Engineers
of Japan (SICE), pp. 372–377.

11