Received 7 July 2023, accepted 30 July 2023, date of publication 10 August 2023, date of current version 16 August 2023.

Digital Object Identifier 10.1109/ACCESS.2023.3304242

Deep Learning-Based Multi-Modal Ensemble

Classification Approach for Human Breast
Cancer Prognosis
EHTISHAM KHAN JADOON 1 , FIAZ GUL KHAN 1, SAJID SHAH 2, AHMAD KHAN 1,

AND MUHAMMED ELAFFENDI 2

1 Department of Computer Science, COMSATS University Islamabad, Abbottabad Campus, Abbottabad 22010, Pakistan
2 EIAS Data Science Laboratory, College of Computer and Information Sciences, Prince Sultan University, Riyadh 11586, Saudi Arabia
Corresponding author: Fiaz Gul Khan (fiazkhan@[Link])
This work was supported by the Emerging Intelligent Autonomous Systems (EIAS) Data Science Laboratory, Prince Sultan University,
Saudi Arabia.

ABSTRACT Ensemble models based on deep learning have made significant contributions to the medical
field, particularly in the area of disease prediction. Breast cancer is a highly aggressive disease with a
high mortality rate. Timely and effective prediction of breast cancer can reduce the risk of it progressing
to later stages and the need for unnecessary medications. While previous studies have focused on predicting
breast cancer using single-modal datasets, multi-modal datasets that include gene expression (gene exp),
clinical, and copy number variation (CNV) data have become available in recent years for predictive model
development. However, despite multiple studies using multi-modal data for disease prediction, models
designed for breast cancer are typically homogeneous neural networks. This article proposes a heterogeneous
deep learning-based ensemble model for effective breast cancer prediction using multi-modal data. The
model consists of three phases: feature extraction, stacked feature set creation, and using extracted features
as input for a stacked-based model using a random forest algorithm for effective prediction. For feature
extraction, convolutional neural networks (CNNs) are used for clinical and gene expression data, and deep
neural networks (DNNs) are used for CNV data. The extracted features from CNNs and DNNs are stacked
to create a comprehensive feature set. The simulation results demonstrate the superiority of the proposed
framework in terms of accuracy compared to uni-modal and homogeneous model-multi-modal frameworks.

INDEX TERMS Breast cancer, deep learning, feature extraction, machine learning, prognosis prediction.

I. INTRODUCTION tissues, whereas non-invasive breast cancer is confined to

Breast cancer is a highly lethal disease that predominantly the milk ducts or lobules of the breast [3], as illustrated in
affects women, with a significant mortality rate [1]. The Figure 1. Although breast cancer is more commonly diag-
condition is characterized by abnormal cell growth in the nosed in females, it can also affect males, but at a lower
breast tissue, resulting in the formation of primary tumors, frequency [4].
which can be either benign or malignant. Benign tumors According to the report of [Link] (ASCO website) an
are confined to the affected area and do not spread, while estimated 2,261,419 new cases were diagnosed with breast
malignant tumors can invade and affect other body parts. cancer around the globe in 2020. According to the data,
Breast cancer is classified into two types: invasive and non- in the United States, approximately 287,850 women will
invasive [2]. Invasive breast cancer can spread to neighboring receive a diagnosis of invasive breast cancer, while about
51,400 women will be impacted by non-invasive breast can-
The associate editor coordinating the review of this manuscript and cer. Approximately, 43,580 death cases were forecast for the
approving it for publication was Gustavo Callico . year 2022 due to breast cancer in the United States. The

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

85760 For more information, see [Link] VOLUME 11, 2023
E. K. Jadoon et al.: Deep Learning-Based Multi-Modal Ensemble Classification Approach

FIGURE 1. a) Anatomy of female breast b) Noninvasive cancer c) Invasive cancer.

survival rate is classified as long-term survival i-e more than models through stacking. Our study targets to improve
5 years and short-term survival i-e less than 5 years [3]. breast cancer prognosis by identifying novel prognostic
According to Cancer.Net1 report, the 5-year survival rate for factors that can help in the treatment of the disease.
non-invasive cases is 90 percent and the 10-year survival • The primary aim of the research is to develop het-
rate for women with non-invasive diagnosed breast cancer erogeneous stacking-based models that will enable the
is 84 percent. If invasive breast cancer is confined to the efficient prognosis of breast cancer. This is achieved
breast, the 5-year survival rate is 99 percent. But if it is spread through three main steps, namely feature extraction,
to lymph nodes the 5-year survival is 86 percent, and if it feature stacking, and classification algorithm.
spread to other parts of the body it becomes 29 percent. It is • The innovation of this proposed method lies in the use of
arduous for physicians to effectively diagnose and prognoses multiple deep learning models to extract features from a
invasive breast cancer due to its variety of clinical outcomes vast array of data modalities.
and complexity [3]. Initiating a prognosis is significant for the • The heterogeneous models are designed to extract highly
treatment of breast cancer. Firstly, a clear understanding of the informative features and produce more generalized
disease allows physicians and patients to make informed deci- results. The findings show that the proposed model
sions regarding treatment options tailored to the individual’s outperforms the current benchmarks for breast cancer
specific needs [5]. prognosis.
In the past decade, rapid advances in machine learning and • Our study has important clinical implications, as it can
deep learning open up the ways for medical fields. Various guide the development of new prognostic tools and
techniques and tools were proposed by the researcher in treatment strategies for breast cancer patients.
the medical domain for the identification of health-related The paper is organized in a way that Section II of the paper
issues [6], [7]. Machine learning demonstrates superior per- will provide an overview of existing methods and models,
formance in diagnosing and treating diseases that can lead while Section III will detail the methods and materials used
to death. On the other hand, deep learning is specifically in the proposed approach. Section IV will describe the exper-
developed to identify the most relevant features for predict- imental setup for the proposed model, and Section V will
ing disease [8], [9]. Research reports that deep learning is present the results, discussions, and conclusions.
capable of diagnosing cancer before revealing its symptoms
to patients with clinical standards [10]. II. RELATED WORK
Optimizing hyper-parameters for deep learning models can The related work can be studied in the following subsections:
improve training and prediction, leading to more effective
therapy plans for patients and doctors. Breast cancer prog- A. BREAST CANCER PROGNOSIS WITH SELECTIVE
nosis can be improved by using multi-modal data such as FEATURES
clinical data, gene expression data (gene-exp), copy number During past decades, with the fast implementation of gene
variation (CNV), and other genomics data [11] through the expression analysis, and microarray techniques, many con-
application of deep learning models. Currently, researchers tributions help to understand the molecular signatures of
have proposed a limited number of methods that use a breast cancer based on gene expression patterns in previous
homogeneous deep learning model applied to all relevant literature. Yixin wang and his colleague identify 76 gene
modalities. Ensemble models are considered the optimal tool expression signatures from 115 tumor tissues that can be
for improving performance and generalization [12], [13]. useable in the prediction of distant metastasis lymph nodes
keeping the point in view we proposed a model with the specifically negative breast cancer [14]. From the indepen-
following key characteristics: dent data of 171 lymph node-negative patients, the results
• The objective of this study is to create a breast cancer show 48% specificity and 93% sensitivity. As Breast cancer
prognosis method that utilizes a variety of deep-learning is a genetic disease, so for prediction and prognosis various

VOLUME 11, 2023 85761

 E. K. Jadoon et al.: Deep Learning-Based Multi-Modal Ensemble Classification Approach

TABLE 1. Overall METABRIC dataset information. work Wei Shao. Proposed a method for feature selection
for multi-modal data that were useful to identify the rela-
tionship between prognosis and prediction. The proposed
framework efficiently highlights the relationship between
prognosis and prediction by identifying the related features
from multi-modal data [22]. Breast cancer modeling has
proven to be a useful tool in comprehending the intricate
nature of tumor growth during treatment. In a similar vein,
the authors have introduced a mathematical model for breast
cancer that employs a system of differential equations with
researchers contribute with upgraded techniques that hold
piecewise constant arguments to investigate tumor growth
multi-modal data like clinical data, gene expr, and CNV to
and the effects of chemotherapy treatment. The authors have
enhance the prediction and prognosis accuracy [15].
validated their theoretical findings through numerical sim-
ulations [23]. Luis A. presented MultiSurv, a multi-modal
B. BREAST CANCER PROGNOSIS WITH MULTIMODAL method for predicting long-term survival in PAN cancer.
DATA USING MACHINE LEARNING AND DEEP LEARNING MultiSurv employs separate sub-models for each modality,
The multi-modal is becoming more prevalent in capturing including imaging, clinical, and other omics data, to establish
the complication of biological process, as it can unveil the feature representations. The sub-model outputs are merged
underlying interconnections [16]. To enhance breast can- to generate conditional survival predictions. The proposed
cer prognosis Sun. integrates multi-modalities (clinical and method can handle missing data and be evaluated on 33 types
genomic data). In their work, they proposed a new method of of cancer that show an accurate survival curve [24].
GPMLK of multi-kernel learning. GPMKL was trained for
130 selected pathological features out of 1990 and 20 features
of gene expression and it shows better results than the previ- C. BREAST CANCER PROGNOSIS WITH MULTIMODAL
ous contribution. As it holds limited data so the performance DATA USING ENSEMBLE DEEP LEARNING
of GPMKL is not efficient [17]. Whereas Shimizu. targeted Ensemble deep learning models have become increasingly
to design a novel prognosis score mPS that applies to a important in the medical field, particularly in disease diagno-
wide range of breast cancer patients. In the contribution, sis and prognosis. The EDLCDS-BCDC technique, as sug-
researchers identify 184 prognosis-related genes by training gested by the authors [25], utilizes ultrasound images for the
Random Forest and Neural Network. mPS system is tuned detection of breast cancer. To extract features, the VGG-16,
to 23 expression status and it is applicable where long rank VGG-19, and SqueezeNet models are employed in the pro-
P < 0.05 specifically independent of platform. It shows better posed approach. The final classification is achieved through
performance when the mPS system is integrated with clin- the application of Cat Swarm Optimization in combination
ical data and also provides value that can be used to avoid with a Multilayer Perceptron for the identification of breast
over-treatment [18]. cancer tissue. In [26], The authors incorporated a CNN into
Previous work relies on selected features of genomic their study to predict progression-free survival, pathological
data for breast cancer prognosis. But Sun et al. work on complete response, and residual cancer burden. They utilized
multi-dimensional data with a deep learning model. Where MRI scans, demographic data, and molecular subtypes as
the dense neural network is applied to multi-modal data inputs. The researchers evaluated three different procedures:
i-e CNV, gene expression, and clinical independently. The stacking, concatenation, and integration. The results demon-
output of DNN is fused in the middle layer. At the out- strated that integration outperformed the other two methods in
put, layer modalities were integrated with the score level addressing the concerned problem. The authors of this article
fusion at the final prediction results. It shows better results [27] have presented an approach that employs hybrid transfer
than the existing work with 79% accuracy [19]. Li Tong. learning with a modified VGG architecture, utilizing both 2D
Integrates the multi-omics data to predict breast cancer and 3D mammogram images in the dataset. The experimental
survival. In this work, researchers implement ConcatAE findings demonstrate that this proposed method achieves a
(concatenation autoencoder) that merges the feature that a prediction accuracy of 89.8%, surpassing the performance
model learns from each modality and CrossAE to gain an of existing methods. In this study, the authors suggest two
invariant representation of modality. Proposed models show automated approaches for breast cancer (BC) classifica-
remarkable performance on TCCA multi-omics data of breast tion that utilize a combination of the Whale Optimization
cancer [20]. Han introduced a Model Class Structure Based Algorithm (WOA) and Dragonfly Algorithm (DA) along
Deep Convolutional Neural Network (CSDCNN) that uses with Radial Basis Function Kernel Support Vector Machines
a nonlinear representation to efficiently classify breast can- (RBF-SVM). The proposed methods aim to enhance the accu-
cer into multiple categories. With the BreaKHis dataset, racy of BC classification by identifying the optimal SVM
which contains eight subclasses of breast cancer, CSDCNN parameters, validated on the WBCD dataset. The proposed
achieved an impressive accuracy of 93.2% [21]. In other model outperforms the benchmarks [28]. J. Gao introduced

85762 VOLUME 11, 2023

E. K. Jadoon et al.: Deep Learning-Based Multi-Modal Ensemble Classification Approach

FIGURE 2. The detailed architecture a) CNN model architecture for feature extraction b) DNN model architecture for feature
extractions.

TABLE 2. Feature selection detail. some limitations. Initially, researchers proposed models that
focused on a single modality for breast cancer progno-
sis. Vijver [33] introduced the first prognosis model that
was based on gene modality. However, relying solely on
gene expression is insufficient to effectively predict breast
cancer survivability. Incorporating additional sources of
information, such as clinical data, copy number variations,
and gene expression, can lead to more efficient prognosis
a novel framework called the Multimodal Graph Neural models. To address the limitations of unimodal architec-
Network, which utilizes gene expression, CNV, and patho- tures, researchers have proposed multi-modal architectures.
logical data to classify short and long-term survivals. The Sun et al. proposed one of the most effective multi-modal
framework constructs bipartite graphs between patients and architectures in which authors designed a Multimodal Deep
multimodal data and fuses all features at the final layer. Neural Network by integrating Multi-dimensional Data
With an accuracy of 94% and an AUC of 0.97 [29]. In [30], (MDNNMD) [19]. The authors utilized clinical data, copy
the author has proposed for breast cancer diagnosis using a number variations, and gene expression modality for the
snapshot ensemble using neural networks with the feature proposed model. DNN neural network is applied to all con-
reduction procedure using t-SNE that provides an efficient cerned data for feature extraction and then performs fuse
visualization and an accurate classification than the existing scoring. The result shows better performance than other
benchmarks. The authors proposed a multimodal adversarial prognosis models. [Link] proposes another breast cancer
representation learning approach for breast cancer prognosis prognosis model citearya2020multi, which is a dual-staged
prediction. The proposed approach integrates both clinical deep learning-based stacked ensemble model. This model
and imaging data to create a unified representation that can utilizes clinical data, copy number variations, and gene
accurately forecast the prognosis of breast cancer patients. expression datasets. In the first stage, CNNs are used for
Based on the experimental findings, this approach performs feature extraction, and in the second stage, the extracted
better than existing methods in terms of predictive accuracy features are inputted into the stacked ensemble model. The
[31]. Nikhilanand Arya developed a model for prognosis proposed model uses a homogeneous model-CNN for each
that uses multi-modal data which were trained on CNN with modality to extract features. However, the literature suggests
stacking. Random forest is applied to stacked files for breast that heterogeneity in ensemble deep learning is more effec-
cancer prognosis that give AUC = 0.93 and ACC = 90% [3]. tive and provides better generalization [34]. Our proposed
Liao et al. proposed a hybrid deep learning model that com- architecture utilizes a heterogeneous model for multi-modal
bines multiple modalities, including imaging and gene data. datasets to ensure a more accurate survivability model for
The researchers developed a fusion framework that utilized breast cancer prognosis.
feature selection networks for each modality and then merged The motivation behind the proposed work is to address the
the outputs using a weighted linear aggregation. The fused limitations of existing breast cancer prognosis models, which
features were then used to classify breast cancer subtypes, often rely on a single modality and may not provide accurate
achieving an accuracy of 88.07% [32]. predictions for patient survivability. The research gaps in the
The literature has pointed out that several breast cancer field have highlighted the need for multi-modal approaches
prognosis models have been successful in making accu- that incorporate diverse sources of information. The objective
rate predictions for patient survival, but they still have of the proposed work is to develop a heterogeneous model
VOLUME 11, 2023 85763
 E. K. Jadoon et al.: Deep Learning-Based Multi-Modal Ensemble Classification Approach

FIGURE 3. The systematic proposed framework.

TABLE 3. CNN tuned parameters for clinical and gene expression. the average survivability in months is 125.1 months. Long-
term survivors were labeled as ‘0’ and short-term survivors
were labeled as ‘1’. The overall METABRIC data set infor-
mation is given in Table 1. Out of 1980 patients, 64 patients
were lost to follow-up for five years, representing 3.23% of
the total samples. For our model, suspicious samples from
METABRIC were considered long-term survivors. Any miss-
ing values in the gene expression and copy number variation
dataset were imputed using a weighted nearest neighbor
algorithm [36]. Sun et al. have discretized and normalized the
gene expression data into three levels: under-expression (−1),
baseline (0), and over-expression (1) in correspondence to
Gevaert et al. [37]. The copy number variation-CNV dataset
has been used with five values [−2, −1, 0, 1, 2]. Normaliza-
tion of the clinical dataset is performed by using min-max
that leverages multiple modalities to improve the accuracy algorithm [38] in the range of [0,1] by considering the eq.(1).
of breast cancer prognosis models, ultimately leading to bet- xi,n − min(xi )
′
ter patient outcomes. By justifying the research gaps and xi,n = (nMax − nMin) + nMin (1)
max(xi ) − min(xi )
objectives, the proposed work aims to make a significant
contribution to the field of breast cancer research. 2) FEATURE SELECTION
Considering the nature of the dataset having the high dimen-
III. MATERIAL AND METHODS sional and low sample size (HDLSS) curse would cost the
1) DATASET efficiency of deep learning models [39]. In the proposed
The preprocessed METABRIC dataset has been directly architecture, multi-modal data consisting of copy number
used and is publicly available at: [Link] variation, gene expression, and clinical profiles are taken into
HIlab/MDNNMD. The METABRIC dataset comprises account. These data have 26,000, and 24,000 features for
(n = 1980) valid breast cancer patient records [35]. Patients CNV and gene expression modality. The clinical modality for
diagnosed with breast cancer have access to multi-modal each patient consists of 27 features, such as age at diagnosis,
data, including gene expr, clinical information, and CNV. cellularity, size, lymph node positivity, and others that can be
The samples were categorized into two cohorts: long-term seen in the provided link.1 To alleviate the curse of HDLSS,
survivors, who lived for more than 5 years after diagnosis, we used the well-known algorithm mRMR to reduce the
and short-term survivors, who lived for less than 5 years after dimensionality of our data [40], [41]. Feature extraction was
diagnosis. METABRIC consists of 1489 and 491 samples for
long-term survival and short-term survival respectively. The 1 [Link]
diagnosis median age for the available samples is 61 years and expression-profiles-metabric

85764 VOLUME 11, 2023

E. K. Jadoon et al.: Deep Learning-Based Multi-Modal Ensemble Classification Approach

TABLE 4. DNN tuned parameters for copy number variation. TABLE 5. Confusion metrics for the validation set.

TABLE 6. Class evaluation of heterogeneous stacked model.

the corresponding values of the input matrix and the filter

matrix.
In [3], The Glorot normal initializer is utilized to initialize
done incrementally using the mRMR algorithm, and the final the filter values. This approach involves selecting values that
selection was based on AUC values obtained from various have a mean of zero and standard deviation within a specific
cohorts. The model’s performance was evaluated using the range.
top 100 features, which were selected in the first iteration. s s
In subsequent iterations, the number of features increased to 2 2
200, 300, and finally, 500, and the model’s performance was − , (2)
ni , no ni , no
evaluated each time. The features with high AUC scores were
selected for the heterogeneous stacked RF model, and Table 2 The input and output values are denoted by ni and n0, respec-
provides a detailed description of these selected features. The tively. A constant seed value of 0.1 is utilized by both the
selection of features was based on their high AUC score, and CNN and DNN models. The stride rate for the convolutional
for CNV data, Gene expression profile, and clinical data, 200, layer is 2 which shifts the filter with the difference of 2 to
400, and 25 features were selected, respectively. perform convolution over the input matrix. Padding is also
added to convolution layers for the control of feature size.
After that flattened layer is used to flatten the output of the
IV. EXPERIMENTAL SETUP
convolutional layer then we passed the output from the dense
For our experimental configuration, we utilized ten-fold
layer with 150 hidden units. The regularization technique
cross-validation to assess the proposed framework, as in prior
Dropout [42], was applied for DNN after each layer to pre-
research [3]. Our dataset consisted of 1980 patients, which we
vent overfitting. Whereas L2 regularization is used for CNN
randomly divided into ten subsets. We combined nine of these
as it has vast applications in deep learning [43], [44]. The
subsets to form a training set, while one was reserved as the
activation function used at the convolutional layer is Tanh
testing set. Additionally, we further partitioned the merged
whereas the sigmoid activation function is used at the dense
training set into an 80% training subset and a 20% validation
layer. The detailed architecture of CNN and DNN is shown
subset Figure 3 shows the systematic proposed framework
in Figure 2.
where Keras 2.12.0, along with Tensorflow 2.12.0, is used
For model training, a recently proposed optimizer
for implementing the source code of the model.
Adam [45] was used. The parallel cross-entropy is used,
and work was upgraded utilizing that enhancer since it is
1) A CNN AND DNN-BASED PREDICTION FOR A computationally proficient. It has required a little memory
UNI-MODAL DATASET and is all around applied to those issues that are huge as far
The Heterogeneous stacked model proposed in this study as boundaries and additional information. A loss function is
utilizes both a convolutional neural network (CNN) and a used to measure or see how well a deep learning classifier
deep neural network (DNN) for predicting breast cancer in fits or suites to empirical data. In our case, since the problem
humans and extracting features for the subsequent phase is a binary classification problem, the ground truth (y) can
of the model. The proposed model comprises both CNN only have two states, which are one or zero. To evaluate the
and DNN and incorporates learnable filters directly on the prediction performance of the deep learner, we used binary
Multimodal sequences. The CNN is applied to clinical and cross-entropy loss in this study.
gene expression data, while the DNN is applied to the copy Finally, the CNN model is composed of a single convolu-
number variation dataset. The CNN and DNN are used at tional layer, a flattened layer, a dense layer, and an output
phase one for feature extraction for the next stage of the layer. On the other hand, the DNN comprises four dense
model while having certain numbers of filters. It generates layers with varying filter sizes and neurons. Each layer is
a feature map that is the output of the convolution and followed by a 50% dropout layer, and an output layer is
dense process. The feature mapping is achieved through present at the end. The AUC metric is evaluated for different
element-wise multiplication followed by addition between subgroup sizes ranging from 8 to 128, and the results show

VOLUME 11, 2023 85765

 E. K. Jadoon et al.: Deep Learning-Based Multi-Modal Ensemble Classification Approach

FIGURE 4. ROC curve for the heterogeneous stacked model having both CNN and DNN for stacked feature extraction for prognosis prediction a) AUC of
CNN for Gene expression b) AUC of DNN for CNV c) AUC of CNN for clinical.

FIGURE 7. Result evaluation of heterogeneous stacked model with

existing benchmarks.

FIGURE 5. AUC comparison of individual modality.

FIGURE 8. Heterogeneous model evaluation parameters.

FIGURE 6. Accuracy comparison of individual modality.

Phase 1: In phase 1, we train the CNN for clinical data at the

that a batch size of 8 yields the best performance. Hence, first layer, secondly CNV data is used to train a DNN at the
a batch size of 8 is chosen for the final model. The detailed second layer, and finally we train the CNN for the modality
parameter configurations of CNN and DNN are presented of Gene Expr data.
in Figure 3. Phase 2: In this phase, we shape a stacked feature set by
using extracted features of the output of CNN and DNN.
2) HETEROGENEOUS STACKED-BASED MODEL FOR Phase 3: Finally, we pass this stacked feature set to Random
MULTI-MODAL PREDICTION Forest (RF) algorithm for further classification. We evaluate
The proposed model is divided into three phases. These the performance of our proposed framework with the parame-
phases were described as under: ters given as Sensitivity, Specificity, Precision, and Accuracy.

85766 VOLUME 11, 2023

E. K. Jadoon et al.: Deep Learning-Based Multi-Modal Ensemble Classification Approach

TABLE 7. Heterogeneous stacked model results along with existing results were not as anticipated, primarily due to a high false
benchmarks.
positive rate that carries potential implications for patients.
To overcome the issue of variance caused by the limited
size of the dataset, we employed ten-fold cross-validation to
evaluate the proposed model. The dataset of 1980 patients
was divided into 10 subsets, and in each iteration, 9 subsets
were combined to form the training set while the remaining
subset was used as the testing set.
After analyzing the AUC the features from the individual
neural network are taken as stacked features. Where RF is
applied to the stacked features, it is evident from the liter-
ature that RF outperforms stacked features as compared to
other classifiers [17]. So we validate our model at the last
level using an RF classifier. Where we determine the con-
TABLE 8. Performance comparison of the proposed model with existing fusion metrics, Sensitivity, F1 score, Precision, Specificity,
benchmark.
and accuracy of the proposed model. The confusion metrics
for the validation set are shown in Table 3. The table pre-
sented below confirms that the proposed models are efficient
in predicting the samples. Out of a total of 495 samples
in the validation set, 485 samples were correctly predicted,
as shown in the table. There were only 10 false negative
predictions and no false positive predictions. The lack of false
Mathematically, these evaluation parameters are described as:
positives may be attributed to imbalanced classes. Whereas
Tp the class evaluation of the heterogeneous stacked model is
Sensitivity = (3)
Tp + Fn shown in Table 4. It is clear from the table below that the
Tn heterogeneous stacked model shows better results. It shows
Specificity = (4) the precision, recall, and F1 score for the binary classes.
Tn + Fn
Tp Where the long-term survival holds 1.00, 0.97, and 0.98 for
Precision = (5) precision, recall, and F1 score respectively. For short-term
Tp + Fp
Tp + Tn survival, the heterogeneous stacked model secures 0.92, 1.00,
Accuracy = (6) and 0.96 precision, recall, and F1 score respectively. The
Tp + Tn + Fp + Fn
stacked features are passed to the random forest where the
The true positive (TP), true negative (TN), false positive overall results of the model were evaluated with multiple
(FP), and false negative (FN) is used to calculate the model’s threshold values. The overall result evaluation metrics were
performance. Additionally, the area under the curve (AUC) accuracy, sensitivity, and specificity, where it secure 0.97,
value is computed using the receiver operating characteristic 0.92, and 1.0 respectively as depicted in Figure 8. It is evi-
(ROC) curve to evaluate the model’s efficiency. dent from our results that the heterogeneous stacked models
show better results than the existing benchmarks as shown
V. RESULTS AND DISCUSSION in Table 5.
This section presents a detailed comparison of our proposed
framework with existing benchmarks. B. COMPARISON OF HETEROGENEOUS STACKED MODEL
WITH EXISTING BENCHMARKS
A. HETEROGENEOUS STACKED MODEL PERFORMANCE It is evident from the result above that the heterogeneous
A heterogeneous stacked model is designed to combine the stacked model has improved the result compared to the
final feature extractions from the CNN and DNN models. The current benchmarks. We compare our proposed heteroge-
AUC metric, calculated from the ROC curve, is utilized as neous stacked model with popular prediction methods such
the performance evaluation criterion for the feature extraction as MDNNMD, Stacked RF-based ensemble model [3], [17].
models, in addition to the accuracy metric. The model with We compute the AUC values under the ROC for the het-
a higher AUC value is deemed more effective compared to erogeneous stacked-based model, in comparison to stacked
the one with a lower AUC value. In Figure 4, we present the based ensemble model, MDNNMD. The literature demon-
AUC values obtained from the ROC curve analysis. As the strates that stacked-based ensemble and Heterogeneous
graph depicted AUC for the CNN applied to gene exp and stacked-based methods outperform the other benchmarks.
clinical modality is 0.90 and 0.85 respectively. AUC for In comparison to the stacked RF model and MDNNMD,
DNN applied to CNV data modality is 0.72. Furthermore, the the heterogeneous stacked model had higher AUC and ACC
accuracies for CNN-gene Exp, DNN-CNV, and CNN-clinical values. In comparison, the stacked RF model gains 0.83, 0.92,
are 80.56%, 80.45%, and 81.21%, respectively. The AUC and 0.70 AUC values for CNN-Gene Exp, CNN-Clinical,

VOLUME 11, 2023 85767

 E. K. Jadoon et al.: Deep Learning-Based Multi-Modal Ensemble Classification Approach

and CNN-CNV respectively that are lesser than the het- [4] A. Gucalp, T. A. Traina, J. R. Eisner, J. S. Parker, S. R. Selitsky, B. H. Park,
erogeneous stacked RF as shown in Figure 5(a). Whereas A. D. Elias, E. S. Baskin-Bey, and F. Cardoso, ‘‘Male breast cancer: A dis-
ease distinct from female breast cancer,’’ Breast Cancer Res. Treatment,
Figure 5(b) demonstrates the individual modality accuracy vol. 173, no. 1, pp. 37–48, Jan. 2019.
in comparison to the existing benchmarks. The graph shows [5] C. Siotos, M. McColl, K. Psoter, R. C. Gilmore, M. E. Sebai,
that the heterogeneous stacked model shows better prediction K. P. Broderick, L. K. Jacobs, S. Irwin, G. D. Rosson, and M. Habibi,
‘‘Tumor site and breast cancer prognosis,’’ Clin. Breast Cancer, vol. 18,
on individual modality as compared to the existing prognosis no. 5, pp. 1045–1052, Oct. 2018.
model. [6] K. Rautela, D. Kumar, and V. Kumar, ‘‘A systematic review on breast
It is determined from the simulations that our heteroge- cancer detection using deep learning techniques,’’ Arch. Comput. Methods
neous stacking of neural networks achieves upto-mark results Eng., vol. 29, pp. 4599–4629, Apr. 2022.
[7] F. M. Calisto, N. Nunes, and J. C. Nascimento, ‘‘Modeling adoption
as compared to existing frameworks such as stacked RF and of intelligent agents in medical imaging,’’ Int. J. Hum.-Comput. Stud.,
MDNNMD as shown in table 7. The heterogeneous stacked vol. 168, Dec. 2022, Art. no. 102922.
model gains up-to-the-mark results for each modality trained [8] A. B. Nassif, M. A. Talib, Q. Nasir, Y. Afadar, and O. Elgendy,
‘‘Breast cancer detection using artificial intelligence techniques: A sys-
under different neural networks. The overall model shows tematic literature review,’’ Artif. Intell. Med., vol. 127, May 2022,
better results than the existing models. Compared to these Art. no. 102276.
benchmarks, our proposed framework achieves 0.97, 0.97, [9] W. El-Shafai, A. A. Mahmoud, A. M. Ali, E.-S. M. El-Rabaie, T. E. Taha,
O. F. Zahran, A. S. El-Fishawy, N. F. Soliman, A. A. Alhussan, and
0.98, 0.98 for accuracy, sensitivity, precision, and F1 score
F. E. Abd El-Samie, ‘‘Deep CNN model for multimodal medical image
respectively as shown in Figure 7. While comparing the dif- denoising,’’ Comput., Mater. Continua, vol. 73, no. 2, pp. 3795–3814,
ferent evaluation parameters it is validated that the proposed 2022.
model earned better results than the existing benchmarks as [10] E. Landhuis, ‘‘Deep learning takes on tumours,’’ Nature, vol. 580, no. 7804,
pp. 551–554, 2020.
shown in Table 8. [11] W. Zhu, L. Xie, J. Han, and X. Guo, ‘‘The application of deep learning in
cancer prognosis prediction,’’ Cancers, vol. 12, no. 3, p. 603, Mar. 2020.
VI. CONCLUSION [12] A. Petrakova, M. Affenzeller, and G. Merkurjeva, ‘‘Heterogeneous versus
homogeneous machine learning ensembles,’’ Inf. Technol. Manag. Sci.,
This article presents a novel approach to predicting breast vol. 18, no. 1, pp. 135–140, Jan. 2015.
cancer prognosis using ensemble deep learning. Unlike pre- [13] E. Mustafa, E. K. Jadoon, S. Khaliq-Uz-Zaman, M. A. Humayun, and
vious research, we utilized a stacked ensemble model with M. Maray, ‘‘An ensembled framework for human breast cancer survivabil-
ity prediction using deep learning,’’ Diagnostics, vol. 13, no. 10, p. 1688,
individual neural networks for feature extraction from differ-
May 2023.
ent data modalities. Our model incorporated CNN for feature [14] Y. Wang, J. G. Klijn, Y. Zhang, A. M. Sieuwerts, M. P. Look, F. Yang,
extraction of clinical data and gene expression data, and DNN D. Talantov, M. Timmermans, M. E. Meijer-van Gelder, J. Yu, T. Jatkoe,
for CNV data. In the second phase, the extracted features E. M. Berns, D. Atkins, and J. A. Foekens, ‘‘Gene-expression profiles to
predict distant metastasis of lymph-node-negative primary breast cancer,’’
were stacked and used as input for the RF for the classi- Lancet, vol. 365, no. 9460, pp. 671–679, Feb. 2005.
fication of short-term and long-term survivals. Our results [15] M. Khademi and N. S. Nedialkov, ‘‘Probabilistic graphical models and
demonstrate that our heterogeneous framework outperformed deep belief networks for prognosis of breast cancer,’’ in Proc. IEEE 14th
Int. Conf. Mach. Learn. Appl. (ICMLA), Dec. 2015, pp. 727–732.
homogeneous stacking and other benchmarks, achieving an
[16] S. R. Stahlschmidt, B. Ulfenborg, and J. Synnergren, ‘‘Multimodal deep
accuracy of 97%. It is important to note that further val- learning for biomedical data fusion: A review,’’ Briefings Bioinf., vol. 23,
idation or integration with other sources of information is no. 2, Mar. 2022, Art. no. bbab569.
necessary before utilizing the predicted output for clinical [17] D. Sun, A. Li, B. Tang, and M. Wang, ‘‘Integrating genomic data and
pathological images to effectively predict breast cancer clinical outcome,’’
decision-making. Additionally, our approach can be extended Comput. Methods Programs Biomed., vol. 161, pp. 45–53, Jul. 2018.
to predict other diseases with different algorithms and can [18] H. Shimizu and K. I. Nakayama, ‘‘A 23 gene-based molecular prognos-
include more data modalities for prognosis such as gene tic score precisely predicts overall survival of breast cancer patients,’’
EBioMedicine, vol. 46, pp. 150–159, Aug. 2019.
methylation and miRNA.
[19] D. Sun, M. Wang, and A. Li, ‘‘A multimodal deep neural network for
human breast cancer prognosis prediction by integrating multi-dimensional
ACKNOWLEDGMENT data,’’ IEEE/ACM Trans. Comput. Biol. Bioinf., vol. 16, no. 3, pp. 841–850,
May 2019.
The authors would like to acknowledge the support of Prince
[20] L. Tong, J. Mitchel, K. Chatlin, and M. D. Wang, ‘‘Deep learning based
Sultan University for paying the Article Processing Charges feature-level integration of multi-omics data for breast cancer patients
(APC) of this publication. survival analysis,’’ BMC Med. Informat. Decis. Making, vol. 20, no. 1,
pp. 1–12, Dec. 2020.
[21] Z. Han, B. Wei, Y. Zheng, Y. Yin, K. Li, and S. Li, ‘‘Breast cancer multi-
REFERENCES classification from histopathological images with structured deep learning
[1] G. Chugh, S. Kumar, and N. Singh, ‘‘Survey on machine learning and deep model,’’ Sci. Rep., vol. 7, no. 1, pp. 1–10, Jun. 2017.
learning applications in breast cancer diagnosis,’’ Cognit. Comput., vol. 13, [22] W. Shao, T. Wang, L. Sun, T. Dong, Z. Han, Z. Huang, J. Zhang, D. Zhang,
no. 6, pp. 1451–1470, Nov. 2021. and K. Huang, ‘‘Multi-task multi-modal learning for joint diagnosis and
[2] S. Kumar and S. Sharma, ‘‘Sub-classification of invasive and non-invasive prognosis of human cancers,’’ Med. Image Anal., vol. 65, Oct. 2020,
cancer from magnification independent histopathological images using Art. no. 101795.
hybrid neural networks,’’ Evol. Intell., vol. 15, no. 3, pp. 1531–1543, [23] F. B. Yousef, A. Yousef, T. Abdeljawad, and A. Kalinli, ‘‘Mathematical
Sep. 2022. modeling of breast cancer in a mixed immune-chemotherapy treatment
[3] N. Arya and S. Saha, ‘‘Multi-modal classification for human breast cancer considering the effect of ketogenic diet,’’ Eur. Phys. J. Plus, vol. 135,
prognosis prediction: Proposal of deep-learning based stacked ensem- no. 12, pp. 1–23, Dec. 2020.
ble model,’’ IEEE/ACM Trans. Comput. Biol. Bioinf., vol. 19, no. 2, [24] L. A. Vale-Silva and K. Rohr, ‘‘MultiSurv: Long-term cancer survival
pp. 1032–1041, Mar. 2022. prediction using multimodal deep learning,’’ MedRxiv, Aug. 2020.

85768 VOLUME 11, 2023

E. K. Jadoon et al.: Deep Learning-Based Multi-Modal Ensemble Classification Approach

[25] M. Ragab, A. Albukhari, J. Alyami, and R. F. Mansour, ‘‘Ensemble [44] M. Havaei, A. Davy, D. Warde-Farley, A. Biard, A. Courville, Y. Bengio,
deep-learning-enabled clinical decision support system for breast cancer C. Pal, P.-M. Jodoin, and H. Larochelle, ‘‘Brain tumor segmentation with
diagnosis and classification on ultrasound images,’’ Biology, vol. 11, no. 3, deep neural networks,’’ Med. Image Anal., vol. 35, pp. 18–31, Jan. 2017.
p. 439, Mar. 2022. [45] D. P. Kingma and J. Ba, ‘‘Adam: A method for stochastic optimization,’’
[26] H. Dammu, T. Ren, and T. Q. Duong, ‘‘Deep learning prediction of patho- 2014, arXiv:1412.6980.
logical complete response, residual cancer burden, and progression-free
survival in breast cancer patients,’’ PLoS ONE, vol. 18, no. 1, Jan. 2023,
Art. no. e0280148.
[27] A. Khamparia, S. Bharati, P. Podder, D. Gupta, A. Khanna, T. K. Phung, EHTISHAM KHAN JADOON received the B.S.
and D. N. H. Thanh, ‘‘Diagnosis of breast cancer based on modern mam- degree from the Abbottabad University of Sci-
mography using hybrid transfer learning,’’ Multidimensional Syst. Signal ence and Technology, and the M.S. degree
Process., vol. 32, no. 2, pp. 747–765, Apr. 2021. from COMSATS University Islamabad, Abbot-
[28] A. S. Elkorany, M. Marey, K. M. Almustafa, and Z. F. Elsharkawy, tabad Campus. His research interests include deep
‘‘Breast cancer diagnosis using support vector machines optimized by learning, machine learning, and neural networks.
whale optimization and dragonfly algorithms,’’ IEEE Access, vol. 10,
pp. 69688–69699, 2022.
[29] J. Gao, T. Lyu, F. Xiong, J. Wang, W. Ke, and Z. Li, ‘‘MGNN: A multi-
modal graph neural network for predicting the survival of cancer patients,’’
in Proc. 43rd Int. ACM SIGIR Conf. Res. Develop. Inf. Retr., 2020,
pp. 1697–1700.
[30] N. Sharma, K. P. Sharma, M. Mangla, and R. Rani, ‘‘Breast cancer clas-
FIAZ GUL KHAN received the master’s and Ph.D.
sification using snapshot ensemble deep learning model and t-distributed
degrees from Politecnico di Torino, Italy, in 2013.
stochastic neighbor embedding,’’ Multimedia Tools Appl., vol. 82, no. 3,
pp. 4011–4029, Jan. 2023.
He is currently an Associate Professor with the
Computer Science Department, COMSATS Uni-
[31] X. Du and Y. Zhao, ‘‘Multimodal adversarial representation learning
for breast cancer prognosis prediction,’’ Comput. Biol. Med., vol. 157, versity Islamabad, Abbottabad Campus, Pakistan.
May 2023, Art. no. 106765. His research interests include concurrent comput-
[32] T. Liu, J. Huang, T. Liao, R. Pu, S. Liu, and Y. Peng, ‘‘A hybrid ing, machine learning, artificial intelligence, and
deep learning model for predicting molecular subtypes of human breast GPU computing.
cancer using multimodal data,’’ IRBM, vol. 43, no. 1, pp. 62–74,
Feb. 2022.
[33] M. J. Van De Vijver, Y. D. He, L. J. Van’t Veer, H. Dai, A. A. Hart,
D. W. Voskuil, G. J. Schreiber, J. L. Peterse, C. Roberts, and M. J. Marton,
‘‘A gene-expression signature as a predictor of survival in breast cancer,’’ SAJID SHAH received the M.S. and Ph.D.
New England J. Med., vol. 347, no. 25, pp. 1999–2009, 2002. degrees from Politecnico di Torino, Italy. He was
[34] M. A. Ganaie, M. Hu, A. K. Malik, M. Tanveer, and P. N. Suganthan, an Assistant Professor with COMSATS Univer-
‘‘Ensemble deep learning: A review,’’ 2021, arXiv:2104.02395. sity Islamabad, Abbottabad Campus, Pakistan.
[35] A. Mukherjee, R. Russell, S.-F. Chin, B. Liu, O. M. Rueda, He is currently a Postdoctoral Researcher with
H. R. Ali, G. Turashvili, B. Mahler-Araujo, I. O. Ellis, S. Aparicio, the EIAS Laboratory, Prince Sultan Univer-
C. Caldas, and E. Provenzano, ‘‘Associations between genomic sity, Riyadh, Saudi Arabia. His research inter-
stratification of breast cancer and centrally reviewed tumour pathology ests include data mining, text mining, machine
in the METABRIC cohort,’’ Npj Breast Cancer, vol. 4, no. 1, pp. 1–9, learning, bioinformatics, and image processing.
Mar. 2018.
[36] B. Al-Helali, Q. Chen, B. Xue, and M. Zhang, ‘‘A new imputation method
based on genetic programming and weighted KNN for symbolic regres-
sion with incomplete data,’’ Soft Comput., vol. 25, no. 8, pp. 5993–6012,
Apr. 2021. AHMAD KHAN received the Ph.D. degree from
[37] O. Gevaert, F. D. Smet, D. Timmerman, Y. Moreau, and B. D. Moor, the National University of Computer and Emerg-
‘‘Predicting the prognosis of breast cancer by integrating clinical and ing Sciences (FAST-NU), Islamabad, Pakistan,
microarray data with Bayesian networks,’’ Bioinformatics, vol. 22, no. 14, in 2015. He is currently an Assistant Profes-
pp. 184–190, Jul. 2006. sor with the Department of Computer Science,
[38] D. Singh and B. Singh, ‘‘Investigating the impact of data normalization COMSATS University Islamabad (CUI), Abbot-
on classification performance,’’ Appl. Soft Comput., vol. 97, Dec. 2020,
tabad Campus. His research interests include com-
Art. no. 105524.
puter vision, machine learning, and evolutionary
[39] S. L. Verghese, I. Y. Liao, T. H. Maul, and S. Y. Chong, ‘‘An empirical
algorithms.
study of several information theoretic based feature extraction methods for
classifying high dimensional low sample size data,’’ IEEE Access, vol. 9,
pp. 69157–69172, 2021.
[40] M. Van Gerven and S. Bohte, ‘‘Artificial neural networks as models of MUHAMMED ELAFFENDI is currently a Pro-
neural information processing,’’ Frontiers Media Ltd., London, U.K., Tech. fessor of computer science with the Department
Rep., 2017.
of Computer Science, College of Computer and
[41] Y. Eroǧlu, M. Yildirim, and A. Çinar, ‘‘Convolutional neural networks
Information Sciences (CCIS), Prince Sultan Uni-
based classification of breast ultrasonography images by hybrid method
versity, Riyadh, Saudi Arabia. He is the Founder
with respect to benign, malignant, and normal using mRMR,’’ Comput.
Biol. Med., vol. 133, Jun. 2021, Art. no. 104407. and the Director of the Center of Excellence in
[42] J. Roder, C. Oliveira, L. Net, M. Tsypin, B. Linstid, and H. Roder, CyberSecurity (CYBEX) and the EIAS Data Sci-
‘‘A dropout-regularized classifier development approach optimized for ence Research Laboratory, AIDE to the Rector, the
precision medicine test discovery from omics data,’’ BMC Bioinf., vol. 20, Director of the Institutional Policy and Develop-
no. 1, pp. 1–14, Dec. 2019. ment Unit, and the Former Dean of CCIS. For the
[43] V. A. Kumar, S. Gupta, S. S. Chandra, S. Raman, and S. S. Channappayya, past few years, he has been performing research in the areas of machine
‘‘No-reference quality assessment of tone mapped high dynamic range learning, natural language processing, emerging distributed architectures,
(HDR) images using transfer learning,’’ in Proc. 9th Int. Conf. Quality next generation computer systems, and complexity science.
Multimedia Exper. (QoMEX), May 2017, pp. 1–3.

VOLUME 11, 2023 85769