Received 1 September 2024, accepted 24 September 2024, date of publication 30 September 2024,

date of current version 28 October 2024.

Digital Object Identifier 10.1109/ACCESS.2024.3470850

Repeatability of Fine-Tuning Large Language

Models Illustrated Using QLoRA
SAEED S. ALAHMARI1 , (Member, IEEE), LAWRENCE O. HALL 2 , (Life Fellow, IEEE),
PETER R. MOUTON 2,3 , AND DMITRY B. GOLDGOF 2 , (Life Fellow, IEEE)
1 Department of Computer Science, Najran University, Najran 66462, Saudi Arabia
2 Department of Computer Science and Engineering, University of South Florida, Tampa, FL 33620, USA
3 SRC Biosciences, Tampa, FL 33606, USA
Corresponding author: Saeed S. Alahmari (ssalahmari@[Link],saeed3@[Link])
This work was supported in part by the Deanship of Graduate Studies and Scientific Research-Najran University Easy Funding Program
under Grant NU/EFP/SERC/13/29; and in part by the National Science Foundation under Award 1746511, Award 1926990, and Award
1513126.

ABSTRACT Large language models (LLMs) have shown progress and promise in diverse applications
ranging from the medical field to chat bots. Developing LLMs requires a large corpus of data and significant
computation resources to achieve efficient learning. Foundation models (in particular LLMs) serve as the
basis for fine-tuning on a new corpus of data. Since the original foundation models contain a very large
number of parameters, fine-tuning them can be quite challenging. Development of the low-rank adaption
technique (LoRA) for fine-tuning, and the quantized version of LoRA, also known as QLoRA, allows for
fine-tuning of LLMs on a new smaller corpus of data. This paper focuses on the repeatability of fine-tuning
four LLMs using QLoRA. We have fine-tuned them for seven trials each under the same hardware and
software settings. We also validated our study for the repeatability (stability) issue by fine-tuning LLMs
on two public datasets. For each trial, each LLM was fine-tuned on a subset of the dataset and tested on a
holdout test set. Fine-tuning and inference were done on a single GPU. Our study shows that fine-tuning of
LLMs with the QLoRA method is not repeatable (not stable), such that different fine-tuned runs result in
different performance on the holdout test set.

INDEX TERMS QLoRA, LoRA, LLMs, deep learning, repeatability, foundation models, fine-tuning.

I. INTRODUCTION summarization, translation, and classification such as T5 [4],

Large Language Models (LLMs) developed in the last few SwitchTransformers [5], ChatGPT [6], and Llama2 [7]. These
years have shown human-level performance for some tasks LLM models are created with self-supervised training on a
such as dialogue-based chat bots [1]. The main building data corpus followed by alignment using methods such as
block of LLMs is the transformer architecture [2] that Reinforcement Learning with Human Feedback (RLHF) [8].
enables deep neural networks to learn from sequences Fine-tuning LLMs is an efficient approach to improve
of data more efficiently than Long Short-term Memory LLMs’ performance [9], [10], [11], especially for datasets
(LSTM) recurrent neural networks [3]. As a result, many of a new domain, or to improve LLMs behaviour or
companies have started the race to develop LLMs using a remove undesirable behaviour [11]. A current downside
corpus of data freely available on the internet. Some of the for fine-tuning LLMs is the need for high amounts of
developed LLMs hold promise as AI assistants that are able to memory and computational costs, primarily available to large
interact efficiently with humans through dialogue-based chat technology companies. To overcome this obstacle, different
sessions. Other promising LLMs have been developed for text approaches have been proposed to reduce pre-trained model
sizes for fine-tuning such as low-rank adaptation (LoRA)
The associate editor coordinating the review of this manuscript and [12] and quantized low-rank adaptation (QLoRA) [13].
approving it for publication was Daniel Augusto Ribeiro Chaves . By modifying the pre-trained model’s weight matrix these
2024 The Authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
VOLUME 12, 2024 For more information, see [Link] 153221
 S. S. Alahmari et al.: Repeatability of Fine-Tuning LLMs Illustrated Using QLoRA

FIGURE 1. An illustration of the difference between repeatability, replicability, and reproducibility.

approaches can effectively reduce memory requirements such as the GPU operation for enhancing the computation
without a loss of performance. speed (atomic addition operation). Other factors such as
Repeatability is a critical consideration for maintaining the weight initialization randomization, order of data by a data
performance and behaviour of LLM fine-tuning. Repeatabil- loader and regularization methods can effect repeatability of
ity is considered the ability to obtain the same performance training/fine-tuning of deep learning models. However, the
after retraining (re-fine-tuning) the model multiple times on latter randomization can be disabled by seeding or setting
same hardware with the same software settings [14], [15]. determinism options using the deep learning development
In this paper, we examine the repeatability of fine-tuning tools [16], [17], [18].
LLMs using the state-of-the-art low rank adaption (i.e. Prior work has investigated the non-determinism of
QLoRA). For this purpose, we fine-tuned two LLMs using training deep neural networks and the impact of development
a quantized low rank adaption approach on a single GPU and tools and the hardware. Zhuang et al. studied the impact
report the results of the models on a separate (unseen) held- of development tools and hardware in the training of deep
out test set for each fine-tuned trial. Our contribution can be neural networks determinism [19]. An experimental study by
summarized as follows: Summers et al. was done where the authors experimentally
• A study of repeatability using LLM fine-tuning using studied the effect of different parameters and settings on
QLoRA is done for the first time. the determinism of training deep neural networks [20].
• Comparison of repeatability of LLMs was done using Such parameters and settings include: weight initialization,
four LLMs: GPT-NeoX-20B, GPT-NeoXT-chat-20B, data augmentation, data shuffling, and stochastic regular-
Llama2-7B and Llama2-chat-7B. ization. The authors proposed two solutions for addressing
• Validation of the repeatability of LLMs using two public the non-deterministic training of deep neural networks:
datasets. accelerated ensemble; and test-time augmentation (TTA).
In the following sections, we provide an overview of related Similarily, non-repeatable (non-deterministic) training of
previous studies of repeatability. We discuss the dataset neural networks was studied in [21]. The authors proposed
and the four LLMs picked for our study. Furthermore, minimum entropy regularizer to increase neural network
we explain the QLoRA concept and provide our fine-tuning model confidence. Another approach for reducing the effect
methodology and results. To the best of our knowledge this of non-repeatable results of training deep neural networks
is the first direct assessment of the repeatability of fine-tuned was proposed by Lemay et al. [22]. This approach uses
LLMs using the QLoRA approach. sampling Monte Carlo dropout predictions at test time
to reduce variations in deep neural network predictions.
II. RELATED WORK An assessment of repeatability of training deep neural
Repeatability is a critical consideration for maintaining the networks was done by Alahmari et al. [15]. After training
performance and behaviour of LLM fine-tuning. Repeata- two deep neural networks for segmentation and classification
bility refers to the ability to obtain the same results using the same hardware and software settings for multiple
by retraining or re-fine tuning of a model by the same trials, these authors reported non-deterministic results when
team multiple times on the same hardware and software training deep neural networks.
settings [14], [15]. As previously shown, repeatability Dodge et al. studied weight initialization, data order
remains a major challenge for deep learning models [15]. and early stopping impact on fine-tuning BERT on four
Repeatability requires higher precision computation due to datasets from the GLUE benchmark [23]. Prior to assess-
randomization in the deep learning training and fine-tuning ing performance, the authors chose different seeds for

153222 VOLUME 12, 2024

S. S. Alahmari et al.: Repeatability of Fine-Tuning LLMs Illustrated Using QLoRA

weight initialization and data shuffling. Since they allowed and different hardware and software setup for multiple trials.
randomization in weight initialization and data shuffling, Reproducibility refers to obtaining the same exact results
different runs provided different results. Mosbach et al. with a model trained by a different team in a different
assessed fine-tuning stability of BERT [24]. Their findings location but with the same hardware and software settings
showed that the catastrophic forgetting and fine-tuning in for multiple runs. Figure 1 shows a visual illustration of the
small datasets are not the cause of instability in fine- differences between the repeatability, replicability, and the
tuning LLMs [23], [25], [26]. They suggested the use reproducibility.
of small learning rates and an increase of number of
epochs (iterations) for fine-tuning. However, the results V. LARGE LANGUAGE MODELS
showed a small variance between different runs of fine- In this section we describe the large language models (LLM)
tuning, which indicates the fine-tuning of LLMs is not which we have used for fine-tuning on the datasets described
repeatable. Thus, substantial challenges remain with regard in Section III.
to generating repeatable, reproducible, and replicable results The first LLM we have used is GPT-NeoX-20B from
when fine-tuning models in deep learning. EleutehrAI. This model is a 20 billion parameter transformer
autoregressive decoder language model that was pre-trained
III. DATASETS on the Pile dataset [38]. This LLM was inspired by GPT-3 [1],
The datasets for our experiments are the English quotes with some changes such as the use of rotary embeddings [39]
dataset and Open-Platypus dataset. The English quotes instead of learnable positional embedding [40], the use of
dataset was originally retrieved from Goodreads quotes1 , parallelism in computing the attention and feed-forward
and can be used for multi-label text classification and text layers rather than running them in series, initialization of
generation [27], [28]. The dataset consists of three columns feed-forward layers was done based on the Wang et al.
quote, author, and tags. The quote column contains the quote method [41], and the use of dense layers instead of
text in English, and the author column contains the name of sparse layers used in GPT-3 to reduce complexity. This
the author, and the tags column contains keywords describing LLM is based on 44 layers. The Pile is an open-source
the quote. The dataset contains 2508 quotes. We have English language modeling dataset that contains different
processed the dataset to remove author and tags columns, high-quality datasets constructed and derived from academic
while keeping the quote column. Furthermore, we have split or professional sources. The motivation of using GPT-NeoX-
the dataset into train (80%) and test (20%) while seeding the 20B is because it is an open sourced autoregressive LLM with
split function to ensure the same split is used for each run. code and weights publicly available.
The dataset is also publicly available on HuggingFace2 . The second model we have used is GPT-NeoXT-chat-20B
The Open-Platypus dataset was used for improving LLM which is based on fine-tuning the EluteherAI GPT-NeoX-20B
logical reasoning and has been used in developing Platypus2 model using instructions in dialogue-based interaction. The
models [29]. This dataset is comprised of multiple datasets developer of GPT-NeoXT-chat-20B focused on tasks such
of math and science which have been processed to remove as text summarizing, content extraction, question answering,
redundancy and similarity using an 80% threshold [30], and text classification [42].
[31], [32], [33], [34], [35], [36]. This dataset contains The third LLM is Meta Llama-2, which is an updated
24900 rows which includes instructions and outputs. We have version of Llama-1 with a number of differences such as
processed the datasets to include the needed tags for each training on a new dataset of publicly available data, where
LLM as detailed in Section VII. Furthermore we have split the size of the training corpus is 40% larger than the
the dataset into train (80%) and test (20%), while seeding one used for Llama-1 [7]. Furthermore, Llama-2 used a
the split function to ensure the same split is returned for double content length and grouped-query attention (GQA)
each fine-tuning trial. The dataset is publicly available on [43]. Llama-2 comes in different sizes such as Llama2-7B,
HuggingFace3 . Llama2-13B, Llama34B, and Llama2-70B which is based on
the number of parameters in the LLM (i.e., indicating the
IV. REPEATABILITY size). The training of Llama2 was based on the approach
Repeatability is defined as the ability to obtain the same proposed by Touvron et al. [7] with an optimized auto-
results with a model trained by the same team with regressive transformer. The training of Llama-2 was done
the same precision in the same location with the same on 2 trillion tokens of cleaned data, where training used
hardware and software settings [15], [37]. It is worth noting standard transformer architecture [2] with pre-normalization
that replicability and reproducibility are not the same as using RMSNorm [44], activation function was SwiGLU [45].
repeatability. Replicability is the ability to obtain exact same In this paper, we have used Llama2-7B LLM model for
results of a model by a different team with different algorithm assessing repeatability of fine-tuning for text generation using
the English quotes dataset.
1 [Link] The fourth LLM we have used in this paper is Llama2-chat-
2 [Link] 7B by Meta. This model was developed for dialogue-based
3 [Link] chatting based on the Llama2 model. This model was tested

VOLUME 12, 2024 153223

 S. S. Alahmari et al.: Repeatability of Fine-Tuning LLMs Illustrated Using QLoRA

settings for three epochs. The QLoRA approach freezes the

pre-trained model W0 where W0 ∈ Rd×k , and decompose the
back-propagation weight matrix 1W into two matrices A and
B, where A ∈ Rd×r , and B ∈ Rr×k with the rank hyper-
parameter r. Note, W0 is frozen and no gradient updates
are applied to it, and A and B receive the gradient updates.
During a forward-pass the following operation is applied
h = W0 x + ABx, where h is the hidden representation and
x is the input data. At the end of each fine-tuning trial, the
QLoRA fine-tuned adapter weights 1Wi , where i represents
the trial number is saved for testing. During the testing phase,
loading the pre-trained model W0 and the fine-tuned adapter
FIGURE 2. Low rank adaption method, where the pre-trained weights W weights 1Wi is done followed by merging using the Peft
is frozen, and the updated weights matrix is decomposed into two software package to form a single model ready for testing
matrices A and B. The embedding is denoted by h′ which is combined
from the embedding of the frozen weights h and the embedding h from using the test set. The merged model W is obtained using
the update decomposed weight matrices (A and B). W = W0 +AB. The fine-tuning of aforementioned four LLMs
was done using a single A100 GPU on the GAIVI cluster
for safety and showed good performance compared to other at the University of South Florida using two datasets: the
closed source models such as ChatGPT and PaLM [7], [46]. English quotes dataset and the Open-Platypus dataset.
Fine-tuning was done on 80% of the dataset, and testing
VI. QLORA was done on the remaining 20%. The hyper-parameter
Both low rank adaption (LoRA) and its extension, quantized selection was based on [13]. Although, better results may
low rank adaption (QLoRA) [12], [13], aim to reduce obtained by tuning the hyper-parameters, our goal in this
the footprint of GPU memory during fine-tuning. LoRA study is to assess the repeatability of the results rather getting
freezes a pre-trained LLM weight matrix and decomposes the best results.
the back-propagation update matrix 1W into A and B using We have assessed the effects of two parameters on
low-rank adaption method. The decomposed weight matrices generated text and answers to questions by LLMs. These
are small and require less memory, and therefore require two parameters are temperature and nucleus sampling. The
smaller GPU memory and less computation during fine- default value for these two parameters is 1, and we have
tuning. Figure 2 shows LoRA approach for freezing the set the value to 0.1 to reduce the randomization in text
pre-trained model and decomposing weight update matrices generation. The following two subsections provide the details
1W into low rank matrices A and B. QLoRA expands for fine-tuning each of the four LLMs.
LoRA by quantizing the low-rank weight matrices using
three components: 4-bits NormalFloat (NF4) data type;
double quantization for reducing memory usage during
fine-tuning of LLMs; and a paged optimizer for memory
spike management and to reduce memory consumption
during fine-tuning. QLoRA enables fine-tuning of LLMs on
a single GPU of 48GB while maintaining task performance
using half-precision (16-bits). Finally, QLoRA freezes and
quantizes the weights of the pre-trained model.

VII. EXPERIMENTS
The goal of the present experiment is to assess the FIGURE 3. Fine-tuning of LLMs using QLoRA approach, where text
repeatability of LLMs fine-tuned with the state-of-the-art dataset is tokenized and used for fine-tuning a pre-trained LLM models
(step 1). The fine-tuned model is used for generating text using a text
QLoRA approach. For this work we used four LLMs, namely, prompt (step2).
GPT-NeoX-20B and Meta-Llama (Llama-2-7b-hf) for text
generation and GPT-NeoXT-chat-20B and Meta-Llama-chat
(Llama-2-chat-7b-hf) for dialogue-based question answering. A. GPT-NEOX AND GPT-NEOXT-CHAT FINE-TUNING
As depicted in Figure 3, our approach consists of two steps: For fine-tuning the GPT-NeoX-20B model on the English
1) fine-tuning LLMs using the QLoRA approach while quotes dataset (described in Section III), we have downloaded
saving the fine-tuned model, and 2) text generation using the pre-trained model from the Huggingface website using the
fine-tuned model. AutoModelForCausalLm function available in the transform-
Given a pre-trained model W0 , the goal is to fine-tune ers API. The tokenizer for the input text was also downloaded
the pre-trained model for seven trials ti where i ∈ from the pretrained model using the AutoTokenizer method.
{1, 2, 3, 4, 5, 6, 7} under the same hardware and software For loading the model in a low memory footprint, we have

153224 VOLUME 12, 2024

S. S. Alahmari et al.: Repeatability of Fine-Tuning LLMs Illustrated Using QLoRA

used the QLoRA approach with bits and bytes configurations Llama2-7b-hf using AutoTokenizer. Using the Parameter
such as loading the model in 4 bits, and using double Efficient Fine-Tuning library (PEFT), we set the configura-
quantization, the type of quantization used is normal float tion for LoRA, where r was set to 8, LoRA Alpha was set
4 bits (NF4), and the computation precision was set to float16. to 32. Using the Transformer training arguments setting, the
For fine-tuning GPT-NeoX-20B using the English quotes learning rate was set to 1e−4 , the optimizer was set to paged
dataset on a GPU, we have used QLoRA, where r was set AdamW, learning rate scheduling was set to constant, training
to 8, lora alpha was set to 32, and lora dropout ratio was set to epochs was set to 3. The fine-tuning of Llama2-7b-hf was
0.05. Fine-tuning was done for three epochs with batch size of done for three epochs on the training subset of the dataset
1 and learning rate of 2e−4 , the optimizer was paged AdamW. where the maximum sequence size was set to 512.
Although, the model was loaded in 4 bits using the QLoRA We fine-tuned the Llama2-7b-hf pre-trained model for
approach, we have done the fine-tuning in half-precision (i.e., seven runs for each optimizer precision (i.e., paged AdamW
float16). The fine-tuning was done using the transformer 8bits and paged AdamW 32 bits) using the English quotes
trainer API. dataset. For each run, downloading the pre-trained model,
GPT-NeoX-20B was fine-tuned for seven runs (trials) for and setting-up the configuration of QLoRA LLM fine-tuning
the purpose of evaluating the consistency of the performance under the same hardware and software settings. Further-
of the fine-tuned models on text generation for the test subset. more, each run was fine-tuned independently on an Nvidia
Each trial was fine-tuned independently on the English A100 GPU.
quotes dataset, with a new loading of the pre-trained model, After fine-tuning Llama2-7b-hf on the training subset of
and setting-up the configurations each time using the same the dataset for seven runs per optimization algorithm type,
hardware and software. Furthermore, we have studied the we tested the fine-tuned models on the test subset for the
impact of the optimizer precision (i.e., paged AdamW 8bits purpose of evaluating the repeatability of the fine-tuning
vs. paged AdamW 32 bits). In this context we have fine-tuned under the same settings.
seven runs for each of paged AdamW 8bits, and 32 bits For question-answering repeatability experiment we have
optimizer for the purpose of comparing the repeatability of used Llama2-chat-7B model provided by Meta 5 . After
the fine-tuned models. Furthermore, each run was fine-tuned downloading the pretrained model, we have processed the
independently on an Nvidia A100 GPU. dataset (i.e., Open-Platypus) for the format of question-
For the question-answering repeatability assessment answering. For this purposes, we have placed the starting
experiment, we have used GPT-NeoXT-chat-20B available of sentence tag <s> at the beginning of each sentence and
on HuggingFace 4 using the Open-Platypus dataset. We have </s> at the end of the sentence in the dataset. Furthermore,
downloaded the tokenizer and applied the same parameters we have placed the instruction between the two tags <INST>
that were applied for GPT-NeoX-20B discussed in the last and </INST>. The answer to each instruction comes right
two paragraphs. Furthermore, we have processed the dataset after the end of the closing tag of the instruction. All the
to include the beginning of sentence tag <s> and the end of parameters used for fine-tuning the Llama2-7B model are
sentence tag </s>. Moreover, we have added the instruction used for fine-tuning Llama2-chat-7B model as detailed in
tag <human> and the response tag <bot>. Fine-tuning of the previous three paragraphs. Fine-tuning of Llama2-chat-
GPT-NeoXT-chat-20B was done for three epochs using an 7B was done using the Open-Platypus dataset for seven trials
80% split of Open-Platypus dataset. Fine-tuning was done for each using two versions of AdamW optimizer: 8bits AdamW
seven trials for each of AdamW 8bits and 32bits optimizer. and 32bits AdamW. The fine-tuning was done on 80% of
Fine-tuned models for each trial are tested using an unseen Open-Platypus dataset whereas the test was done on 20% of
20% of the Open-Platypus dataset. the dataset.
VIII. EVALUATION
B. LLAMA2-7B-HF AND LLAMA2-CHAT-7B FINE-TUNING
To evaluate the performance of the fine-tuned GPT-NeoX-
For all LLMs we have downloaded the models weights from
20B and Llama2-7b-hf, we prompted each fine-tuned
the Huggingface website using the AutoModelForCausalLM
model to complete the quotes ‘‘Your silence will’’ and
function available from the transformers class. For the
‘‘Let the improvement of yourself keep you’’. The response
Llama2-7B-hf and Llama2-chat-7B-hf model, a user must
of each fine-tuned model was recorded for comparison.
have an account on Huggingface. Furthermore, a form
Furthermore, we have used the Perplexity metric for
is required to be completed and submitted to Meta
measuring the quality of generated text by calculating
requesting access to Llama models. Then a user can
the exponentiated average negative of log-likelihood of a
generate a token which is required for authentication
tokenized sequence [47]. The perplexity metric is shown in
when downloading the pre-trained Llama2-7b-hf model.
Equation 1.
We generated a read token and downloaded Llama2- t
7b-hf using AutoModelForCausalLM from HuggingFace. 1X
Perplexity(X ) = exp(− log pθ (xi |x<i )) (1)
We also, downloaded the tokenizer from the pre-trained t
i

4 [Link] 5 [Link]

VOLUME 12, 2024 153225

 S. S. Alahmari et al.: Repeatability of Fine-Tuning LLMs Illustrated Using QLoRA

where, log pθ (xi |x<i ) is the log likelihood of the current • A: If you are not able to write your essays using a word
token i conditioned in the preceding tokens x<i . It is worth processing program, you have no keyboarding skills.
noting that the perplexity metric is computed using the • B: If you are able to write your essays using a word
LLM true probability distribution output before sampling processing program, you have at least some keyboarding
or temperature scaling applied. In Table 1, the perplexity skills.
of the fine-tuned GPT-NeoX-20B is shown for each run • C: If you are not able to write your essays using a word
and optimizer setting. As can be observed, the perplexity is processing program, you are not able to use a computer.
different among trials which indicates fine-tuning the LLM • D: If you have some keyboarding skills, you will be able
using the QLoRA approach is not repeatable. Furthermore, to write your essays using a word processing program.’’
a lower perplexity is obtained by using paged AdamW and instruction 2 is ‘‘Tommy: Many people claim that
optimizer 32 bits. Also, the perplexity of the fine-tuned the voting public is unable to evaluate complex campaign
Llama2-7b-hf model is provided for each of seven runs, issues. The radio advertisements for Peterson in the national
where in each run training was done twice with two different campaign, however, discuss complex campaign issues, and
optimizers. From this table it is clear that the perplexity Peterson is currently more popular than the other candidates.
is different for each run, which is indicated by varying Jamie: Yes, Peterson is the most popular. However, you
perplexity using paged AdamW optimizer during fine-tuning are incorrect in claiming that this is a result of Peterson’
of Llama2-7b-hf. A different result with small variation in s discussion of complex campaign issues. Peterson simply
perplexity was observed among the fine-tuned Llama2-7b- strikes the voters as the most competent and trustworthy
hf runs. However, both indicate that fine-tuning using either candidate. Which one of the following, if true, most supports
optimizer, the perplexity and results of Llama2-7b-hf is not Jamie’s counter to Tommy?
repeatable.
In Table 2, we provide the perplexity for different • A: Polling data shows that most voters cannot identify
trials (total of seven trials) for each of GPT-NeoXT-chat- Peterson’s positions on campaign issues.
20B and Llama2-chat-7B fine-tuning. As observed from • B: Polling data shows that Peterson’s present popularity
the results, the perplexity for AdamW 8-bits optimizer is will probably diminish over time.
higher than the perplexity for AdamW 32-bits optimizer • C: Peterson’s opponents are discussing some of the same
which indicates that the AdamW 8-bits optimizer-based issues as Peterson is discussing.
fine-tuning is less repeatable than using AdamW 32-bits • D: Polling data shows that some voters consider
optimizer. Peterson competent and trustworthy.’’.
In Table 3, we provide a summary of the GPU allocation These two instructions were taken from the test split of the
for our four fine-tuned LLMs when using AdamW 8bits opti- Open-Platypus dataset. The results showed the fine-tuning
mizer and AdamW 32bits optimizer. As observed, the GPU of LLMs for question-answering is not repeatable. More
memory allocated for AdamW 32bits optimizer is slightly realistic results are observed for fine-tuning LLMs using
higher than the GPU memory allocated for fine-tuning LLMs AdamW 32 bits, however, we provided only the answer letter,
using AdamW 8bits optimizer. and excluded the models’ generated text due to the space
In Tables 4 and 5, we show the results of prompting limitations.
the fine-tuned models using a prompt from the test split A visual illustration of the disparities between different
of the English quotes dataset, the responses of both GPT- fine-tuning models in terms of the loss are shown in first and
NeoX-20B and Llama2-7b-hf are different for each test. third row of Figure 4 for GPT-NeoX-20B and GPT-NeoXT-
Furthermore, more understandable responses are obtained chat-20B fine-tuned using the QLoRA approach with 8bits
using fine-tuned LLMs using QLoRA technique with 32bit paged AdamW optimizer. Also the loss for fine-tuning GPT-
paged AdamW optimizer. In Tables 6 and 7, we provide the NeoX-20B and GPT-NeoXT-chat-20B using QLoRA with
results for prompting GPT-NeoXT-20B and Llama2-7B using 32bits paged AdamW optimizer are shown. Each sub-figure
a different prompt (‘‘Let the improvement of yourself keep shows the loss for each run of fine-tuning GPT-NeoX-20b
you’’). The results show the variations in the response from and GPT-NeoXT-chat-20B, and it is clear that each fine-tuned
one fine-tuned model to the other. Moreover, more realistic model has a variation in terms of loss.
and understandable responses are observed for fine-tuned Similarly, the loss for fine-tuned Llama2-7B-hf and
models using the Adamw 32bits optimizer. Llama2-chat-7B-hf models for different runs showed dif-
In Tables 8 and 9, we provide the results of prompting ferences from one run to the other for the QLoRA based
the fine-tuned models: GPT-NeoXT-chat-20B and Llama2- approach when fine-tuning with two optimizers 8bits and
chat-7B using two instructions. Instruction 1 is ‘‘If you have 32bits as shown in Figure 4 second row and fourth row.
no keyboarding skills at all, you will not be able to use a A summary of fine-tuning and inference time is shown for
computer. And if you are not able to use a computer, you each LLM for each run in Tables 10 and 11. A notable obser-
will not be able to write your essays using a word processing vation is the difference in fine-tuning time and inference time
program. If the statements above are true, which one of the among trials. AdamW 32bits optimizer-based fine-tuning
following must be true? requires slightly more training and testing time. However, that

153226 VOLUME 12, 2024

S. S. Alahmari et al.: Repeatability of Fine-Tuning LLMs Illustrated Using QLoRA

TABLE 1. The perplexity for fine-tuned GPT-NeoX-20B and Llama2-7B-hf TABLE 4. The results of prompting the fine-tuned GPT-NeoX-20B models
models when tested on an unseen subset of the data (test subset). The with the prompt: Your silence will. The first column shows the number of
perplexity of the model trials fine-tuned using the 8bit AdamW optimizer runs (i.e., the number of fine-tuning trials), the second column shows the
and the 32bits AdamW optimizer are provided. results using the model fine-tuned with the AdamW 8bits optimizer, and
the third column shows the results using the model fine-tuned with the
AdamW 32bits optimizer.

TABLE 2. The perplexity of fine-tuned GPT-NeoXT-chat-20B and

Llama2-chat-7B models when tested on an unseen subset of the dataset
(Open-Platypus dataset) when using AdamW 8bits and 32bits optimizer.

TABLE 3. GPU Memory allocation (%) per LLM fine-tuned using AdamW
8bits and 32bits optimizer.

perplexity for GPT-NeoX-20B and GPT-NeoXT-chat-20B for

is acceptable given the realistic and understandable results seven test runs are 14.35 ± 0.0 and 16.625 ± 0.0 respectively.
fine-tuning LLMs with AdamW 32bits can yield. These consistent perplexity results are because perplexity
is computed using the raw probability distribution before
IX. ABLATION STUDY applying the temperature or the sampling technique.
In this ablation study, we have tested the repeatability of
the four fine-tuned LLMs when setting the temperature X. DISCUSSION
and sampling to a lower value (i.e., temperature = 0.1, Large language models (LLMs) development and training
and sampling (top-p) = 0.1). The results of setting the require large computation resources and memory. However,
temperature and nucleus sampling parameters to a lower there is a need to improve LLMs for particular domains
value to reduce the randomization in text generation enabled such as medical records summarization. The improvement
repeatable results when testing the exact same fine-tuned requires fine-tuning pre-trained LLMs with a new and
model multiple times. However, non-repeatable results are specialized data corpus reflecting the new domain. With
observed across different independently fine-tuned models. the low rank adaptation method (LoRA) and the quantized
This indicates that fine-tuning with low precision is causing version of low rank adaptation (QLoRA), fine-tuning can
the variation in weights parameters between different fine- be done on a single GPU. However, many iterations of
tuned models, which consequently caused the differences in fine-tuning may be done over time to keep the LLMs updated
the results of the test set. and to solve privacy, bias, and behaviour concerns. Such
Furthermore, we have evaluated the pre-trained models fine-tuning can yield different behaviour (sometimes worse
(before any fine-tuning) on the test set and report the than expected) due to the randomization inherent in fine-
perplexity score. We found the same per-trained LLM model tuning, rather than the updated data corpus, which leads to
outputs have the same perplexity for all test trials. The increased power-consumption costs, workforce time cost, and
mean perplexity for Llama2-7B-chat-hf for seven test runs delay in availability of trusted services for the end user.
is 17.68 ± 0.0, whereas the mean perplexity for Llama2-7B- Therefore, addressing the repeatability concerns of LLMs is
hf for seven test runs is 7.04 ± 0.0. Furthermore, the mean important for quick and interpretable fine-tuning.

VOLUME 12, 2024 153227

 S. S. Alahmari et al.: Repeatability of Fine-Tuning LLMs Illustrated Using QLoRA

FIGURE 4. Loss for fine-tuning GPT-NeoX-20B and Llama2-7B. The first row shows the loss for fine-tuning GPT-NeoX-20B, second row shows the loss for
fine-tuning Llama2-7B, the third row shows the loss for fine-tuning GPT-NeoXT-chat-20B, and the fourth row shows the loss for fine-tuning
Llama2-chat-7B. The first and third column are the loss per run when using 8bits AdamW optimizer, whereas the second and the fourth column show the
loss per epoch when using 32bits AdamW optimizer. Due to the small differences between the loss bars in the third column, we have indicated the final
loss next to each bar for the LLMs fine-tuned using AdamW 32bits.

Our study focused on LLM fine-tuning using publicly fine-tuning run was done for three epochs. Although, better
available datasets. We report differences in responses after performance may be obtained by fine-tuning for more epochs,
prompting each fine-tuned model. While using the 32-bits our purpose is to assess repeatability under the same settings
AdamW optimizer showed more realistic responses due to rather than finding the optimal results.
higher precision used during the optimization process, there Repeatability is a critical consideration for understanding
was nevertheless variation in responses from one fine-tuned the performance and behavior of LLMs when fine-tuning.
LLM trial to another. Furthermore, the perplexity was Different factors in deep learning training and fine-tuning
different from one trial to another for all LLMs. Therefore, such as GPU operations and weight initialization can impact
it is important to address the repeatability of fine-tuning repeatability as discussed in Section II. Fine-tuning of LLMs
LLMs using QLoRA and LoRA approaches for obtaining using QLoRA is not repeatable and leads to inconsistent
reliable models. results, where different runs of fine-tuning yield different
This study focused on assessing the repeatability of outputs and performance. Therefore, fine-tuning LLMs
fine-tuning LLMs under the same hardware and software requires attention to the differences in performance that may
settings for multiple trials using QLoRA. For this purpose, occur. As observed in this study, the optimizer precision
we have fine-tuned four LLMs for multiple runs, where each such as paged AdamW with 8-bit and 32-bits precision

153228 VOLUME 12, 2024

S. S. Alahmari et al.: Repeatability of Fine-Tuning LLMs Illustrated Using QLoRA

TABLE 5. The results of prompting the fine-tuned Llama2-7b-hf models TABLE 7. The results of prompting the fine-tuned Llama2-7B-hf models
with the prompt: Your silence will. The first column shows the number of with the prompt ‘‘Let the improvement of yourself keep you’’. The first
runs (i.e., the number of fine-tuning trials), the second column shows the column shows the run number, the second and third columns show the
results using the model fine-tuned with the AdamW 8bits optimizer, and results for the fine-tuned models using AdamW 8bits and 32bits
the third column shows the results using the model fine-tuned with the respectively.
AdamW 32bits optimizer.

TABLE 8. The results of prompting the fine-tuned GPT-NeoXT-chat-20B

models with two instructions provided in Section X. We show the output
TABLE 6. The results of prompting the fine-tuned GPT-NeoX-20B with the for each fine-tuned model using both AdamW 8bits and 32bits.
prompt ‘‘Let the improvment of yourself keep you’’. The first column
shows the number of runs (fine-tuned trials), the second column shows
the results for fine-tuning GPT-NeoX-20 using the AdamW 8bits optimizer,
whereas the third column shows the results when using the AdawW
32bits optimizer.

TABLE 9. The results of prompting the fine-tuned GPT-NeoXT-chat-20B

models with two instructions provided in Section X. We show the output
for each fine-tuned model using both AdamW 8bits and 32bits.

fine-tuning. Furthermore, fine-tuning time and inference can

vary among different trials. Therefore, the observed behavior
when fine-tuning LLMs using QLoRA approach highlights
can result in varying perplexity and model response and the need for stability and interpretability guidelines in deep
as observed from the variation in the loss graph during learning field.

VOLUME 12, 2024 153229

 S. S. Alahmari et al.: Repeatability of Fine-Tuning LLMs Illustrated Using QLoRA

TABLE 10. The fine-tuned time and test time in seconds for [4] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou,
GPT-NeoX-20B and Llama2-7B models for seven runs. W. Li, and P. J. Liu, ‘‘Exploring the limits of transfer learning with a unified
text-to-text transformer,’’ J. Mach. Learn. Res., vol. 21, no. 140, pp. 1–67,
2020.
[5] W. Fedus, B. Zoph, and N. Shazeer, ‘‘Switch transformers: Scaling
to trillion parameter models with simple and efficient sparsity,’’ J.
Mach. Learn. Res., vol. 23, no. 120, pp. 1–39, 2022.
[6] X. Sun, X. Li, J. Li, F. Wu, S. Guo, T. Zhang, and G. Wang, ‘‘Text
classification via large language models,’’ 2023, arXiv:2305.08377.
[7] H. Touvron et al., ‘‘Llama 2: Open foundation and fine-tuned chat models,’’
2023, arXiv:2307.09288.
[8] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov,
‘‘Proximal policy optimization algorithms,’’ 2017, arXiv:1707.06347.
[9] S. Min, M. Lewis, L. Zettlemoyer, and H. Hajishirzi, ‘‘MetaICL: Learning
TABLE 11. The fine-tuned time time and test time in seconds for
to learn in context,’’ 2021, arXiv:2110.15943.
GPT-NeoXT-chat-20B and Llama2-chat-7B models for seven runs. [10] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. Wei Yu, B. Lester, N. Du,
A. M. Dai, and Q. V. Le, ‘‘Finetuned language models are zero-shot
learners,’’ 2021, arXiv:2109.01652.
[11] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright,
P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman,
J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder,
P. Christiano, J. Leike, and R. Lowe, ‘‘Training language models to follow
instructions with human feedback,’’ in Proc. Adv. Neural Inf. Process.
Syst. (NIPS), vol. 35, 2022, pp. 27730–27744.
[12] E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang,
and W. Chen, ‘‘Low-rank adaptation of large language models,’’ 2021,
arXiv:2106.09685.
[13] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, ‘‘QLoRA:
Efficient finetuning of quantized LLMs,’’ 2023, arXiv:2305.14314.
Although, this study focused on the repeatability of LLMs [14] H. E. Plesser, ‘‘Reproducibility vs. Replicability: A brief history of a
using four foundation models on two tasks text generation and confused terminology,’’ Frontiers Neuroinform., vol. 11, p. 76, Jan. 2018.
[15] S. S. Alahmari, D. B. Goldgof, P. R. Mouton, and L. O. Hall, ‘‘Challenges
question-answering. There are some limitation to this study for the repeatability of deep learning models,’’ IEEE Access, vol. 8,
including studying repeatability in text classification, content pp. 211860–211868, 2020.
summarization. This study also focused on two open source [16] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin,
A. Desmaison, L. Antiga, and A. Lerer, ‘‘Automatic differentiation in
LLMs, however for our future work, we aim to study repeata- PyTorch,’’ in Proc. NIPS-W, 2017, pp. 1–4.
bility for more large models on different tasks including [17] F. Chollet. (2015). Keras. [Online]. Available: [Link]
text classification and content summarization. Furthermore, [18] M. Abadi et al. (2015). TensorFlow: Large-Scale Machine Learning on
Heterogeneous Systems. [Online]. Available: [Link]
our future work includes studying the repoducibility and
[19] D. Zhuang, X. Zhang, S. Song, and S. Hooker, ‘‘Randomness in neural
replicability of fine-tuning LLMs. network training: Characterizing the impact of tooling,’’ in Proc. Mach.
Learn. Syst., vol. 4, 2022, pp. 316–336.
[20] C. Summers and M. J. Dinneen, ‘‘Nondeterminism and instability in
XI. CONCLUSION
neural network optimization,’’ in Proc. Int. Conf. Mach. Learn., 2021,
Repeatability of deep learning is important for ensuring pp. 9913–9922.
stability and interpretability of deployed models. We studied [21] S. Bhojanapalli, K. Wilber, A. Veit, A. S. Rawat, S. Kim, A. Menon, and
the repeatability of fine-tuning LLMs on the same hardware S. Kumar, ‘‘On the reproducibility of neural network predictions,’’ 2021,
arXiv:2102.03349.
and software settings for multiple runs. The observed [22] A. Lemay, K. Hoebel, C. P. Bridge, B. Befano, S. D. Sanjosé, D. Egemen,
behaviour showed that fine-tuning of LLMs with the QLoRA A. C. Rodriguez, M. Schiffman, J. P. Campbell, and J. Kalpathy-Cramer,
technique using a single GPU is not repeatable for four LLMs ‘‘Improving the repeatability of deep learning models with Monte Carlo
dropout,’’ NPJ Digit. Med., vol. 5, no. 1, p. 174, Nov. 2022.
fine-tuned on two publicly available text corpuses. Thus, [23] J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith,
updating LLMs on a newer corpus of data requires awareness ‘‘Fine-tuning pretrained language models: Weight initializations, data
and attention to the differences in the performance that may orders, and early stopping,’’ 2020, arXiv:2002.06305.
[24] M. Mosbach, M. Andriushchenko, and D. Klakow, ‘‘On the stability of
occur after fine-tuning. fine-tuning Bert: Misconceptions, explanations, and strong baselines,’’
2020, arXiv:2006.04884.
ACKNOWLEDGMENT [25] C. Lee, K. Cho, and W. Kang, ‘‘Mixout: Effective regularization to finetune
large-scale pretrained language models,’’ 2019, arXiv:1909.11299.
Nvidia supported this research with a GPU. The USF [26] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ‘‘BERT: Pre-training
Strategic Investment Pool supported this research project. of deep bidirectional transformers for language understanding,’’ 2018,
arXiv:1810.04805.
[27] Y. Shang, Z. Yuan, Q. Wu, and Z. Dong, ‘‘PB-LLM: Partially binarized
REFERENCES large language models,’’ 2023, arXiv:2310.00034.
[1] T. B. Brown et al., ‘‘Language models are few-shot learners,’’ in Proc. [28] (2023). Aident5. Accessed: Jul. 23, 4. [Online]. Available:
NIPS, 2020, pp. 1877–1901. ttps://[Link]/or4cl3ai/Aiden5A
[2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, [29] A. N. Lee, C. J. Hunter, and N. Ruiz, ‘‘Platypus: Quick, cheap, and
Ł. Kaiser, and I. Polosukhin, ‘‘Attention is all you need,’’ in Proc. Adv. powerful refinement of LLMs,’’ 2023, arxiv:2308.07317.
Neural Inf. Process. Syst., vol. 30, 2017, pp. 1–11. [30] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike,
[3] S. Hochreiter and J. Schmidhuber, ‘‘Long short-term memory,’’ Neural J. Schulman, I. Sutskever, and K. Cobbe, ‘‘Let’s verify step by step,’’ 2023,
Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997. arXiv:2305.20050.

153230 VOLUME 12, 2024

S. S. Alahmari et al.: Repeatability of Fine-Tuning LLMs Illustrated Using QLoRA

[31] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, LAWRENCE O. HALL (Life Fellow, IEEE)
P. Clark, and A. Kalyan, ‘‘Learn to explain: Multimodal reasoning via received the B.S. degree in applied mathematics
thought chains for science question answering,’’ in Proc. Conf. Neural Inf. from Florida Institute of Technology, in 1980,
Process. Syst. (NeurIPS), 2022, pp.2507–2521. and the Ph.D. degree in computer science from
[32] X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, Florida State University, in 1986. He is currently
S. Zhang, Y. Sun, and W. Wang, ‘‘SciBench: Evaluating college-level a Distinguished University Professor with the
scientific problem-solving abilities of large language models,’’ 2023, Department of Computer Science and Engineer-
arXiv:2307.10635.
ing, University of South Florida. He has authored
[33] W. Yu, Z. Jiang, Y. Dong, and J. Feng, ‘‘ReClor: A reading comprehension
or co-authored over 85 publications in journals
dataset requiring logical reasoning,’’ in Proc. Int. Conf. Learn. Represent.
(ICLR), Apr. 2020, pp. 1–26.
and many conference papers and book chapters.
[34] W. Chen, M. Yin, M. Ku, P. Lu, Y. Wan, X. Ma, J. Xu, X. Wang, and He has received over $5M in research funding from agencies, such
T. Xia, ‘‘TheoremQA: A theorem-driven question answering dataset,’’ as the National Science Foundation, the National Institutes of Health,
2023, arXiv:2305.12524. the Department of Energy, DARPA, and NASA. His research interests
[35] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, include learning from big data, distributed machine learning, medical image
D. Song, and J. Steinhardt, ‘‘Measuring mathematical problem solving understanding, bioinformatics, pattern recognition, and integrating AI into
with the math dataset,’’ in Proc. NeurIPS, 2021, pp. 1–22. image processing. He is a fellow of AAAS, AIMBE, and IAPR. He received
[36] T. Sawada, D. Paleka, A. Havrilla, P. Tadepalli, P. Vidas, A. Kranias, the Norbert Wiener Award and the Joseph Wohl Award from the IEEE SMC
J. J. Nay, K. Gupta, and A. Komatsuzaki, ‘‘Arb: Advanced reasoning Society, in 2012 and 2017, respectively.
benchmark for large language models,’’ 2023, arXiv:2307.13692.
[37] Association for Computing Machinery. (2016). Artifact Review
and Badging. Accessed: Apr. 15, 2022. [Online]. Available:
[Link]
current PETER R. MOUTON received the joint Ph.D.
[38] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, degree in neurobiology from the University of
J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, South Florida (USF), Tampa, FL, USA, and
‘‘The pile: An 800GB dataset of diverse text for language modeling,’’ 2021, the Karolinska Institute, Stockholm, Sweden.
arXiv:2101.00027. He completed postdoctoral training at the
[39] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, ‘‘RoFormer: University of Copenhagen, Denmark, and the
Enhanced transformer with rotary position embedding,’’ 2021, Johns Hopkins University School of Medicine,
arXiv:2104.09864. Baltimore, MD, USA, before spending 14 years
[40] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, ‘‘Improving at the Johns Hopkins Department of Pathology
language understanding by generative pre-training,’’ OpenAI, 2018. and the National Institute on Aging, Baltimore.
Accessed: Oct. 17, 2024. [Online]. Available: [Link] He is currently the Founder and the Chief Scientific Officer with SRC
language-unsupervised/ Biosciences and the Courtesy Faculty Member with the Department of
[41] B. Wang, ‘‘ Mesh-transformer-JAX: Model-parallel implementation of
Computer Science and Engineering, USF. He is a Stereologist and a
transformer language model with JAX,’’ 2021. Accessed: Oct. 17, 2024.
Neuroscientist with scientific expertise in the development and analysis of
[Online]. Available: [Link]
[42] (2023). GPT-Neox-Chat-20b. Accessed: Jul. 22, 2024. [Online]. Available:
human neuropathology in animal models of Alzheimer’s disease, autism,
[Link] and Down’s syndrome. Another long-time interest is the development of
[43] J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and biotechnology to assist neuroscience researchers. He has published over
S. Sanghai, ‘‘GQA: Training generalized multi-query transformer models 100 peer-reviewed books, book chapters, and scientific papers. He is the PI
from multi-head checkpoints,’’ 2023, arXiv:2305.13245. on research grants from the National Science Foundation and the National
[44] B. Zhang and R. Sennrich, ‘‘Root mean square layer normalization,’’ in Institutes of Health. He serves as member of the editorial board for Journal
Proc. Adv. Neural Inf. Process. Syst., vol. 32, 2019, pp. 1–12. of Chemical Neuroanatomy. He is a member of the National Academy
[45] N. Shazeer, ‘‘GLU variants improve transformer,’’ 2020, of Inventors and the Education Committee for the Digital Pathology
arXiv:2002.05202. Association. Since 2016, he has been a Standing Member of the NIH
[46] (2023). LLAMA2-Chat Model. Accessed: Jul. 22, 2024. [Online]. Avail- Study Section [Drug Discovery for Aging, Neuropsychiatric and Neurologic
able: [Link] Disorders (ETTN-11)]. In 2014, he was named one of Top 12 Technology
[47] Perplexity of Fixed-Length Models—[Link]. Innovators in Florida by Florida High Technology Magazine.
Accessed: Feb. 2, 2024. [Online]. Available: [Link]
co/docs/transformers/perplexity

DMITRY B. GOLDGOF (Life Fellow, IEEE) is

currently an Educator and a Scientist working
in the area of medical image analysis, image
and video processing, computer vision and AI,
ethics and computing, bioinformatics, and bio-
engineering. His research interests include two
broad thrusts, such as first is in the area of
SAEED S. ALAHMARI (Member, IEEE) received biomedical image analysis and machine learning
the Ph.D. degree in computer science from the with application in MR, CT, PET, and microscopy
University of South Florida, in 2020. He is cur- images, radiomics, and bioinformatics; the second
rently an Assistant Professor of computer science thrust is the area of video motion analysis with biometrics, surveillance, and
with Najran University, Najran, Saudi Arabia. biomedical applications. He is also a Distinguished University Professor and
He has authored and co-authored many journals the Vice Chair of the Department of Computer Science and Engineering,
and conference papers. His research interests University of South Florida, Tampa. He has graduated 29 Ph.D. and 45 M.S.
include learning from noisy and limited labeled students. He has published over 100 journals and 200 conference papers,
data, machine learning, deep learning, medical 20 book chapters, and edited five books (over 10,000 citations and an
image understanding, computer vision, and deep H-index of 52). He is a fellow of IAPR, AAAS, and AIMBE. His full CV
learning repeatability and explainability. His full CV can be found at: can be found at [Link] goldgof/.
[Link]

VOLUME 12, 2024 153231