Comparative Profiling: Insights Into Latent Diffusion

Model Training
Bradley Aldous Ahmed M. Abdelmoniem
Queen Mary University of London Queen Mary University of London
London, United Kingdom London, United Kingdom
[Link]@[Link] [Link]@[Link]

Abstract 1 Introduction
Generative AI models are at the forefront of advancing cre- The advent of generative artificial intelligence (AI) has ush-
ative and analytical tasks, pushing the boundaries of what ered in a new era of innovation, enabling the creation of
machines can generate and comprehend. Among these, la- content that blurs the line between human and machine-
tent diffusion models represent significant advancements in generated output. At the forefront of this revolution are
generating high-fidelity audio and images. This study intro- latent diffusion models (LDMs), a class of generative models
duces a systematic approach to study GPU utilisation during that have demonstrated remarkable capability in producing
the training of these models by leveraging Weights & Biases high-fidelity audio and images. Among these, AudioLDM
and the PyTorch Profiler for detailed monitoring and profil- [15] and Stable Diffusion [21] represent significant advance-
ing. Our methodology is designed to uncover inefficiencies ments, leveraging the power of deep learning to synthesise
in GPU resource allocation, pinpointing bottlenecks in the content that is not only diverse and detailed but also respon-
training pipeline. The insights gained aim to guide the de- sive to nuanced prompts and inputs.
velopment of strategies for enhancing training efficiency, However, the computational demands of training such
potentially reducing computational costs and accelerating sophisticated models are substantial, necessitating efficient
the development cycle of generative AI models. This con- utilisation of hardware resources, particularly Graphics Pro-
tribution not only highlights the critical role of resource cessing Units (GPUs). GPUs have become the linchpin of
optimisation in scaling AI technologies but also opens new deep learning research and application, offering the parallel
avenues for research in efficient model training. processing capabilities required to manage the immense com-
putational load of these models. Despite their capabilities,
CCS Concepts: • Computing methodologies → Distributed optimising GPU usage to accommodate the intensive re-
artificial intelligence; • General and reference → Mea- quirements of generative AI remains a challenge. Inefficient
surement; Performance. resource utilisation not only prolongs training times but also
escalates costs and environmental impact, highlighting the
Keywords: deep learning, diffusion model, profiling need for focused research using profiling and monitoring
tools that can guide the optimisation of GPU usage. In the
ACM Reference Format: development of generative AI, identifying where and why
Bradley Aldous and Ahmed M. Abdelmoniem. 2024. Comparative bottlenecks occur in training processes is crucial for making
Profiling: Insights Into Latent Diffusion Model Training. In 4th them more efficient. Finding these bottlenecks can help in
Workshop on Machine Learning and Systems (EuroMLSys ’24), Apr improving the overall performance and efficiency of model
22, 2024, Athens, Greece. ACM, New York, NY, USA, 8 pages. https: training.
//[Link]/10.1145/3642970.3655847
This paper aims to identify bottlenecks and inefficiencies
in the training process, proposing optimisation strategies to
enhance computational efficiency. The key contributions of
Permission to make digital or hard copies of all or part of this work for this work are:
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights • It introduces a systematic approach to studying GPU
for components of this work owned by others than the author(s) must utilisation during the training of LDMs, using popular
be honored. Abstracting with credit is permitted. To copy otherwise, or profiling and monitoring tools;
republish, to post on servers or to redistribute to lists, requires prior specific • It identifies specific areas of computational inefficiency
permission and/or a fee. Request permissions from permissions@[Link]. across two advanced machine learning models, high-
EuroMLSys ’24, Apr 22, 2024, Athens, Greece
lighting CPU-bound overhead and memory manage-
© 2024 Copyright held by the owner/author(s). Publication rights licensed
to ACM. ment in AudioLDM, and emphasising the computa-
ACM ISBN 979-8-4007-0541-0/24/04 tional load of convolution operations and matrix mul-
[Link] tiplications in Stable Diffusion;

176
EuroMLSys ’24, Apr 22, 2024, Athens, Greece Bradley Aldous and Ahmed M. Abdelmoniem

Add Noise Add Noise

can significantly prolong training and inference times, mak-
... ing them computationally intensive.
To address these limitations, LDMs [21] were developed.
These models represent a significant leap forward by oper-
𝑥0 𝑥1 𝑥2 𝑥𝑇
ating in a lower-dimensional latent space, thereby reducing
computational demands while still enabling the creation of
... highly realistic and diverse outputs across various modalities.
Reverse Reverse
These models have found extensive applications, ranging
from generating photo realistic images [21] and high-fidelity
(a) The fundamental process of a diffusion model
audio [15, 16], to facilitating advancements in drug discovery
and material science [10]. In the domain of content creation,
LDMs are instrumental in producing artwork and music that
L1/L2 Reconstruction Loss can mimic human creativity, thereby transforming industries
by automating creative processes.
The architectural foundation of LDMs is a blend of diffusion-
based generative modelling and deep learning techniques,
characterised by their ability to gradually transform noise
Input Encoder z Decoder Output
into structured data. The core principle of diffusion involves
starting with a random distribution (noise) and iteratively
Latent Space refining this noise through a series of learned reverse dif-
fusion steps until it resembles the target distribution. Dif-
fusion is illustrated in Figure 1a. In LDMs, this process is
(b) A standard autoencoder architecture facilitated by encoding the data into a lower-dimensional
latent space, which reduces the computational complexity
Figure 1. High-level architecture diagrams of the two pri- and allows the model to capture the essential features of
mary components of LDMs. In (a), input data (𝑥 0 ) is gradually the data more efficiently. The autoencoder architecture typi-
transformed into a standard Gaussian distribution (𝑥𝑇 ), and cally comprises two main components: an encoder that maps
the reverse process gradually generates data samples from high-dimensional data to a latent space, and a decoder that
the noise. In (b), the input data is passed through an encoder performs the reverse process to generate data from the latent
and transformed into a latent representation, the decoder representations. The architecture is shown in Figure 1b. In
reverses this process. an LDM, the diffusion model generates data from random
noise in the latent space and the data is then decoded into the
original data space. The elegance of this approach lies in its
• It showcases the effectiveness of PyTorch’s Distributed ability to produce highly detailed and coherent outputs, mak-
Data-Parallel Strategy [14] in significantly reducing ing LDMs particularly effective for tasks requiring nuanced
training time for AudioLDM, demonstrating distributed understanding and generation of complex data patterns.
training as a viable strategy to mitigate computational Generative diffusion models present unique challenges
demands; and that diverge significantly from those encountered with tradi-
• It provides insights into optimisation strategies for en- tional deep learning models. The unique iterative refinement
hancing computational efficiency, including potential process of diffusion models requires significant computa-
areas for future research in optimising specific inef- tional resources, making the efficient use of hardware a crit-
ficiencies and the exploration of distributed training ical challenge. While the incorporation of latent spaces aims
strategies. to reduce the dimensionality and computational overhead,
the overall complexity of these models still necessitates ad-
vanced profiling and optimisation techniques. Traditional
2 Background & Motivation profiling tools may not adequately capture the nuances of
2.1 Latent Diffusion Models LDMs’ performance characteristics, underscoring the need
Generative models have emerged as a cornerstone of AI, en- for specialised approaches to aid in optimisation.
abling machines to create content that is increasingly indis-
tinguishable from that produced by humans [4, 18]. Among
these, diffusion models [26] have gained prominence for 2.2 GPU Utilisation in Deep Learning
their ability to generate high-quality and diverse outputs. The integration of GPUs into the deep learning ecosystem
However, a major concern with diffusion models is that their has been a transformative development, significantly acceler-
iterative generation process in a high-dimensional data space ating the training and inference processes of neural networks

177
Comparative Profiling: Insights Into Latent Diffusion Model Training EuroMLSys ’24, Apr 22, 2024, Athens, Greece

[6]. GPUs, with their highly parallel structure, are exception- 2.4 Motivation
ally well-suited to handle the matrix and vector operations Despite the availability and potential of these profiling tools,
that are ubiquitous in deep learning algorithms. This capabil- their application in the optimisation of LDMs, particularly
ity has enabled the training of larger, more complex models in the context of sophisticated generative tasks like those
than was previously possible, pushing the boundaries of ar- performed by AudioLDM and Stable Diffusion, remains an
tificial intelligence research and application. In the realm of underexplored area. The integration of detailed profiling and
generative models, such as LDMs, efficient GPU utilisation monitoring into the training workflow of such models is cru-
becomes not just advantageous but essential. Even though cial for advancing our understanding of their performance
they are computationally lighter than many other state-of- characteristics and for unlocking new efficiencies in their
the-art methods, these models require substantial computa- use of GPU resources. While existing profiling studies have
tional resources to iteratively refine noise into high-fidelity made significant strides in understanding and improving the
outputs, a process that involves millions of parameters and performance of deep learning models, they predominantly
operations. Optimising GPU usage can lead to significant focus on standard benchmarks, such as ResNet and other
improvements in training speed, model performance, and image-based architectures [11, 28]. This approach neglects
energy efficiency, which is crucial in a time when compu- the unique challenges and computational demands posed
tational costs and environmental impacts are of increasing by LDMs, and more generally audio models, which differ
concern. markedly in architecture and data processing requirements.
Despite the critical role of GPUs in deep learning, achiev- Moreover, the scarcity of detailed profiling and optimisa-
ing optimal utilisation poses considerable challenges. These tion studies on LDMs for audio highlights a critical oversight
challenges stem from a variety of factors, including memory in the development of efficient and accessible generative
management, data transfer bottlenecks, and the efficient allo- models. By dedicating research efforts to the specific profil-
cation of computational tasks across GPU cores. Addressing ing of audio LDMs, this study represents a first effort towards
these issues requires a good understanding of both the model illuminating the computational bottlenecks and inefficiencies
architecture and the underlying hardware. The pursuit of inherent in these models and their training configurations.
efficient GPU utilisation thus stands at the crossroads of Uncovering these insights is crucial for driving future op-
technological innovation and responsible computing, high- timisations, enabling LDMs to be more efficiently trained
lighting the need for continuous improvement in the tools on a wider range of hardware, and democratising access to
and techniques that support deep learning workflows. cutting-edge generative technologies.

2.3 Existing Profiling Tools & Techniques 3 Design Methodology

The landscape of deep learning has been significantly en- 3.1 Models
riched by the development and adoption of various profiling This research focuses on two advanced generative LDMs
tools and techniques, designed to optimise computational which exemplify the power of latent diffusion technology in
resource allocation and performance. These tools serve as their respective domains of audio and visual content genera-
critical instruments for developers and researchers, enabling tion. These models were selected for their distinct capabilities
them to dissect and understand the intricacies of model train- to generate high-fidelity outputs and their substantial rele-
ing and execution. Profiling in the context of GPU utilisa- vance to current AI research and applications. This section
tion involves the detailed monitoring of how deep learning provides an overview of the technical underpinnings and
models interact with hardware resources, highlighting areas unique characteristics of each model.
where improvements can be made to enhance efficiency and
performance. Tools like NVIDIA’s Nsight1 and the PyTorch 3.1.1 Stable Diffusion. This model has emerged as a promi-
Profiler [19] offer deep insights into the execution of neural nent model for image generation, known for its ability to
networks, providing metrics such as execution time, memory produce detailed and diverse visuals. The specific model used
usage, and GPU compute utilisation. Additionally, Weights here is the unconditional variant [21] trained on 2562 images
& Biases [5] offers a platform for experiment tracking and from the CelebA-HQ dataset2 , which leverages a latent space
monitoring, allowing for the aggregation and visualisation of representation of images to perform the diffusion process.
performance data across multiple training runs, facilitating The versatility and quality of outputs have garnered signifi-
the comparison and optimisation of models. The utilisation cant attention, establishing it as a key tool for creative and
of these tools is paramount in identifying bottlenecks and commercial applications alike.
inefficiencies within the training process. 3.1.2 AudioLDM. This model stands as a pioneering model
in the realm of generative audio, utilising latent diffusion
2 [Link]
1 [Link] 256x256

178
EuroMLSys ’24, Apr 22, 2024, Athens, Greece Bradley Aldous and Ahmed M. Abdelmoniem

Profiling Output to Console 3.2 Profiling & Monitoring Approach

To address the computational challenges posed by the train-
ing of advanced generative models such as AudioLDM and
Profiler: Post- Stable Diffusion, this study employs a methodology centred
Processing around the integration of Weights & Biases [5] and the Py-
Torch Profiler [19], tools selected for their robust capabilities
in tracking and analysing model performance and resource
Profiler: Collection
utilisation.
The PyTorch Profiler [19] is configured to monitor the
End Training execution times of the different operations present in the
training over one epoch, ordered by ’CUDA total’ (the to-
tal time spent on GPU-related operations when profiling
CUDA applications). This enables the identification of spe-
Log Weights & Biases cific operations or layers within the models that contribute
Metrics Every Epoch disproportionately to computational load or inefficiency.

Profiler
3.2.1 Metrics. Concurrently, Weights & Biases [5] is utilised
to track and visualise the metrics discussed below through-
Begin Training
out training runs, facilitating a thorough understanding of
training performance.

Initialise Weights
• GPU Utilisation measures the percentage of time the
& Biases
GPU is actively processing data over a certain period.
High utilisation indicates the GPU is being heavily
used.
• GPU Memory Allocated refers to the amount of GPU
Load memory (VRAM) assigned to a process. It’s crucial for
Model + Data (+) Text managing large datasets or complex computations that
require significant memory.
• GPU Power Usage shows the amount of power the
Figure 2. A high-level system diagram showing the experi- GPU is consuming in real-time. It’s important for man-
mental setup and the relation between the tools and models. aging the energy efficiency of the system and ensuring
Text data was only used with AudioLDM. the power supply can meet the GPU’s demands.

3.3 Experimental Setup

principles to synthesise realistic and coherent audio clips. The hardware setup features 8 NVIDIA A5000 GPUs (24GB)
Its architecture is designed to efficiently handle the complex and Intel Xeon Gold 5318Y CPUs, selected for their balance
temporal dynamics of audio data, capturing subtle nuances of memory capacity, computational power, and energy ef-
and variations across a wide range of sounds and musical ficiency crucial for training deep learning models. Initial
genres. The model operates by encoding raw audio into a runs were tested on separate servers comprised of NVIDIA
lower-dimensional latent space, where the diffusion process RTX 2080Ti GPUs, which struggled with the large memory
is applied to generate new audio samples. This approach al- requirements of these models.
lows for the efficient generation of high-quality audio while On the software side, the study utilised the environments
managing the computational load, making it an ideal can- provided by both implementations. The models and training
didate for exploring optimisations in GPU resource utilisa- were constructed and run using PyTorch, which allowed
tion. This model was trained on the AudioCaps dataset [13]. for seamless coupling with the PyTorch Profiler [19]. Note,
For both of these models, it is the training of the diffusion for AudioLDM, PyTorch Lightning [8] was used to train the
model that is profiled, as this is the generative part of both model, and so their built-in profiler was used, whereas the
architectures. For AudioLDM, this amounts to 185 million profiler was wrapped around the training loop for Stable
parameters and, for the Stable Diffusion variant, this is 104 Diffusion. Weights & Biases [5] is initialised before training
million parameters. to ensure full coverage of the process.

179
Comparative Profiling: Insights Into Latent Diffusion Model Training EuroMLSys ’24, Apr 22, 2024, Athens, Greece

4 Evaluation Table 1. Profiling results of various operations

4.1 Profiling Results
AudioLDM
4.1.1 AudioLDM Results. The profiling results reveal a
Operation CPU Total Self CUDA # Calls
distinct focus on optimisation and convolution operations.
[Link] 37.92% 0.00% 3
The ‘[Link]‘ operation, consuming 37.92% of the
ProfilerStep* 50.40% 0.00% 3
CPU total (see Table 1), signifies a heavy computational load
aten::cudnn_conv 1.20% 20.37% 388
on the CPU for model updates, while not directly involv-
aten::mul 1.43% 15.72% 4896
ing CUDA operations. This indicates that the optimisation
aten::bmm 1.10% 12.12% 846
process, crucial for adjusting model parameters, is primarily
Stable Diffusion
CPU-bound. The significant CPU total percentage seen in
Operation CPU Total Self CUDA # Calls
‘ProfilerStep*‘ suggests considerable overhead in moni-
DataLoader3 61.85% 0.00% 1876
toring model performance. Conversely, CUDA utilisation is
aten::copy_ 21.47% 2.16% 530911
dominated by ‘aten::cudnn_conv‘ and ‘aten::mul‘ opera-
aten::to 18.01% 0.00% 244036
tions, which are integral to the model’s forward and back-
aten::cudnn_conv 0.81% 25.05% 155625
ward passes, highlighting the intensive use of GPU resources
aten::bmm 0.32% 23.30% 170625
for convolutional operations and element-wise multiplica-
tions that underpin the model’s ability to process and gener-
ate audio data efficiently.

as long as those on the GPU for both models. This signifi-

4.1.2 Stable Diffusion Results. The profiling showcases cant discrepancy indicates a CPU bottleneck in the training
an emphasis on data loading and memory operations along- pipeline, suggesting that tasks managed by the CPU—such
side convolutional computations. The ‘DataLoader‘ opera- as data preprocessing, loading, and supplying data to the
tion, with a CPU total of 61.85%, underscores the substan- GPU—are notably more time-consuming. This imbalance
tial CPU resources dedicated to preparing and feeding data likely causes the GPU to idle while awaiting data, under utilis-
into the model, a crucial step for training efficiency but ing its computational capacity and impeding overall training
not directly involving GPU computations. Memory opera- efficiency. By streamlining CPU operations, there’s potential
tions such as ‘aten::copy_‘ and ‘aten::to‘ also command to significantly boost data throughput to the GPU, thereby
a significant portion of CPU resources, indicating the im- optimising the training process and enhancing model per-
portance of data manipulation and transfer in the model’s formance.
training process. The CUDA landscape is markedly defined The observed disparity in the number of operation calls
by ‘aten::cudnn_conv‘ and ‘aten::bmm‘, which account for between the two models can be attributed to the distinct
the majority of CUDA utilisation, emphasising the model’s training methodologies employed. Specifically, Stable Dif-
reliance on convolutional and batch matrix multiplication fusion underwent training via a custom loop with direct
operations for generating high-fidelity images. These CUDA- profiler integration, while the other was trained utilising the
intensive operations are pivotal for the model’s performance, PyTorch Lightning framework, which leverages an in-built
facilitating complex feature extraction and transformation PyTorch profiler. This difference in approach significantly
necessary for Stable Diffusion’s generative capabilities. influences the profiling data collected. PyTorch Lightning
Comparing the AudioLDM and Stable Diffusion models, abstracts and optimises several training pipeline aspects, po-
the profiling results illustrate distinct computational and tentially aggregating operations and streamlining execution
resource utilisation patterns. AudioLDM emphasises CPU- paths, which can affect the number and type of calls recorded
intensive optimisation and GPU-intensive convolution oper- by the profiler. In contrast, the custom training loop might
ations, indicating a balance between parameter updates and capture a more granular, less optimised sequence of oper-
data processing in neural networks. Stable Diffusion, how- ations, leading to a higher apparent number of calls. This
ever, allocates a significant portion of CPU resources to data distinction highlights the impact of training and profiling
loading and memory operations, reflecting the importance methodologies on the interpretation of profiling data, so the
of efficient data handling and transfer in its training process. main conclusions drawn here are based on a relative compar-
CUDA usage in both models is dominated by convolutional ison of execution time for operations rather than the number
operations, but Stable Diffusion additionally highlights the of calls. For interested readers; the full profiling output can
role of batch matrix multiplications, suggesting a diverse be found on GitHub4 .
computational strategy for handling generative tasks.
The comprehensive analysis of our profiling data reveals a
pronounced disparity in processing times between the CPU
and GPU, with the CPU times being approximately twice 4 [Link]

180
EuroMLSys ’24, Apr 22, 2024, Athens, Greece Bradley Aldous and Ahmed M. Abdelmoniem

GPU Memory Allocated (%) GPU Memory Allocated (%) Training Loss GPU Utilisation (%)

GPU Power Usage (%) GPU Power Usage (%)

Figure 4. Plots showing the convergence and GPU utilisation
of two separate training runs of AudioLDM. the red plots
are for a single GPU training run, and the blue plots are for
a data parallel training run on two GPUs

Figure 3. Plots showing the GPU memory allocated and the

GPU power usage as percentages for the training of both Figure 4. The left plot shows the evolution of the loss during
AudioLDM (the red plots to the left) and Stable Diffusion training (here both examples were run for 10 epochs with
(the green plots to the right) identical hyperparameters), and here you can see the time
efficiency of distributing the data over two GPUs. During
these two runs, the GPU utilisation was significantly higher
4.2 Performance Analysis in the distributed setting, showing a 20% increase. This, in
Figure 3 shows the result of monitoring the training process turn, led to a significant speedup in terms of training time,
of these two models. During the profiling of memory allo- with the single GPU run taking 13 hours and 47 minutes to
cation for both AudioLDM and Stable Diffusion models, it complete 10 epochs, compared to 5 hours and 29 minutes
was observed that AudioLDM consistently required a higher for the dual GPU run. In this experiment, distributing the
amount of memory across all training epochs. This increased training across two GPUs reduces the training time by a
memory demand is attributed to its complex architecture, factor of 2.51, beyond linear scaling.
designed to handle audio-visual data, which inherently re-
quires processing a larger volume of information. Notably, 4.4 Discussion
memory usage was consistent across the training of both
The profiling results suggest that while both AudioLDM
models.
and Stable Diffusion models face challenges in optimising
In terms of power usage, AudioLDM demonstrated a higher
performance, the areas requiring improvement vary signifi-
and more uniform consumption compared to Stable Diffu-
cantly. Stable Diffusion demonstrates significant CUDA time
sion. This pattern indicates a steady and intensive utilisa-
consumption in convolution operations and matrix multipli-
tion of GPU resources throughout training, reflecting the
cations, highlighting the computational intensity of these
model’s consistent computational demands. The uniform
processes. The study reveals that computational demands
power usage by AudioLDM contrasts with the more variable
and resource utilisation patterns differ significantly between
consumption observed for Stable Diffusion, suggesting dif-
the two models. AudioLDM exhibits a higher memory re-
ferences in their operational efficiencies and the nature of
quirement and more uniform power consumption through-
tasks being performed.
out its training process, attributed to its complex architecture
Beyond memory allocation and power usage, the profiling
tailored to processing audio-visual data.
highlighted the impact of external factors and the choice of
Additionally, PyTorch’s Distributed Data Parallel Strategy
profiling tools on the accuracy and efficiency of performance
[14], when applied to AudioLDM, resulted in a notable reduc-
measurement. The comparison between the PyTorch Profiler
tion in training time. This suggests that distributed training
[19] and PyTorch Lightning’s built-in profiler revealed differ-
is a viable strategy for mitigating computational demands
ences in data collection and processing efficiency, suggesting
and accelerating the training process.
that the choice of tool can significantly affect the overhead
and precision of performance profiling.
5 Related Work
4.3 Distributed Performance Recent studies have delved into performance analysis be-
An additional training run was performed on AudioLDM in tween CPUs and GPUs for deep learning tasks, with findings
which PyTorch’s Distributed Data Parallel Strategy [14] was indicating GPUs’ superior processing speeds and efficiency,
used to divide the training across two GPUs. The results are particularly for complex models [6, 11]. Research on profil-
compared to the single GPU training run and displayed in ing and monitoring tools has shown that while monitoring

181
Comparative Profiling: Insights Into Latent Diffusion Model Training EuroMLSys ’24, Apr 22, 2024, Athens, Greece

tools impose minimal overhead, enabling real-time decision- investigation would offer more nuanced insights into how
making, profiling tools, though resource-intensive, offer valu- different hardware configurations impact the efficiency of
able insights for optimising performance [28]. Furthermore, generative diffusion model training.
the development of SASSI, a precise, customisable profiling Lastly, future investigations into the trade-offs between
tool for GPUs, represents a significant advancement in ap- computational efficiency, model performance, and resource
plication characterisation and architectural exploration [27]. utilisation are crucial for designing models that not only
Efforts to optimise GPU memory usage in large dataset ap- achieve high accuracy but also operate efficiently in resource-
plications have also shown substantial reductions in data constrained environments.
transfer volumes, enhancing efficiency [22].
In the realm of distributed machine learning, innovations
like DC2 [3] and SIDCo [17] demonstrate significant advance- Acknowledgments
ments in minimising communication delays and speeding This work was supported by the UKRI and EPSRC under
up training processes through delay-aware compression con- grant EP/S022694/1.
trol and statistical-based gradient compression techniques,
respectively. Evaluations of distributed deep learning frame- References
works such as Horovod [24], DeepSpeed [20], and PyTorch
[1] Marcel Aach, Eray Inanc, Rakesh Sarma, Morris Riedel, and Andreas
Distributed Data Parallel [14] have highlighted their scala- Lintermann. 2023. Large scale performance analysis of distributed
bility and efficiency, showcasing their ability to effectively deep learning frameworks for convolutional neural networks. Journal
leverage cloud resources and improve training scalability of Big Data 10 (6 2023), 96. Issue 1. [Link]
across various GPU configurations [1, 9]. Additionally, a 00765-w
comparative study of frameworks like Caffe-MPI [12], CNTK [2] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng
Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu
[23], MXNet [7], and TensorFlow [2] has identified key per- Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey
formance bottlenecks and optimisation opportunities for Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser,
training convolutional neural networks with synchronous Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga,
SGD [25]. Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon
Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vin-
cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals,
Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xi-
6 Conclusion & Future Work aoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning
This study provides an in-depth analysis of the computa- on Heterogeneous Systems. [Link] Software
tional efficiency and performance characteristics of Audi- available from [Link].
oLDM and Stable Diffusion, two advanced machine learning [3] Ahmed M. Abdelmoniem and Marco Canini. 2021. DC2: Delay-aware
Compression Control for Distributed Machine Learning. In IEEE IN-
models. The findings underscore the importance of tailored FOCOM 2021 - IEEE Conference on Computer Communications. 1–10.
optimisation strategies for different model architectures. [Link]
Future work should aim to further optimise the identi- [4] Lucas Bellaiche, Rohin Shahi, Martin Harry Turpin, Anya Ragnhild-
fied areas of computational inefficiency. For AudioLDM, this stveit, Shawn Sprockett, Nathaniel Barr, Alexander Christensen, and
Paul Seli. 2023. Humans versus AI: whether and why we prefer human-
could include efforts to reduce CPU-bound overhead and en-
created compared to AI-created artwork. Cognitive Research: Principles
hance memory management efficiency. For Stable Diffusion, and Implications 8 (7 2023), 42. Issue 1. [Link]
the focus could be on optimising convolution operations, 023-00499-6
possibly through the use of more efficient algorithms or [5] Lukas Biewald. 2020. Experiment Tracking with Weights and Biases.
hardware acceleration techniques. Additionally, exploring [Link] Software available from [Link].
the effects of different profiling tools and methodologies [6] Ebubekir BUBER and Banu DIRI. 2018. Performance Analysis and
CPU vs GPU Comparison for Deep Learning. In 2018 6th International
could provide valuable insights into the accuracy and over- Conference on Control Engineering Information Technology (CEIT). 1–6.
head of performance measurements, including the impact of [Link]
external system activities on training processes. [7] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang,
The study also highlights the potential of distributed train- Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. 2015.
ing to significantly reduce training times, suggesting that MXNet: A Flexible and Efficient Machine Learning Library for Het-
erogeneous Distributed Systems. (12 2015). [Link]
further research into these strategies is warranted. Certainly, 01274
here’s a paragraph you could use for the future work section: [8] William Falcon and The PyTorch Lightning team. 2019. PyTorch Light-
In future work, a comprehensive exploration into the scala- ning. [Link]
bility of distributed training across a broader spectrum and [9] Attila Farkas, Krisztián Póra, Sándor Szénási, Gábor Kertész, and
larger quantity of GPU models is essential to validate the Róbert Lovas. 2022. Evaluation of a distributed deep learning frame-
work as a reference architecture for a cloud environment. In 2022
observed superlinear speedup phenomena in greater depth. IEEE 10th Jubilee International Conference on Computational Cyber-
By extending the range of GPU architectures tested, from netics and Cyber-Medical Systems (ICCC). 000083–000088. https:
consumer-grade cards to high-end data center GPUs, this //[Link]/10.1109/ICCC202255925.2022.9922765

182
EuroMLSys ’24, Apr 22, 2024, Athens, Greece Bradley Aldous and Ahmed M. Abdelmoniem

[10] Cong Fu, Keqiang Yan, Limei Wang, Wing Yee Au, Michael McThrow, Francisco, California, USA) (KDD ’16). Association for Computing Ma-
Tao Komikado, Koji Maruhashi, Kanji Uchino, Xiaoning Qian, and chinery, New York, NY, USA, 2135. [Link]
Shuiwang Ji. 2023. A Latent Diffusion Model for Protein Structure 2945397
Generation. arXiv:2305.04120 [[Link]] [24] Alexander Sergeev and Mike Del Balso. 2018. Horovod: fast and easy
[11] Dipesh Gyawali. 2023. Comparative Analysis of CPU and GPU Pro- distributed deep learning in TensorFlow. (2 2018). [Link]
filing for Deep Learning Models. (9 2023). [Link] abs/1802.05799
02521 [25] Shaohuai Shi, Qiang Wang, and Xiaowen Chu. 2017. Performance
[12] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Modeling and Evaluation of Distributed Deep Learning Frameworks
Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. on GPUs. (11 2017). [Link]
Caffe: Convolutional Architecture for Fast Feature Embedding. In Pro- [26] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya
ceedings of the 22nd ACM International Conference on Multimedia (Or- Ganguli. 2015. Deep Unsupervised Learning using Nonequilibrium
lando, Florida, USA) (MM ’14). Association for Computing Machinery, Thermodynamics. Proceedings of the 32nd International Conference on
New York, NY, USA, 675–678. [Link] Machine Learning, 2256–2265.
[13] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee [27] Mark Stephenson, Siva Kumar Sastry Hari, Yunsup Lee, Eiman
Kim. 2019. AudioCaps: Generating Captions for Audios in The Wild. In Ebrahimi, Daniel R. Johnson, David Nellans, Mike O’Connor, and
Proceedings of the 2019 Conference of the North American Chapter of the Stephen W. Keckler. 2015. Flexible software profiling of GPU archi-
Association for Computational Linguistics: Human Language Technolo- tectures. SIGARCH Comput. Archit. News 43, 3S (jun 2015), 185–197.
gies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and [Link]
Thamar Solorio (Eds.). Association for Computational Linguistics, Min- [28] Ehsan Yousefzadeh-Asl-Miandoab, Ties Robroek, and Pinar Tozun.
neapolis, Minnesota, 119–132. [Link] 2023. Profiling and Monitoring Deep Learning Training Tasks. In
[14] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Proceedings of the 3rd Workshop on Machine Learning and Systems
Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, (Rome, Italy) (EuroMLSys ’23). Association for Computing Machinery,
and Soumith Chintala. 2020. PyTorch Distributed: Experiences on New York, NY, USA, 18–25. [Link]
Accelerating Data Parallel Training. (6 2020). [Link]
2006.15704
[15] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo
Mandic, Wenwu Wang, and Mark D. Plumbley. 2023. AudioLDM:
Text-to-Audio Generation with Latent Diffusion Models. (1 2023).
[Link]
[16] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong,
Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley.
2023. AudioLDM 2: Learning Holistic Audio Generation with Self-
supervised Pretraining. (8 2023). [Link]
[17] Ahmed M. Abdelmoniem, Ahmed Elzanaty, Mohamed-Slim Alouini,
and Marco Canini. 2021. An Efficient Statistical-based Gradient Com-
pression Technique for Distributed Training Systems. In Proceedings
of Machine Learning and Systems, A. Smola, A. Dimakis, and I. Sto-
ica (Eds.), Vol. 3. 297–322. [Link]
paper/2021/file/[Link]
[18] Sophie J Nightingale and Hany Farid. 2022. AI-synthesized faces are
indistinguishable from real faces and more trustworthy. Proceedings
of the National Academy of Sciences of the United States of America 119
(2 2022). Issue 8. [Link]
[19] PyTorch. 2021. Introducing PyTorch Profiler - the new and improved
performance tool. [Link] Soft-
ware available from [Link]
[20] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong
He. 2020. DeepSpeed: System Optimizations Enable Training Deep
Learning Models with Over 100 Billion Parameters. In Proceedings of
the 26th ACM SIGKDD International Conference on Knowledge Dis-
covery & Data Mining (Virtual Event, CA, USA) (KDD ’20). Asso-
ciation for Computing Machinery, New York, NY, USA, 3505–3506.
[Link]
[21] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser,
and Björn Ommer. 2021. High-Resolution Image Synthesis with Latent
Diffusion Models. (12 2021). [Link]
[22] Nadathur Satish, Narayanan Sundaram, and Kurt Keutzer. 2009. Op-
timizing the use of GPU memory in applications with large data
sets. 16th International Conference on High Performance Computing,
HiPC 2009 - Proceedings, 408–418. [Link]
5433185
[23] Frank Seide and Amit Agarwal. 2016. CNTK: Microsoft’s Open-Source
Deep-Learning Toolkit. In Proceedings of the 22nd ACM SIGKDD In-
ternational Conference on Knowledge Discovery and Data Mining (San

183