Cluster Computing (2024) 27:2273–2290

[Link] (0123456789().,-volV)(0123456789().
,- volV)

Cloud benchmarking and performance analysis of an HPC application

in Amazon EC2
Tamara Dancheva1,2,3 • Unai Alonso1 • Michael Barton3,4

Received: 13 July 2022 / Revised: 22 May 2023 / Accepted: 23 May 2023 / Published online: 28 June 2023
 The Author(s) 2023

Abstract
Cloud computing platforms have been continuously evolving. Features such as the Elastic Fabric Adapter (EFA) in the
Amazon Web Services (AWS) platform have brought yet another revolution in the High Performance Computing (HPC)
world, further accelerating the convergence of HPC and cloud computing. Other public clouds also support similar features
further fueling this change. In this paper, we show how and why the performance of a large-scale computational fluid
dynamics (CFD) HPC application on AWS competes very closely with the one on Beskow—a Cray XC40 supercomputer
at the PDC Center for High-Performance Computing - in terms of cost-efficiency with strong scaling up to 2304 processes.
We perform an extensive set of micro and macro benchmarks in both environments and conduct a comparative analysis.
Until as recently as 2020 these benchmarks have notoriously yielded unsatisfactory results for the cloud platforms
compared with on-premise infrastructures. Our aim is to access the HPC capabilities of the cloud, and in general to
demonstrate how researchers can scale and evaluate the performance of their application in the cloud.

Keywords AWS  C5n.x18large  ParallelCluster  Benchmark  HPC  EFA

1 Introduction made computing power and data storage more affordable

and accessible than ever before [1]. The existence of
Cloud computing has revolutionized the way computa- hundreds of cloud providers today attests to the massive
tional resources are delivered by making them available success of cloud computing.
on-demand and at a low cost. Consistent with the democ- Small companies and individuals can now get access to
ratization of technology in this era of globalization, it has the IT infrastructure they need without any investment in
hardware and maintenance. It has considerably cut down the
cost of IT and enabled companies to scale up with little effort
Unai Alonso and Michael Barton have contributed equally to compared with before. Naturally, the potential of using cloud
this work. service providers has widely attracted researchers’ attention,
some of whom have conducted feasibility studies of moving
& Tamara Dancheva
tdancheva001@[Link]; tdancheva@[Link] their research in the cloud ([2–4] among the first). When it
comes to large-scale high-performance computing applica-
Unai Alonso
[Link]@[Link] tions, which is the topic of this paper, the question becomes
the following. Can the performance of HPC applications in
Michael Barton
barton@[Link] the cloud match the one in on-premises infrastructures?
In this paper, we pursue the idea of moving HPC in the
1
Mechanical Engineering, University of the Basque Country, cloud and provide a thorough performance analysis making
Alameda Urquijo s/n., Bilbao 48013, Vizcaya, Spain use of the latest advances of AWS in the direction of HPC.
2
University of the Basque Country, Barrio de Sarriena, s/n, However, the general procedure we outline to evaluate the
48940 Leioa, Vizcaya, Spain performance of the application and the trends we describe
3
Simulation of Wave Propagation, Basque Center for Applied could serve as blueprint for any comparative study of this
Mathematics, Mazarredo 14, Bilbao 48009, Vizcaya, Spain type regardless of the cloud platform or on-premise
4
Ikerbasque - Basque Foundation for Sciences, Maria Diaz de infrastructure.
Haro 3, Bilbao 48013, Basque Country, Spain

123
2274 Cluster Computing (2024) 27:2273–2290

2 Literature review evaluation of Azure cloud for HPC [24], Maliszewski’s

study of interconnection in Azure Cloud [25, 26] and
According to the taxonomy of efforts in HPC presented by Guidi’s et al. [27] summary/survey on cloud perfomrance
Netto et al. [5], the contribution of this paper is in the over the last ten years.
category of HPC in the cloud or specifically, viability Finally, with the release of EFA, Zhuang et al. [28]
studies of moving HPC to cloud environments. Early show that Amazon EC2 achieves comparable performance
explorations have been initiated as early as in 2008. Studies to local supercomputing clusters for their GEOS-Chem
from this early period of cloud computing such as the ones atmospheric chemistry simulations, due to recent devel-
performed by Walker [2], Napper et al. [3] and Ostermann opments in the network performance and reduction of
et al. [4] categorically concluded that the cloud is not yet virtualization overhead. They dispute the validity of the
mature for tightly coupled HPC applications. paradigm that the cloud is not mature for HPC applications
In 2010, Amazon EC2 introduced the first generation of EFA is an effort in the direction of long-term cooperation
HPC infrastructure. However, various studies from 2010 with the industry, as highlighted by the MVAPICH project
[6–9], and more importantly 2011, still indicated that while [29]. Fernandez [30] conducts the HPCG benchmark in five
the cloud is cost effective and delivers satisfactory per- cloud platforms, highlighting among others the scalability
formance for ‘‘low communication-intensive applications of the C5n.18xlarge instances. The computational fluid
such as embarrassingly parallel and tree-structured com- dynamics (CFD) package, Ansys Fluent [31], also show-
putations’’ [10], it is still not mature enough for HPC cases the performance results for a benchmark on AWS
applications. Zhai et al. [11] and the Magellan Report [12] EC2 of the external flow over a Formula-1 Race car with
both concluded that the lack of of a high performance 140 million hex-core cells using the Finite Volume Metod
network severely limits the execution of tightly-coupled (FVM). Similarly, Turner et al. [32] showcase the perfor-
HPC applications. mance of a Reynolds Averaged Navier–Stokes (RANS)
Numerous studies [13–17] from the period between simulation of a full aircraft on Amazon EC2 c5n.18xlarge
2012 and 2015 indicate that while cloud computing is not instances. In contrast, our application uses the Finite Ele-
yet mature for HPC, the performance is becoming com- ment Method on tetrahedral meshes of a F1 Perrinn model,
parable. In particular, Mehrotra et al. in their ‘‘Performance which is a conceptually different method to be compared
evaluation of amazon EC2 for NASA HPC applications’’ directly. We also conduct thorough benchmarking in con-
[13], state that although the HPC performance in the AWS junction with the performance of the application.
lags behind the one on their reference supercomputer due Today practically all major public cloud providers offer
to the network technology and virtualization overhead, it is HPC capabilities [33–35]. It is already a competitive
quickly catching up. established market that caters to a notable number of pri-
Starting from 2016 to 2018, feasibility studies [18, 19] vate companies (aerospace, automotive, biochemistry
suggest that a hybrid approach would make the most of the among many). In comparison, in literature one finds very
two environments, cloud and on-premise clusters, given few studies performed in academia showcase the perfor-
that the network latency is still higher in the cloud [20, 21]. mance of HPC applications since the introduced changes.
Despite this, Bartosz et al. [21] find that the elastic cloud The largest bottleneck identified for running HPC
can provide a better turnaround time, ‘‘reducing the time to applications right now in the cloud is the networking by
science’’. A comparative paper by Mohammadi also claims large consensus [5, 12, 13, 23–25, 28, 36, 37]. Different
that ‘‘the performance on public cloud can be comparable clouds have addressed this problem to different degrees in
to modern traditional supercomputing systems’’ [22]. In reference to each other and internally based on the choice
short, the outlook for HPC in the cloud brightens. of node. De et al. [37] provide a very detailed study of the
For a more complete literature review of feasibility network performance using MPI benchmarks for various
studies in the cloud until 2018 we refer the readers to [5]. public clouds. The important takeaway relevant here is the
The needs of researchers regarding HPC have been fact that network noise for HPC nodes i.e. with at least 100
heard and taken into account by cloud providers. In the Gbps bandwidth is comparable to the one on-premise,
case of Amazon, the result is the announcement of the further supporting the results we are about to present.
RDMA/ EFA (Elastic Fabric Adapter) networking interface
with up to 100 Gbps bandwidth, in 2019. Breuer et al. [23]
show promising results for Seismic Simulations in the 3 Goals and contributions
cloud and state that this new development, once released, is
expected to further improve the strong scalability of their Networking, compute power and storage speed are the most
applications. Other promising studies include NASA’s important factors that impact the performance of scientific
applications. Firstly, we benchmark these factors separately.

123
Cluster Computing (2024) 27:2273–2290 2275

Compute power and networking are the most relevant ones Table 1 Hardware specifications of both environments
for our application. We then evaluate their impact on the Hardware AWS EC2 Beskow
performance of our massively parallel CFD application. specifications
Typically, applications are categorized into four main cate-
# Nodes auto, max 40 2060
gories: compute-intensive, memory-intensive, data-inten-
sive, and high-throughput. We should note that the Processor 2x E5-2698v3 Haswell
Intel(R) Xeon(R) Platinum
application we selected is partly compute-intensive, partly 8124 M
memory-intensive. Therefore, networking and compute Frequency 3.00GHz (Skylake) 2.3 GHz
power have an especially strong impact on the performance. # Physical 18 16
Researchers can use a wide range of tools to characterize cores
their own applications (for example VTune Profiler [38]). # SMT 2 2
In this paper we report the results from benchmarking L1 cache 64KB 64KB
and examining the performance of HPC applications on L2 cache 1024KB 256KB
Amazon EC2, using the open-source tool developed by L3 cache 25MB 20.48MB
AWS, ParallelCluster (released in November 2018 [39]). Main Memory 192 GiB 64 GB
Relatively recent technological advancement such as the NVRAM per 35 GB –
AWS Nitro Hyperviso, a very light hypervisor that node
removes a large part of the virtualization overhead, was Networking Elastic Fabric Adapter (EFA) Cray Aries
introduced in 2017. The Elastic Fabric Adapter (EFA) technology 100 Gbps (Dragonfly
feature is available in ParallelCluster as of April 2019 [40]. topology) 100
Gbps
These technologies have largely made possible the results
Latency  15.69 ls 1.92 ls
presented in the subsequent sections.
Power 400W per node  376W per node
The contributions of this work are: consumption
I Thorough benchmarking using the OSU micro bench- Turbo boost On On
marks and the NASA macro benchmarks. The results mode
can be used as indicators to make predictions about the
performance of HPC applications by other researchers
In conjunction, we describe in detail the configuration of
who might want to migrate their research to the cloud.
the two environments in which we have run both the
II Performance analysis of an HPC application con-
benchmarks (micro and macro) and the HPC application.
sisting of strong scaling and profiling. This applica-
Simultaneously, we outline how to create an AWS EC2
tion represents a large class of problems that involve
cluster and which tools to use.
solving a system of non-linear PDEs, using the
Newton method, solving a linear system of equations
in each iteration. It generalizes to many scientific
4.1 Amazon EC2 and AWS ParallelCluster
applications in different fields, discretized with the
Amazon EC2 stands for Amazon Elastic Compute Cloud. It
Finite Element method, with sparse systems of
is one of the Amazon Web Services which offers users
equations. In conjunction with I), we identify the
access to computational resources physically distributed in
underlying factors that lead to improved
data centers across the globe.
performance.
ParallelCluster is an open-source, command-line tool
Both I) and II) are conducted in Amazon EC2 and on developed by Amazon that enables a quick way to set up
Beskow, a Cray XC40 supercomputer, followed by a and manage a cluster in Amazon EC2 from a configuration
comparative analysis. Throughout the process, we explain file. It consists of various groups of configuration options
and highlight general trends that are relevant for these or sections. Table 2 details a configuration that yields
types of feasibility studies. satisfactory performance for an initial exploration of the
cloud’s capabilities with ParallelCluster v2. The latest
version at the time of writing, v3, offers an interactive
4 Configuration setup using either CLI or UI interface for the same options.
For a full list of options, see the documentation for AWS
Table 1 shows a summary of the hardware specifications of ParallelCluster [41].
both environments discussed in this section, in terms of The C5n instances in Amazon’s Compute Optimized
node compute power, memory configuration and family are fit for computationally intensive workloads due
networking.

123
2276 Cluster Computing (2024) 27:2273–2290

Table 2 An example of an AWS ParallelCluster configuration file The C5n.x18large instances are equipped with two
Configuration options
sockets x 18 Intel(R) Xeon(R) Platinum 8124 M CPU @
[global] section 3.00GHz (Skylake), a total of 72 vCPUs with Simultaneous
Multithreading and 192 GiB of memory. Our configuration
cluster_template = NAME_CLUSTER
consists of up to 40 compute nodes, C5n.x18large type
update_check = true (max_queue_size setting). For the master node responsible
sanity_check = true for coordinating compute nodes, compiling code, and
[aws] section, confidential - id, access key and region interacting with the job scheduler, we balance cost and
[cluster NAME_CLUSTER] section performance by opting for a C5n.4xlarge instance. We
key_name = SSH_PRIVATE_KEY_NAME utilize Ubuntu 18.04 as the base_os (see Table 2).
master_instance_type = c5n.4xlarge Amazon uses its proprietary Nitro System to create and
compute_instance_type = c5n.18xlarge manage virtual machines in the cloud. This system includes
master_root_volume_size = 25 the light-weight Nitro Hypervisor, offloading hypervisor
compute_root_volume_size = 35 workload to dedicated components: Nitro Cards, and Nitro
base_os = ubuntu1804 Security Chip. This reduces virtualization overhead - one
max_queue_size = 40 of the major bottlenecks in the past, and improves perfor-
scheduler = slurm mance [45].
placement_group = DYNAMIC The job scheduler handles job initiation, scheduling, and
vpc_settings = NAME_VPC monitoring. The latest version of ParallelCluster provides
ebs_settings = NAME_EBS several options for cluster schedulers, including Son of
enable_efa = compute Grid Engine, Slurm Workload Manager, Torque Resource
cluster_type = spot Manager, and AWS Batch. In the specific configuration
[vpc NAME_VPC] section - id and subnet id mentioned in Table 2, Slurm is chosen as the scheduler,
[ebs NAME_EBS] section same as the reference supercomputer.
shared_dir = NAME_SHARED_FOLDER The HPC application we focus on in this paper makes
volume_type = gp2 use of the standardized Message Passing Interface (MPI).
volume_size = 6000 The choice of MPI implementation can impact the per-
formance of your application dramatically. It is noteworthy
to mention that alternatives to using MPI implementations,
such as Chapel, UPC, UPC??, Spark, and others, might be
to their Enhanced Network Bandwidth support for high- worth looking into as well. Techniques such as multilevel
speed communication between nodes, taking advantage of parallelism [46] can further improve performance or
the Nitro System and the Elastic Fabric Adapter (EFA) to algorithms that employ non-blocking MPI routines espe-
deliver network bandwidth of 100 Gbps [40]. The EFA cially for Graviton and AMD based nodes with more
feature was introduced in 2019 [42], at which point the C5n compute power such as Hpc6a, C6gn, etc [47].
instances were state-of-the-art. They remained so at the We have tested two MPI implementations on our clus-
time of conducting this study and until the end of 2020 ter: Open MPI and Intel MPI. Both of them wrap around
when new instances have been added to the Compute the open-source GNU compiler. The IntelMPI version
Optimized family i.e C6a, C6i, C6in, C6g, C6gn, and C7g available through AWS is the open-source version of the
[43]. AWS has also created a new HPC family that includes Intel MPI wrapper that wraps around the GNU compiler.
hpc6a and hpc6id, introduced at the end of 2021 and 2022 The placement of the nodes within a chosen time zone is
accordingly [44]. Since our reference cluster runs on Intel also crucial. To minimize the distance between the nodes
processors, in this study our aim was to get to as even and achieve a low-latency network performance, we can
baseline as possible as a first step, using machines with use the placement group options available (placement_-
same bandwidth (100 Gbps) and type of processor (Intel). group in Table 2).
Next step would be to incrementally introduce extra layers For storage, we utilize a 6000 GB gp2 SSD volume,
of complexity by extending the study to Graviton and providing a maximum of 16000 IOPS and a Max
AMD architectures with same network bandwidth as the Throughput/Volume of 250 MB/s. This mid-range option
C5n nodes i.e. C6gn, and hpc6a respectively. The only offers both competitive pricing and performance, making it
candidates with network bandwidth not inferior or equal to suitable for a diverse range of workloads.
100 Gbps are C6in, C6id, and C7gn, all of them introduced The Nitro card for VPC supports several network
at the end of 2022 with the optimized version of the EFA, acceleration features, such as EFA (it optimizes for latency
that open new interesting possibilities to explore. between 2 instances, scaling elasticity). You can enable the

123
Cluster Computing (2024) 27:2273–2290 2277

EFA (a network device) option through the enable_efa flag 5 Benchmarks

in the configuration Table 2. RDMA (Remote Direct
Memory Access) is a network protocol used to side-step The microbenchmarks are small pieces of code that have
the processor and exchange data between two main mem- the purpose of quantifying the basic building blocks of a
ories directly, bypassing the OS. We observed a significant program separately. The macro-benchmarks, on the other
impact using this feature in our results, which prompted us hand, consist of a more complex code that should be rep-
to write this paper. resentative of the class of problems for which we try to
Last but not least, the advances in load balancing, task extract conclusions. In this case, we are looking at a sci-
scheduling, resource allocation, and distributed computing entific application that requires the discretization and
algorithms for network communication also play a signif- solution of a system of time-dependent partial differential
icant role in the overall performance in the cloud. They equations. We run the benchmarks in the order of
facilitate the ability to have a reliable low communication increasing specificity.
latency between the nodes in a worldwide distributed In this study we run all the OSU micro-benchmark
system, a crucial factor in getting a good performance. configurations for each message size and number of pro-
On the software side, we use the open-source Spack cesses 3000 times skipping the initial 100 runs (i.e.
manager to manage the packages in a modular way warmup iterations which aim to avoid fluctuations in the
[48, 49]. It allows for a quick installation of core depen- runtime) on the same set of nodes (two nodes for the point
dencies we need to run our software, enabling different to point MPI benchmarks and up to 4 nodes for the col-
versions to coexist without any issue - an essential lective MPI and NASA benchmarks). The configurations of
requirement in supercomputing environments. the NAS Parallel Benchmarks were run 5 times each. In the
plots we show the mean value and the standard deviation of
4.2 Beskow—Cray XC40 supercomputer the results.

Beskow [50], a Cray XC40 supercomputer, located at the 5.1 OSU Micro-Benchmarks
PDC Center for High-Performance Computing in Stock-
holm, consists of 11 cabinets with a total of 2060 compute The OSU Micro-benchmarks [51] measure the perfor-
nodes and 67,456 cores. Each node contains 2 Intel CPUs: mance of a comprehensive set of MPI routines. The amount
• 9 cabinets with Xeon E5-2698v3 Haswell 2.3 GHz (16 of time spent in communication between the processes
cores/CPU) should stay within a reasonable limit. As we scale our
• 2 cabinets with Xeon E5-2695v4 Broadwell 2.1 GHz(18 application up to a larger number of cores, it can become a
cores/CPU) bottleneck that will affect the performance severely. Dif-
ferent MPI implementations deploy various strategies and
We have run the benchmarks and the simulations on the algorithms, which results in differing performance. In this
Xeon E5-2698v3 Haswell 2.3 GHz nodes. The nodes paper, we report the results from the following two cate-
interconnect with a High-speed network Cray Aries gories: Point-to-Point MPI and Collective MPI.
(Dragonfly topology) that also supports RDMA operations.
The Haswell nodes have 64 GB of RAM, and the 5.1.1 Point-To-Point MPI benchmarks
Broadwell nodes have 128GB. Beskow has in its disposal a
5 Petabyte Lustre file system. The Point-To-Point MPI communication refers to the MPI
Beskow achieves a peak performance of 2.438 Petaflops send-and-receive routines that send and receive a message
(1015 floating-point operations per second). The Linpack from one process to another. Using these routines, we can
benchmark has a long-standing history as a criterion for derive some fundamental performance indicators, such as
ranking the top supercomputers. This benchmark solves the latency and the bandwidth between two nodes in a
dense systems of linear equations and naturally is not a system (a supercomputing environment or a distributed one
reference for the performance of all kinds of applications. such as the AWS EC2 one). The latency benchmark con-
Beskow lists a Linpack performance of 1.80 Petaflops. sists of a ping-pong exchange of messages between two
Based on the type of application, we need a representative processes, obtaining an average one-way latency value.
benchmark to obtain an accurate estimate of its The bandwidth benchmark aims to determine the maxi-
performance. mum data rate possible by sending multiple messages
Beskow also relies on Slurm as a scheduler. The results without waiting for an acknowledgment by the receiver.
reported in this study have been obtained with the native AWS ParallelCluster offers support for setting up an
Cray MPI implementation, the most optimized one for the HPC cluster with pre-installed Intel MPI and Open MPI. If
underlying hardware.

123
2278 Cluster Computing (2024) 27:2273–2290

EFA is enabled (enable_efa option in Table 2), Paral- provider for EFA not detecting a local process. Given the
lelCluster sets up the following software stack. It starts issue and that their internode latencies are slightly better
with the EFA Kernel Module on the bottom, a Libfabric for Intel MPI, we have chosen to run the rest of the
Network Stack in the middle, and the MPI implementations benchmarks and the final application using Intel MPI.
on the top. The alternative to using EFA is using the The results from the bandwidth benchmark shown in
Transmission Control Protocol (TCP). TCP results in Fig. 2 follow a similar pattern, however in favor of AWS.
longer latencies and is less fit for tightly-coupled applica- For smaller message sizes, up to 1024B (Fig. 2a) both
tions than RDMA. environments achieve similar results. For more than 1024
The results from the OSU latency benchmarks, shown in B, up to 256 KiB (Fig. 2b), AWS shows higher to equal-
Fig. 1, compare the latency between two processes on two ized bandwidth. Beskow shows slightly better results in the
nodes using EFA-enabled Intel MPI and Open MPI on the rest, up to 1 MiB.
AWS EC2 cluster and Beskow using Cray MPI. For These findings are indicators of the shift that has taken
smaller message sizes, up to 4KiB (1 Kibibyte is 1024 place because numerous viability studies for moving HPC
Bytes), Beskow exhibits significantly lower latencies than to the cloud since 2008 (see Section 2) list the lack of a fast
the AWS cluster. For larger message sizes, up to 256 KiB network suitable for HPC applications as an impediment.
(Fig. 1), the gap between the supercomputer and the cloud The latency and the bandwidth are now comparable to the
closes. For messages larger than 256 KiB, the discrepancy reference supercomputer for a wide range of message sizes,
arises again, and Beskow records lower latencies. Intel MPI most notably for the bandwidth, rather than the latency in
and Open MPI on AWS behave similarly on a broad range the window of message size we are examining.
of message sizes from 1 B to 1 MiB, with fractionally MPI implementations offer the possibility of tuning
higher latencies for Open MPI. We find that when running them for our needs (for example tuning transport protocol
the same benchmark on one node, the latencies for Open algorithms and parameters), either through runtime
MPI are abnormally high, due to a confirmed issue at the parameters or configuration files. The very low latencies on
time of running the study (see [52]) with the Libfabric Beskow for very small and very large message sizes are the

(a) Latency for small messages from 256B to 16KiB (b) Latency for large messages from 16KiB to 1MiB

Fig. 1 Latency between 2 nodes on AWS with Intel and Open MPI, and on Beskow with Cray MPI

(a) Bandwidth for small messages from 256B to 16KiB (b) Bandwidth for large messages from 16KiB to 1MiB

Fig. 2 Bandwidth between 2 nodes on AWS with Intel and Open MPI, and on Beskow with Cray MPI

123
Cluster Computing (2024) 27:2273–2290 2279

(a) MPI AllReduce (b) MPI AllGather

Fig. 3 Benchmarking of the MPI Allreduce and Allgather Collective routines

(a) MPI Reduce (b) MPI AllToAll

Fig. 4 Benchmarking of the MPI Reduce and AlltoAll Collective routines

result of the tuning of the Cray MPI implementation, the cluster exhibits good scalability (as evidenced by other
underlying transport and routing protocols (proprietary in recent studies [30, 37]). It performs better or comparable to
the case of AWS) and last level cache cache misses [37]. Beskow, for most of the blocking MPI calls. In some cases,
Similarly, so is the high bandwidth of close to 6000 MB/s such as Fig. 3a 4, 5b, 5b AWS performs considerably
for small message sizes on AWS (Fig. 2a). This is dis- better with much higher latencies for Beskow, as the
cussed in more detail in the discussion. message size increases.
The reason behind the efficiency and low performance
5.1.2 Collective MPI benchmarks variability of the collectives on AWS are the advances in
the underlying distributed computing algorithms, resource
The Collective MPI functions communicate information allocation, hypervisor technology, MPI tuning of the
between a group of processes. In this section, we present algorithms behind the implementation of the collectives,
the results from benchmarking the collectives that take the proprietary transport protocol and EFA adapter.
most of the time spent in communication between pro- We find that the benchmarks for MPI Broadcast
cesses for our HPC application. Additionally, we have run (broadcast a message from one process to all others) and
all the blocking and non-blocking collective MPI bench- MPI Barrier (synchronize all the tasks) stand out from the
marks for up to 128 cores and a message size of 1MiB. rest. Figure 6a shows that MPI Broadcast exhibits up to
Figure 5 shows the results for MPI Allreduce, MPI two times higher latencies than Beskow for 128 processes
Allgather, MPI Reduce, MPI Alltoall, MPI Alltoallv, and and, Fig. 6b shows that MPI Barrier exhibits up to 8 times
MPI Allgatherv. The overall latency for AWS becomes higher latencies than Beskow for 128 processes. It is
distinctly lower as the number of processes increases for expected that one of the reasons is that Beskow (Cray
each of the functions. As a general observation, we can XC40) offloads Barrier to hardware to accelerate it.
conclude that for larger message size, the AWS EC2

123
2280 Cluster Computing (2024) 27:2273–2290

(a) MPI AllToAllV (b) MPI AllGatherV

Fig. 5 Benchmarking of the MPI Alltoallv and Allgatherv Collective routines

(a) Benchmarking of MPI Broadcast (b) Benchmarking of MPI Barrier

Fig. 6 Results from the OSU benchmarks for MPI Broadcast and Barrier up to 256 processes

In our code, these latencies do not pose a problem since diagonal solver (SP), and a Lower-Upper Gauss-Seidel
Barrier and Broadcast constitute a negligible portion of the solver (LU).
total MPI communication time (Fig. 9 in Sect. 6.3).
5.2.1 OpenMP benchmarks
5.2 NAS parallel benchmarks
Figure 7 shows the elapsed time from running the NAS
The NAS benchmarks have been created by the National benchmarks using OpenMP in shared-memory on one node
Aeronautics and Space Administration agency (NASA) to with 32 threads and 64 threads on both Beskow and the
evaluate the suitability of new architectures [53]. The AWS EC2 cluster. On Beskow, the code is compiled with
package has become representative of a wide range of Cray Clang and on AWS with the GNU compiler. In both
categories of scientific applications, although it initially cases, for all benchmarks, AWS yields better performance
aimed to estimate the performance of computational fluid than Beskow. For 32 threads, it achieves a mean speedup of
dynamics applications. In particular, we are interested in 1.7 and for 64 of 1.37.
the high throughput or tightly-coupled applications. We
also hope to be able to contribute in a more general sense 5.2.2 MPI benchmarks
with the results from these benchmarks. Readers who are
interested in porting their application to the cloud may find Figure 8 visualizes the performance from running the
these findings relevant. benchmarks with MPI on two nodes (problem size C1) and
The plots in this section show the results from the eight nodes (problem size D) in elapsed time. On Beskow,
original eight benchmarks that consist of the following:
1
Integer Sort (IS), Embarrassingly Parallel (EP), Conjugate The problem sizes of the NAS Parallel Benchmarks from smallest
Gradient (CG), Multi-Grid (MG), 3D fast Fourier Trans- to largest are categorized into classes S, W, A, B, C, D, E, and F. For
the CG benchmark, the size of the problem ranges from 1.4K rows in
form (FT), a Block Tri-diagonal solver (BT), Scalar Penta-
the left-hand side matrix to 54 million rows. Class C packs 150K rows
and size D packs 1.5 million rows for CG.

123
Cluster Computing (2024) 27:2273–2290 2281

(a) Run with 32 threads (b) Run with 64 threads

Fig. 7 Results from the NAS benchmarks using OpenMP, problem class C

(a) Problem size class C, on 2 nodes (b) Problem size class D, on 8 nodes

Fig. 8 Results from the NAS benchmarks using MPI, problem class C and D

the code is compiled with Cray MPICH and on AWS with • The automation of discrete solutions - at the heart lies the
Intel MPI wrapped around the GNU compiler. Again, AWS solution of a linear system of equations using a variety of
prevails with less time than Beskow in both cases, for all methods available through the external linear algebra back-
benchmarks. On 2 nodes AWS achieves a mean of 1.4 end, PETSc, such as AMG, iterative, and direct methods.
times more bandwidth than on Beskow and on 8 nodes it • The automation of error control - based on a measure of
records a similar value of 1.45 times. choice or an error indicator, that can be obtained from
the dual problem.

6 Massively parallel FEniCS-HPC application 6.1 CFD Model

FEniCS-HPC [54] is a platform written in C?? that con-
The Direct Finite Element Method (Direct FEM) refers to a
sists of Dolfin-HPC, a highly parallel FEM (Finite Element
turbulence model (described in more detail in [55, 56], and
Method) library for solving general partial differential
[57]), implemented within Unicorn. It uses the the piece
equations, and UNICORN, a continuum mechanics solver
wise linear General Galerkin (G2) method and belongs to
built up on top of Dolfin-HPC.
the category of stabilized space-time methods.
The advantages that it gives over other solvers lie in:
Equation (1) shows the weak form of the Euler equa-
• The automation of discretization and assembly - tions with Least Squared Stabilization.
generation of a wide gallery of finite element types,  n 
ðU  U n1 Þ

arbitrary polynomial order, and finite element spaces. þ U  rU v þ ðrPn ; vÞ þ r  U^n ; q
^ n ^n
kn

The whole process is based on the weak formulation
þ d1 ðU^n  rU^n þ rPn Þ; U^n  rv þ rq
obtained from the strong form by integration by parts in

þ d2 r  U^n ; r  v
the case of second or higher-order derivatives (similar
¼0
to the analytical process of solving a PDE on paper). It
allows the users to rather focus on the problem. ð1Þ

123
2282 Cluster Computing (2024) 27:2273–2290

n
where U^ ¼ U þU
n n1
for all (v,q) 2 Vh x Qh , and Vh 2 ½W n 3 of writing, using the Direct FEM. The solver is completely
2
and Qh 2 ½W n  are finite element approximation spaces, Vh parallelized with MPI and MPI I/O.
being a vector finite element space. d1 ¼ j1 ðkn2 þ
6.3 MPI profiling
1=2
kU n1 k2 h2
n Þ and d2 ¼ j2 hn are the stabilization
parameters with constants j1 and j2 . We have profiled the HPC application with Integrated
In order to solve the non-linear system of equations, we Performance Monitor (IPM) [58]. The information we aim
use the Newton method, with the Conjugate Gradient linear to extract from the reports is the most time-consuming MPI
solver with a Block Jacobi preconditioner for the continuity routines in our application and statistics about the message
and the Biconjugate gradient stabilized method with a sizes. Figure 9 shows the results from the profiling of our
Block Jacobi preconditioner for the momentum. application with 256 processes distributed between 8
nodes.
6.2 Aerodynamics simulation of a Perrinn F1 car Figure 9a displays a pie plot of the most consuming MPI
routines. For all the functions except MPI_Irecv, MPI does
We will now analyze the performance of a CFD (Com- not collect the message size for a message with status MPI_
putational fluid dynamics) application implemented in STATUS_IGNORE, which affects receiving, probing, and
FEniCS-HPC. We present the results for an incompressible waiting functions. Consequently, it is safe to only take into
CFD simulation around a F1 car (Perrinn model with 25 account MPI send and the MPI collectives. Excluding these
million cells). We chose this particular simulation as a functions, we have shown the results from the OSU micro-
showcase because it is one of the most challenging CFD benchmarks for the top six time-consuming MPI collec-
simulations, and it is representative of the other CFD tives (see Section 5.1.2).
simulations we run with FEniCS-HPC, as well as other Figure 9b shows the communication balance sorted by
FEM applications that involve the solution of non-linear task ID. There is more work done in the beginning due to
PDEs and sparse solvers. This application is developed on the initialization, and afterward, it starts to equalize. We
top of the latest development branch of Unicorn at the time

(a) MPI Communucation Time Pie (b) Communication balance by task (sorted by task ID)

(c) Message Buffer Size Distributions: time (d) Message Buffer Size Distributions: Ncalls

Fig. 9 IPM profiling results with 256 processes

123
Cluster Computing (2024) 27:2273–2290 2283

can see how much time each rank (or process ID) spends in in the assembly stage [59], as can be observed from the
each communication routine. Having a good load balance first two plots in Fig. 10.
is crucial for the performance of HPC applications. • Solving the resulting non-linear systems of PDEs by
From Fig. 9c and Fig. 9d, ignoring the receive, probe, iteratively solving a series of linear systems of PDEs
and wait-like functions, we can see that the message size that result from the momentum equation Eq. (1) line (1)
ranges from 4 Bytes to 1 MB. Both the number of calls and and the continuity equation Eq.(1) line (2). The
the elapsed time for each of the MPI routines represent continuity equation requires considerably more itera-
non-decreasing functions of the message size. Most of the tions to solve, thereby taking more time than the
messages fall in the range from 4 B to 256 KB. This trend momentum Fig. 10.
holds for up to 1152 processes on 16 nodes. Given that the In the results below, we analyze these parts of the runtime
problem size is constant, the messages get smaller and separately and together to have a better overview. The first
barely exceed the 256 KiB mark. Therefore, in the analysis plot in Fig. 10 compares the elapsed time for solving the
that follows, we focus on the benchmarks results with momentum and the continuity equation with 128, 256, 512,
message size up to 256 KiB. and 1024 processes on Beskow and the AWS EC2 cluster.
The second plot in Fig. 10 compares the elapsed time
6.4 Results for assembling both the left-hand side and the right-hand
side in both environments. The time is plotted on a log
We now show the results from the F1 car CFD simulation. scale to be able to represent both functions of different
It was expected that the performance would be at least orders in one plot.
similar to the one on Beskow, given that the latency, The first and second plot in Fig. 11 show the total
bandwidth, and most relevant MPI collectives results for elapsed time and speedup from 128 to 1024 processes (with
our targeted message size range of 4 B to 256 KiB are one core per processor). The third plot in Fig. 11 shows the
competing, and the NAS benchmarks show favorable relative speedup of the two major components of the run-
results for AWS. We verified that this is indeed the case. time, associated with the momentum and the continuity in
The performance of the scientific application was mea- both environments (with the first datum taken as a
sured similarly to the benchmarks, the mean of 10 runs, on reference).
the same set of nodes, for 200 iterations of the transient speedup AWS
simulations. Table 3 shows the ratio speedup Beskow. In terms of
strong scaling, relative to the first datum in Fig. 10 ( for
6.4.1 Performance comparison 128 processors), Beskow exhibits slightly better scaling
than AWS given that the ratios are less than 100%
The largest part of the application runtime consists of: (Table 3, and the second plot in Fig. 11).
Table 4 quantifies the plots shown in Fig. 10. From it,
• Assembling the matrix (left-hand side of the system)
we can conclude that we get a considerable speedup on the
and the vector (right-hand side of the system) to be
AWS EC2 cluster for the assembly of the system compared
solved. The majority of time in our application is spent

Fig. 10 Performance
comparison between AWS and
Beskow by components, strong
scaling

123
2284 Cluster Computing (2024) 27:2273–2290

Fig. 11 Performance
comparison between AWS and
Beskow, strong scaling

Table 3 Speedup AWS/Beskow % processes per node Fig. 12. Since the size of the problem is
more suitable for a smaller number of nodes, we observe an
Speedup 256p 512p 1024p Mean
even better scaling for up to 1024 processes with 32 pro-
Momentum Total 94.2 94.9 93.66 94.26 cesses per node Fig. 13.
Continuity Total 98.55 81.92 92.84 91.1
6.4.3 Compute cost comparison

In this section we only look at the cost of the compute time.

Table 4 Elapsed time ratio Beskow/AWS % We highlight that depending on the type of EBS volume
Elapsed time ratio 128p 256p 512p 1024p chosen, the cost of EBS can add considerably to the total
cost.
Mom. Solve 114.34 102.13 102.22 100.58 AWS offers three alternatives for the cluster type that
Mom. Assemble 141.75 140.78 142.94 141.9 affect the cost of the compute time (see cluster_type in
Cont. Solve 103.05 99.44 71.61 90.46 Table 2): on-demand (most reliable, with reservation), spot
Cont. Assemble 128.26 129.14 126.21 125.41 (use unused EC2 instances for a lower price), dedicated
(serving a single customer), and mixed strategies.
Figure 14 shows the fluctuations of the price per hour
to Beskow (see line 2 and line 4 in Table 4). For the
for the C5n.18xlarge instances in the past year in the us-
momentum, it is up to 1.4 times faster, and for the conti-
east region, and all its zones. We have run calculations in
nuity, it is up to 1.2 times faster on the AWS EC2 cluster.
the us-east-1a zone - the most stable region. We note that
Solving the systems of PDEs using the Stabilized version
the most volatile zone is us-east-1d with minimum and
of BiConjugate Gradient for the momentum and Conjugate
maximum price of 1.3157$ and 1.5803$ per hour, respec-
Gradient for the continuity (in combination with successive
tively. Figure 15 shows the savings of on-demand over
over-relaxation (SOR) preconditioning) results in a very
spot on the 18th of February 2022, with savings from
similar performance in both environments. Moreover, we
61.41% to 69.86% for the us-east-1a zone.
can see that for the momentum, we get a slightly better
Cost-wise, we observe that the prices for the spot
speedup on AWS, while overall Beskow shows slightly
instances are comparable: 0.024€/core-hour on Beskow (a
better speedup for the whole application (second plot in
flat rate set by the PDC Center for High-Performance
Fig. 10). In terms of elapsed time, the CFD application
Computing), 0.030€/core-hour for C5n.18xlarge spot
better performance on the AWS EC2 cluster than on Bes-
instances (based on the average price for zone us-east-1a
kow (see Table 4, values over 100% are in favor of AWS).
shown in Fig. 14), and 0.047€/core-hour for dedicated
C5n.18xlarge instances, with the exchange rates defined by
6.4.2 Strong scaling on AWS EC2 cloud
the European Central Bank (ECB) on the 18th of February
2022. Using this data as reference, the savings over on-
The application shows very satisfactory scaling on the
AWS EC2 cluster for up to 2304 processes with 72

123
Cluster Computing (2024) 27:2273–2290 2285

Fig. 12 AWS Performance, strong scaling, 72 processes per node

Fig. 13 AWS Performance, strong scaling, 32 processes per node

demand for the past year for the zone us-east-1a range from another tenants if the demand is high when using the spot
69.8% to 70%. instances.
We report the results from running on the spot nodes. Additionally, we have taken all the precautions for
The difference between the spot nodes and the on-demand ensuring that we are not taking into account any results
nodes is in the availability. The spot nodes might become from interrupted instances by setting up interrupt notices,
unavailable and interrupt the execution of the program. The defining the interruption behavior, using the AWS console
availability depends on the current demand and the data tools for finding interrupted instances, checking the billing
center/time zone that we are using. In the reported results, and finally verification of the validity of the results. Given
100% of the runs have been uninterrupted and in fact the that we do not run critical applications, we deem that the
frequency of interruption is less than 5% of the time with spot instances are a cost-effective solution for our needs.
70% and up savings over on-demand. For the C5n family, Since we look at the price per core hour as the main
the billing is per hour and choosing between spot and on- parameter for optimization, it means, slightly more work is
demand does not make a difference in the performance done per hour on AWS than on Beskow (despite the fact
because the access to the node does not vary and since it is that is scales slightly worse than Beskow).
the largest in the family, in both cases, we were the only
tenants (a tenant is a guest on the host machine, a virtual
machine) on the node, with exclusive access to all its 7 Discussion and limitations
resources. We chose to use the C5n.x18large instance in
the C5n family because it is the largest in the family, which The purpose of this paper is to show that there are strong
means that whenever we get an allocation we are not indicators that one can get the very similar performance of
sharing with other users. The only difference is that there is HPC applications for the same money on both the super-
a chance that the computation will be interrupted by computer and the cloud. It is a result of very recent trends

123
2286 Cluster Computing (2024) 27:2273–2290

Fig. 14 Spot pricing for the us-

east region for c5n.18xlarge
instances in the period March
2021 to March 2022

Fig. 15 Current On-demand and Spot pricing for the us-east region for c5n.18xlarge instances

and technologies and there is scarce evidence showing this 7.1 Compute power
conclusion for tightly-coupled HPC applications.
The final performance depends on a number of factors, The performance of benchmarks such as the NAS EP
which, while we can analyze separately, we have no way of (Embarrassingly Parallel) benchmark, largely depend on
quantifying completely how much they will affect the the compute power. The frequency in general does not
performance. Therefore we opt not to do an adjustment of necessarily scale the same way as performance, because it
the kind of scaling the strong scaling results by the ratio of depends on other factors such as architecture, instructions
the compute power of the machines (frequency for exam- set, theoretical and sustained number of instructions/sec,
ple). There are other factors apart from compute power, theoretical and sustained FLOPS, dynamic frequency,
such as network speed, the impact of RAM memory, compiler settings, and system processes running in the
storage speed, MPI implementation details and algorithms, background. In order to assume this approximation, we
MPI tuning, message size, instruction set, types of have to make a lot of assumptions, for the compute related
instructions, turbo boost mode, and other architecture parameters, as well as otherwise. Given the many differ-
specific characteristics of the hardware, which cannot be ences in the architectures, we opted for relying on bench-
directly accounted for with two largely different architec- marks to examine the most influential indicators of the
tures. The MPI implementation differences, algorithm performance, and finally optimize for cost per hour and
tuning, and underlying network can potentially have sig- maintainability.
nificant impact on the performance. For example, as it can
be seen from the MPI benchmarks, the collective algo- 7.2 Network
rithms behind the Intel MPI implementation on AWS yield
decidedly better performance for the window of message The network performance is one of the most important
sizes characteristic for our application, while the latency factors alongside compute power in the HPC world. As the
between two processes does not. architectures grow more complex, so is the technology
Therefore we rely on benchmarks relevant for what we improved and occasionally reinvented. However, the fact is
want to test, i.e. run the same code on the machine and look that network technology lags behind compute performance
at the effective performance. In this case we incrementally advances by large in both on-premise architectures and in
increased the specificity of the benchmarks and finally we cloud environments [37].
ran our application. The InfiniBand interconnect is one of the most popular
ones among supercomputers. The Infinibanch technology

123
Cluster Computing (2024) 27:2273–2290 2287

stretches from the data layer to the transport layer in the bandwidth of AWS exceeds the one of Beskow up to
The Open Systems Interconnection model (OSI model 256KiB. For larger messages sizes, up to 1MiB, Beskow
[60]). The latency is very low, the bandwidth is very high has an edge over AWS. In the 256KiB window of message
up to 200Gb/s with HDR links and 100Gb/s with EDR sizes relevant for our application, Beskow has lower
links in the case of Beskow. This interconnect is universal latency, however AWS exhibits higher bandwidth.
for all traffic types (such as communication between nodes,
storage etc). This is made possible by offloading a signif- 7.3 Software maintenance
icant amount of work previously performed in software to
hardware, such as the one done by the transport protocol. It Comments on cost of maintenance: Spack greatly simpli-
also supports bypassing the OS completely to get directly fies the process of deployment in our case. It is an easy way
to the physical layer when sending messages across the of handling dependencies, and allows for a separate envi-
network, with zero copy, from the application space. This ronment, minimizing conflicts with other versions and
RDMA feature is crucial for capitalizing on the large packages. Installing the FEniCS-HPC stack on a new
interconnect bandwidth, given that repeatedly copying the cluster in general is time-consuming because of the mul-
message to be sent incurs time delays due to the limited titude of dependencies and external libraries it makes use
memory bandwidth. Memory latency and bandwidth of. However, with Spack, the process has been consider-
improvements have been stretched to a limit in the last ably accelerated.
years, and innovative technology, such as 3D-stacked
DRAM, is on the rise. 7.4 Overall performance
The Elastic Fabric Adapter (EFA) is a network device
that similarly to the InfiniBand’s RDMA features has the In terms of compute power, AWS is superior to Beskow,
power to bypass the OS to send communications to other given the more powerful processor and the larger number
instances in the same subnet. The compute nodes in the of cores (Table 1). We have confirmed this with the Open
AWS cluster we have created are placed in the same sub- MPI and MPI results for the EP benchmark, part of the
net. The user can configure MPI with the Libfabric NAS Parallel Benchmarks suite. Our application is bal-
framework, the interface between the user space and the anced between compute and memory intensive. Therefore,
kernel space. This framework provides access to the drivers whereas Beskow has better networking, lower latency for
of the EFA network device, making it possible to bypass most of the relevant window of message sizes and com-
the OS. AWS has also developed a new transport protocol parable and superior bandwidth for medium and large
for their needs, the scalable reliable datagram (SRD) [61]. message sizes correspondingly, AWS has more compute
The SRD focuses on minimizing the traffic congestion and power.
therefore latency for large messages, as well as combating On the other hand, the MPI communication makes up a
load imbalance (crucial for HPC applications). It sends large percentage of the application runtime, especially as
packages out of order, on various paths with relatively low the number of processes increases. From the benchmarks
traffic, to achieve a bandwidth of 100Gb/s. SRD is there- Section 5, we can clearly see that the elapsed time
fore faster for larger messages, with a high bandwidth increases with the number of processors and message size.
overall, and a decidedly improved performance of collec- In the profiling case in Subsection 6.3 with 256 processes
tive MPI calls. This is in accordance with the findings in on 8 nodes, the MPI communications takes 14% of the
the Section 5. Similarly to the transport protocol in Infi- runtime. From the profiling and the results of the OSU
niBand, the workings of SRD are offloaded to hardware, benchmarks for the most time-consuming MPI functions,
the in-house Nitro networking card of AWS. we can conclude that on the AWS platform, we get an edge
Cloud providers that offer HPC capabilities use different over Beskow, when it comes to MPI communications.
network and routing protocols, however most of the ones Through the profiling and benchmarking, we have been
that deliver network bandwidth larger than 100 Gbps rely able to gain an insight into the impact of key factors on the
on RDMA or RDMA-like technologies, omnipresent in on- overall performance, as well as the performance of separate
premise infrastructures [37]. components, such as MPI communication, assembly, and
In summary, Beskow has notably lower ping-pong solving the resulting systems of equations. Finally, we
latency for smaller message sizes, however, as the message observe that it is possible to obtain a slightly better per-
size increases, in the middle range up to 256 KiB, AWS is formance on the cloud in terms of elapsed time.
on par with Beskow. For the larger message sizes, Beskow
is again on top in terms of latency. Both environments have
a theoretical bandwidth of 100Gb/s and from the band-
width benchmarks, we can conclude that the effective

123
2288 Cluster Computing (2024) 27:2273–2290

8 Conclusion Declarations

In this paper, we have shown the results from running the Competing interest The authors have not disclosed any competing
interests.
OSU micro-benchmarks relevant for our target HPC
application and the NAS Parallel macro-benchmarks in two Open Access This article is licensed under a Creative Commons
environments: AWS EC2 cluster and Beskow supercom- Attribution 4.0 International License, which permits use, sharing,
puter. We conducted a comparative analysis for a wide adaptation, distribution and reproduction in any medium or format, as
long as you give appropriate credit to the original author(s) and the
range of message sizes and processes of the full sets of source, provide a link to the Creative Commons licence, and indicate
benchmarks to conclude that our application is suitable to if changes were made. The images or other third party material in this
be run in the cloud. The final results show strong scaling up article are included in the article’s Creative Commons licence, unless
to 2304 cores in the cloud, and the performance is highly indicated otherwise in a credit line to the material. If material is not
included in the article’s Creative Commons licence and your intended
competitive with the one on Beskow. use is not permitted by statutory regulation or exceeds the permitted
Finally, we have successfully ported a large-scale HPC use, you will need to obtain permission directly from the copyright
application to the cloud, providing further evidence of a holder. To view a copy of this licence, visit [Link]
shift that began in 2020, contrary to an abundance of prior org/licenses/by/4.0/.
feasibility studies. The performance competes with the one
on the supercomputer. Additionally, we get a fully con-
References
figurable and scalable environment we can tweak for our
needs. As a conclusion, this study confirms that due to 1. Birje, M.N., Challagidad, P.S., Goudar, R., Tapale, M.T.: Cloud
recent advances, most notably on the networking tech- computing review: concepts, technology, challenges and security.
nologies front, the cloud is getting closer to the on-premise Int. J. Cloud Comput. 6(1), 32–57 (2017). [Link]
IJCC.2017.083905
supercomputer, performance-wise, for HPC applications.
2. Walker, E.: Benchmarking amazon ec2 for hig-performance
The increased adoption of cloud within research and fur- scientific computing. ; login:: the magazine of USENIX & SAGE
thermore, the most recent advances in network technology 33(5), 18–23 (2008)
by cloud service platforms (AWS, Azure making available 3. Napper, J., Bientinesi, P.: Can cloud computing reach the top500?
In: Proceedings of the Combined Workshops on UnConventional
network bandwidth of 200 Gbps [62–64]) are opening yet
High Performance Computing Workshop Plus Memory Access
again new frontiers for further exploration of the future of Workshop, pp. 17–20 (2009). [Link]
HPC. 1531671
4. Ostermann, S., Iosup, A., Yigitbasi, N., Prodan, R., Fahringer, T.,
Acknowledgements The authors would like to thank Johan Jansson Epema, D.: A performance analysis of ec2 cloud computing
for initiating the idea of exploring the possibilities of the cloud, services for scientific computing. In: International Conference on
providing support in obtaining the resources, and the initial mesh for Cloud Computing, pp. 115–131 (2009). [Link]
the CFD simulation. This project has received funding from the 978-3-642-12636-9_9. Springer
European Union’s Marie Skłodowska-Curie Actions (MSCA) Inno- 5. Netto, M.A., Calheiros, R.N., Rodrigues, E.R., Cunha, R.L.,
vative Training Network (ITN) H2020-MSCA-ITN-2017 under grant Buyya, R.: Hpc cloud for scientific and business applications:
agreement No. 764979. Taxonomy, vision, and research challenges. ACM Computing
Surveys (CSUR) 51(1), 1–29 (2018). [Link]
Author contributions All authors contributed to the study conception 3150224
and design. The computations and the data analysis were performed 6. Arinze, B., Anandarajan, M.: Factors that determine the adoption
by TD. The first draft of the manuscript was written by TD and all of cloud computing: A global perspective. Int. J. Enterp. Inf. Syst.
authors commented on previous versions of the manuscript. All 6(4), 55–68 (2010). [Link]
authors read and approved the final manuscript. 7. He, Q., Zhou, S., Kobler, B., Duffy, D., McGlynn, T.: Case study
for running hpc applications in public clouds. In: Proceedings of
Funding Open Access funding provided thanks to the CRUE-CSIC the 19th ACM International Symposium on High Performance
agreement with Springer Nature. This work was supported by the Distributed Computing, pp. 395–401 (2010). [Link]
European Union’s Marie Skłodowska-Curie Actions (MSCA) Inno- 1145/1851476.1851535
vative Training Network (ITN) H2020-MSCA-ITN-2017 under grant 8. Jackson, K.R., Ramakrishnan, L., Muriki, K., Canon, S., Cholia,
agreement No 764979. The authors declare that no funds, grants, or S., Shalf, J., Wasserman, H.J., Wright, N.J.: Performance analysis
other support were received during the preparation of this manuscript. of high performance computing applications on the amazon web
services cloud. In: 2010 IEEE Second International Conference
Data availability The datasets generated during and/or analysed dur- on Cloud Computing Technology and Science, pp. 159–168
ing the current study are not publicly available due to awaiting (2010). [Link] IEEE
publication but are available from the corresponding author on rea- 9. Rehr, J.J., Vila, F.D., Gardner, J.P., Svec, L., Prange, M.: Sci-
sonable request. entific computing in the cloud. Comput. sci. Eng. 12(3), 34–43
(2010). [Link]
10. Gupta, A., Milojicic, D.: Evaluation of hpc applications on cloud.
In: 2011 Sixth Open Cirrus Summit, pp. 22–26 (2011). https://
[Link]/10.1109/OCS.2011.10. IEEE

123
Cluster Computing (2024) 27:2273–2290 2289

11. Zhai, Y., Liu, M., Zhai, J., Ma, X., Chen, W.: Cloud versus in- 27. 10 years later: Cloud computing is closing the performance gap.
house cluster: evaluating amazon cluster compute instances for [Link]
running mpi applications. In: State of the Practice Reports, 28. Zhuang, J., Jacob, D.J., Lin, H., Lundgren, E.W., Yantosca, R.M.,
pp. 1–10 (2011) Gaya, J.F., Sulprizio, M.P., Eastham, S.D.: Enabling high-per-
12. Coghlan, S.: The magellan final report on cloud computing. formance cloud computing for earth science modeling on over a
Technical report (dec 2011). [Link] thousand cores: Application to the GEOS-chem atmospheric
13. Mehrotra, P., Djomehri, J., Heistand, S., Hood, R., Jin, H., chemistry model. Journal of Advances in Modeling Earth Sys-
Lazanoff, A., Saini, S., Biswas, R.: Performance evaluation of tems 12(5) (2020). [Link]
amazon ec2 for nasa hpc applications. In: Proceedings of the 3rd 29. Panda, D.K., Subramoni, H., Chu, C.-H., Bayatpour, M.: The
Workshop on Scientific Cloud Computing, pp. 41–50 (2012). mvapich project: transforming research into high-performance
[Link] mpi library for hpc community. J. Comput. Sci. 52, 101208
14. Expósito, R.R., López Taboada, G., Pardo, X.C., Tourino, J., (2021). [Link]
Doallo Biempica, R.: Running scientific codes on amazon ec2: A 30. Fernandez, A.: Evaluation of the performance of tightly coupled
performance analysis of five high-end instances. J. Comput. Sci. parallel solvers and mpi communications in iaas from the public
Technol. 13(3), 153–159 (2013) cloud. IEEE Trans. Cloud Comput. 10(4), 2613–2622 (2022).
15. Expósito, R.R., Taboada, G.L., Ramos, S., Touriño, J., Doallo, [Link]
R.: Performance analysis of hpc applications in the cloud. Futur. 31. White., Emma.: Running ANSYS Fluent on Amazon EC2 C5n
Gener. Comput. Syst. 29(1), 218–229 (2013). [Link] with Elastic Fabric Adapter (EFA). AWS EC2 (2019). https://
1016/[Link].2012.06.009 [Link]/es/blogs/compute/running-ansys-fluent-on-ama
16. Sadooghi, I., Martin, J.H., Li, T., Brandstatter, K., Maheshwari, zon-ec2-c5n-with-elastic-fabric-adapter-efa/
K., de Lacerda Ruivo, T.P.P., Garzoglio, G., Timm, S., Zhao, Y., 32. Appa, J., Turner, M., Ashton, N.: Performance of cpu and gpu
Raicu, I.: Understanding the performance and potential of cloud hpc architectures for off-design aircraft simulations. In: AIAA
computing for scientific applications. IEEE Transactions on Scitech 2021 Forum, p. 0141 (2021). [Link]
Cloud Computing 5(2), 358–371 (2015). [Link] 2021-0141
TCC.2015.2404821 33. Amazon Web Services: AWS HPC. Amazon Web Services
17. Rad, P., Chronopoulos, A., Lama, P., Madduri, P., Loader, C.: (2023). [Link]
Benchmarking bare metal cloud servers for hpc applications. In: 34. Azure High-Performance Computing
2015 IEEE International Conference on Cloud Computing in 35. Google: Google Cloud High-Performance Computing. Google
Emerging Markets (CCEM), pp. 153–159 (2015). [Link] (2023). [Link]
10.1109/CCEM.2015.13. IEEE 36. Chang, Y.-T., Hood, R.T., Jin, H., Heistand, S.W., Cheung, S.H.,
18. Freniere, C., Pathak, A., Raessi, M., Khanna, G.: The feasibility Djomehri, M.J., Jost, G., Kokron, D.S.: Evaluating the suitability
of amazon’s cloud computing platform for parallel, gpu-accel- of commercial clouds for nasa’s high performance computing
erated, multiphase-flow simulations. Comput. Sci. Eng. 18(5), applications: A trade study. Technical report (2018)
68–77 (2016). [Link] 37. De Sensi, D., De Matteis, T., Taranov, K., Di Girolamo, S.,
19. Gupta, A., Faraboschi, P., Gioachin, F., Kale, L.V., Kaufmann, Rahn, T., Hoefler, T.: Noise in the clouds: Influence of network
R., Lee, B.-S., March, V., Milojicic, D., Suen, C.H.: Evaluating performance variability on application scalability. Proceedings of
and improving the performance and scheduling of HPC appli- the ACM on Measurement and Analysis of Computing Systems
cations in cloud. IEEE Trans. Cloud Comput. 4(3), 307–321 6(3), 1–27 (2022). [Link]
(2016). [Link] 38. Intel: HPC Performance Characterization. Intel Corporation
20. Ditter, A., Graf, G., Fey, D.: Fe2vcl2: from bare metal to high (2023). [Link]
performance computing on virtual clusters and cloud infrastruc- profiler/user-guide/2023-0/hpc-performance-characterization-
ture. In: Proceedings of the 4th Workshop on CrossCloud [Link]
Infrastructures & Platforms, pp. 1–7 (2017) 39. Amazon Web Services: AWS Parallel Cluster. Amazon Web
21. Balis, B., Figiela, K., Jopek, K., Malawski, M., Pawlik, M.: Services (2021). [Link]
Porting hpc applications to the cloud: A multi-frontal solver case 40. Amazon Web Services: Elastic Fabric Adapter. Amazon Web
study. J. Comput. Sci. 18, 106–116 (2017). [Link] Services (2021)
1016/[Link].2016.09.006 41. Amazon Web Services: AWS ParallelCluster Documentation.
22. Mohammadi, M., Bazhirov, T.: Comparative benchmarking of Amazon Web Services (2021). [Link]
cloud computing vendors with high performance linpack. In: parallelcluster/
Proceedings of the 2nd International Conference on High Per- 42. Barr., Jeff: Now Available - Elastic Fabric Adapter (EFA) for
formance Compilation, Computing and Communications, pp. 1–5 Tightly-Coupled HPC Workloads. Amazon Web Services (2019).
(2018). [Link] [Link]
23. Breuer, A., Cui, Y., Heinecke, A.: Petaflop seismic simulations in adapter-efa-for-tightly-coupled-hpc- workloads/
the public cloud. In: International Conference on High Perfor- 43. Amazon Web Services: Amazon EC2 Compute Optimized
mance Computing, pp. 167–185 (2019). [Link] Instances. Amazon Web Services (2023). [Link]
978-3-030-20656-7_9. Springer [Link]/AWSEC2/latest/UserGuide/compute-optimized-instan
24. NASA: NASA SC19 - High-Performance Computing in the [Link]
Azure Cloud. NASA Technical Reports Server (2023) 44. Amazon Web Services: Amazon EC2 Instance Types. Amazon
25. Maliszewski, A.M.: Impact of network interconnection in cloud Web Services (2023). [Link]
computing environments for high-performance computing types/
applications (2021) 45. Amazon Web Services: AWS Nitro System. Amazon Web Ser-
26. Maliszewski, A.M., Roloff, E., Carreño, E.D., Griebler, D., vices (2021). [Link]
Gaspary, L.P., Navaux, P.O.A.: performance and cost-aware hpc 46. impact of using multi-levels of parallelism on hpc applications
in clouds: A network interconnection assessment. In: 2020 IEEE performance hosted on azure cloud computing. [Link]
Symposium on Computers and Communications (ISCC), pp. 1–6 1504/IJHPCN.2019.098579
(2020). [Link]

123
2290 Cluster Computing (2024) 27:2273–2290

47. Ouro, P., Lopez-Novoa, U., Guest, M.F.: On the performance of a Higher Packet Processing Performance. Amazon Web Services.
highly-scalable computational fluid dynamics code on amd, arm [Link]
and intel processor-based hpc systems. Comput. Phys. Commun. pute-optimized-and-memory-optimized-amazon-ec2-instances-
269, 108105 (2021). [Link] with-higher-packet-processing-performance/
48. Gamblin, T., LeGendre, M., Collette, M.R., Lee, G.L., Moody, 64. Azure HBv2-Series Virtual Machines. [Link]
A., De Supinski, B.R., Futral, S.: The spack package manager: com/en-us/azure/virtual-machines/hbv2-series
bringing order to hpc software chaos. In: Proceedings of the
International Conference for High Performance Computing, Publisher’s Note Springer Nature remains neutral with regard to
Networking, Storage and Analysis, pp. 1–12 (2015). [Link] jurisdictional claims in published maps and institutional affiliations.
org/10.1145/2807591.2807623
49. AWS: AWS Workshop. AWS (2023). [Link]
[Link]/workshops/dd0ffcb3-ffc1-4b58-8c4b-
09f9846549c7/en-US Tamara Dancheva is a PhD.
50. PDC Center for High Performance Computing: Beskow. PDC student at the Basque Center for
Center for High Performance Computing (2021) Applied Mathematics. She
51. Network-Based Computing (NBC) Laboratory, The Ohio State received her Master in Compu-
University: MVAPICH: MPI over InfiniBand, Omni-Path, Eth- tational Engineering at the
ernet/iWARP, and RoCE. Network-Based Computing (NBC) University of Strasbourg. She
Laboratory, The Ohio State University (2021). [Link] holds a bachelor in Computer
[Link]/benchmarks/ Science and Engineering. Her
52. Lin, H.: Abnormal in-node latency with EFA enabled. Github, research focus is on computa-
aws-parallelcluster (2019). [Link] tional mechanics and HPC.
lelcluster/issues/1143 Within the frame of the H2020
53. NASA Advanced Supercomputing (NAS) Division NASA ENABLE project she is working
Advanced Supercomputing (NAS) Division (2021). [Link] on enhancing mechanical
[Link]/publications/[Link] behavior law mod els within an
54. Hoffman, J., Jansson, J., Jansson, N.: Fenics-hpc: Automated HPC setting.
predictive high-performance finite element computing with
applications in aerodynamics. In: International Conference on Unai Alonso Specialist in the
Parallel Processing and Applied Mathematics, pp. 356–365 thermo-mechanical modelling
(2015). [Link] Springer of machining processes and has
55. Jansson, J., Krishnasamy, E., Leoni, M., Jansson, N., Hoffman, J.: taken part on several national
Time-resolved adaptive direct fem simulation of high-lift aircraft and international projects. His
configurations. In: Numerical Simulation of the Aerodynamics of recent work deals with machin-
High-Lift Configurations, pp. 67–92. Springer, ??? (2018). ing of composites and titanium
[Link] alloys produced by additive
56. Jansson, N., Hoffman, J., Nazarov, M.: Adaptive simulation of manufacturing. Moreover, he
turbulent flow past a full car model. In: SC’11: Proceedings of has worked in the research
2011 International Conference for High Performance Computing, center Ideko [Link] and at the
Networking, Storage and Analysis, pp. 1–8 (2011). IEEE Universities of Magdeburg and
57. Hoffman, J., Jansson, J., de Abreu, R.V., Degirmenci, N.C., the ENSAM in France.
Jansson, N., Müller, K., Nazarov, M., Spühler, J.H.: Unicorn:
parallel adaptive finite element simulation of turbulent flow and
fluid-structure interaction for deforming domains and complex
geometry. Comput. Fluids 80, 310–319 (2013) Dr Michael Barton is currently
58. Integrated Performance Monitor (IPM): Integrated Performance an Iker basque Research Asso-
Monitor. Integrated Performance Monitor (IPM) (2021). http:// ciate and Ramon & Cajal 2017
[Link]/ Research Fellow at BCAM.
59. Jansson, N.: A hybrid mpi?pgas approach to improve strong Prior joining BCAM, he was a
scalability limits of finite element solvers. In: 2020 IEEE Inter- post-doctoral fellow at JKU
national Conference on Cluster Computing (CLUSTER), Linz (Austria), Technion
pp. 303–313 (2020). IEEE (Israel), King Abdullah Univer-
60. Kumar, S., Dalal, S., Dixit, V.: The osi model: overview on the sity of Science and Technology
seven layers of computer networks. Int. J. Comput. Sci. Inf. KAUST (Saudi Arabia). He has
Technol. Res. 2(3), 461–466 (2014) published over 50 peer reviewed
61. Shalev, L., Ayoub, H., Bshara, N., Sabbag, E.: A cloud-optimized research articles. His research
transport protocol for elastic and scalable hpc. IEEE Micro 40(6), spans numerical analysis, geo-
67–73 (2020). [Link] metric modelling and process-
62. Amazon Web Services: AWS EC2 HPC6i Instance Types. ing, and approximation and
Amazon Web Services (2022). [Link] rationalization of curved (NURBS objects.
instance-types/hpc6i/
63. Amazon Web Services: New General Purpose, Compute Opti-
mized, and Memory Optimized Amazon EC2 Instances with

123