Axon: A novel systolic array architecture for improved run time and

energy efficient GeMM and Conv operation with on-chip im2col

Md Mizanur Rahaman Nayan, Ritik Raj, Gouse Basha Shaik, Tushar Krishna, Azad J Naeemi
Department of Electrical and Computer Engineering
Georgia Institute of Technology, USA
ABSTRACT of the systolic array based designs are optimized for the convolution
General matrix multiplication (GeMM) is a core operation in virtu- workload [13, 30, 31, 37, 47].
ally all AI applications. Systolic array (SA) based architectures have Eyeriss [7] addresses this memory disadvantage by proposing a
shown great promise as GeMM hardware accelerators thanks to their novel dataflow titled row stationary. However, the processing ele-
speed and energy efficiency. Unfortunately, SAs incur a linear delay ments (PE) in Eyeriss must have large Multiply-Accumulate (MAC)
in filling the operands, due to unidirectional propogation via pipeline units with large register files (RF) compared to those in simpler sys-
arXiv:2501.06043v1 [[Link]] 10 Jan 2025

latches. In this work, we propose a novel in-array data orchestra- tolic arrays. This is because RF requires storing a filter row into
tion technique in SAs where we enable data feeding on the principal PE’s buffer. Also, IFMAP needs to copy in each set of PEs that are
diagonal followed by bi-directional propagation. This improves the working on producing results for a filter, unlike SA where the same
runtime by up to 2× at minimal hardware overhead. In addition, the IFMAP copy is reused along all the filters. Like Eyeriss’s row station-
proposed data orchestration enables convolution lowering (known ary dataflow, FlexFlow[31] has been proposed to address the above
as im2col) using a simple hardware support to fully exploit input mentioned challenge. Both FlexFlow and row stationary dataflows
feature map reuse opportunity and significantly lower the off-chip do not follow the fundamental systolic array architecture. Although
memory traffic resulting in 1.2× throughput improvement and 2.17× these designs are efficient in computation, data transmission is de-
inference energy reduction during YOLOv3 and RESNET50 work- pendent on bus communication which limits the frequency of the
load on average. In contrast, conventional data orchestration would overall system [47]. Gemmini[16] introduces optional im2col units
require more elaborate hardware and control signals to implement which prevent CPU from managing the im2col but this design still
im2col in hardware because of the data skew. We have synthesized suffers from frequent memory accesses. SPOTS[39] reduces the L2
and conducted place and route for 16×16 systolic arrays based on the or OFF-chip memory access but it introduces a large intermediate
novel and conventional orchestrations using ASAP 7nm PDK and buffer. Recently, some work has been done for on-the-fly im2col that
found that our proposed approach results in 0.211% area and 1.6% replaces software im2col requirement and thus helps to reduce data
power overheads. transfer [15, 28]. However, this approach comes with a major hard-
ware overhead including counters, intermediate registers, FIFO, and
KEYWORDS feed registers, and requires complex control signals.
Systolic array, GeMM, Convolution, im2col, AI accelerators In this work, we introduce a new systolic array architecture called
Axon where we propose a novel in-array data orchestration tech-
1 INTRODUCTION nique along with a new data transfer from the scratchpad to the
array without changing the systolic array’s simple design and its
General Matrix multiplication (GeMM) is at the core of current AI
reuse capability. In Axon data are fed to PE on the principal diag-
applications and in most cases the throughput is limited by how fast
onal followed by bi-directional propagation, unlike systolic array’s
it can be executed. General-purpose computers are quite inefficient
uni-directional propagation. The proposed design provides a direct
when it comes to matrix multiplication because of their complex
improvement on runtime for all workloads regardless of their opti-
architectures and there have been major research and development
mal dataflows including output stationary (OS), weight stationary
efforts on domain-specific architectures to overcome this bottleneck.
(WS), and input stationary (IS). We present a runtime equation that
Hardware accelerators like Graphical Processing Units (GPU) are
quantifies the improvements that one can achieve under various
one of them built according to SIMD (Single Instruction Multiple
scenarios. We also exploit data reuse opportunity when perform-
Data Stream) architecture. GPUs are very popular for their highly
ing convolution by replacing software im2col with a simple 2-to-1
parallelization capability which helps to run AI applications faster.
multiplexer (MUX) which requires a significantly smaller hardware
Unfortunately, they are extremely power-hungry because of their
overhead compared to existing hardware support for im2col [15, 28].
general-purpose design, and the fact that all communication is done
The proposed redesign of in-array data propagation makes it possi-
through memory hierarchy[18, 25].
ble to adopt this data reuse opportunity using a simple 2-to-1 MUX.
Systolic array-based architecture has become popular for GeMM
We also adopted the idea of Zero gating to exploit sparsity in both
because of its simple design, lower memory bandwidth requirement
the input feature map (IFMAP) and FILTERS [15]. Finally, we experi-
and energy efficiency [12, 13, 20, 23, 26, 29, 44, 45] and almost all
mented and validated the runtime improvement through the proposed
commerical DNN accelerators use systolic array [1, 3, 5, 7–10, 18,
orchestration, im2col support, and sparsity handling on existing state-
24, 25, 33, 41, 42, 47]. Open-source frameworks like Gemmini[16]
of-the-art (SOTA) workloads like transformer (GeMM), conformer
also generate systolic array-based designs. Unlike GeMM, Convo-
(Conv and GeMM) and CNN (conv). Also, we present hardware over-
lution operation needs im2col algorithm to map a 3-D tensor to an
head comparison and power consumption reduction with prior work
equivalent 2-D matrix which enables convolution workload to run
by performing the full synthesis and physical design process for an
in GeMM accelerator. In conventional SA, software-based im2col is
application specific integrated circuit (ASIC) implementation on the
used to generate the corresponding matrices which are then stored
proposed architecture. The key contributions of this paper are as
in SRAM buffers to feed the array during execution. Although it
follow:
is computationally efficient, it requires significant on-chip memory
due to repeated elements in convolution windows. However some
 Md Mizanur Rahaman Nayan, Ritik Raj, Gouse Basha Shaik, Tushar Krishna, Azad J Naeemi
Department of Electrical and Computer Engineering
Georgia Institute of Technology, USA

• A novel in-array data orchestration in systolic arrays that The original operand matrices are projected into the available spatio-
speeds up GeMM irrespective of dataflow (OS, IS, WS) and temporal dimensions. For example, during the multiplication of two
workload shapes. Furthermore, Axon achieves 2x speedup matrices of shapes 𝑀 × 𝐾 and 𝐾 × 𝑁 , the dimension 𝑀 is mapped
on relatively memory bound operations like GEMV and DW- to 𝑆𝑅 , dimension 𝑁 is mapped to 𝑆𝐶 and the dimension 𝐾 to 𝑇 for
conv due to lower feeding latency and no data skew. OS dataflow. Mapping for three dataflows are summarized in Table.1.
• An im2col hardware support that is simple to program and However, Large GEMM problems are managed on smaller systolic
adds insignificant hardware overhead (using 2-to-1 MUXes
only) for exploiting input feature map reuse opportunity Dataflow Mapping
which significantly lowers the off-chip memory traffic during OS (𝑆𝑅 = 𝑀, 𝑆𝐶 = 𝑁 ,𝑇 = 𝐾)
convolution WS (𝑆𝑅 = 𝐾, 𝑆𝐶 = 𝑀,𝑇 = 𝑁 )
• Unified PE design compatible with Axon architecture which IS (𝑆𝑅 = 𝐾, 𝑆𝐶 = 𝑁 ,𝑇 = 𝑀)
is programmable to exploit all the three dataflows (WS/IS/OS)
Table 1: Mapping of GEMM dimension along the array during
• The capability of exploiting sparsity in IFMAP and FILTERS to
different dataflows
reduce power consumption using the Zero gating technique arrays through tiling in scale-up (one large array) or scale-out (mul-
• Physical design and layout implementation of Axon using a tiple smaller arrays) as depicted in Fig. 2. Runtime equation for the
7-nm FiNFET PDK for scalability and backend analysis scale up and scale out is modified as Eq. 2 and 3, respectively,
2 BACKGROUND AND MOTIVATION
𝜏𝑠𝑐𝑎𝑙𝑒𝑢𝑝 = (2𝑅 + 𝐶 + 𝑇 − 2) ∗ (𝑆𝑅 /𝑅) ∗ (𝑆𝐶 /𝐶) (2)
2.1 Conventional systolic array and dataflows
𝜏𝑠𝑐𝑎𝑙𝑒𝑜𝑢𝑡 = (2𝑅 + 𝐶 + 𝑇 − 2) ∗ (𝑆𝑅′ /𝑅) ∗ (𝑆𝐶′ /𝐶) (3)
Dataflows in a systolic array define how the elements of operand ma-
where 𝑅 and 𝐶 are the number of rows and columns of the systolic
trices and partial sums propagate inside the array. In the OS dataflow,
array, 𝑆𝑅′ and 𝑆𝐶′ are (𝑆𝑅 /𝑃𝑅 ) and (𝑆𝐶 /𝑃𝐶 ), respectively, 𝑃𝑅 is the
partial sums remain stationary while operands from both input and
partition number across the rows, and 𝑃𝐶 is the partition number
weight propagate through the array in a systolic order. At the end of
across the columns. If we break down the runtime equation, we find
the operation, outputs are extracted from the array. In WS, weights
the following three components:
are pre-filled inside the array and remain stationary until the input
propagates through the whole systolic array. In each cycle, partial • Time for both operands to reach the farthest PE with respect
sums are generated in each PE and propagate downward to be added to the feeder PEs, 𝑅 + 𝐶 − 2
in the next cycle with another PE-generated partial sum. Finally out- • Number of multiplications each PE performs, 𝑇
puts are collected from the bottom row of the array. IS is similar to • Readout from the array, 𝑅
WS with the only difference being that inputs instead of weights are For any dataflow (OS, WS, and IS) the number of multiplications
pre-filled in the array and remain stationary. Fig. 1 demonstrates in- each PE performs is determined by the temporal dimension, 𝑇 . The
array dataflows in the conventional systolic array, in which the input readout time is determined by the number of rows of the systolic
feature map/one operand matrix is loaded from the input feature map array, 𝑅. However, the first component is determined by the in-array
buffer to the systolic array through the leftmost PEs and propagates data propagation direction. For the conventional systolic array, it is
from left to right. Weights/another operand matrix are loaded from the Manhattan distance where the distance is defined by the distance
the weight buffer through PEs in the top rows and propagate from top from the vertical and horizontal axes. For the farthest PE (bottom
to bottom. Each PE in the array is responsible for one MAC operation right corner) in the array, the distance is (𝑅 + 𝐶 − 2).
and propagates inputs and outputs based on the dataflow type.

Figure 2: a) Scale up (left) b) Scale out (right). In scale up one

large monolithic array is used whereas in scale out multiple
Figure 1: Data feeder from the buffer and in-array dataflow in systolic arrays are used to generate output.
conventional systolic array with PE’s architecture
2.2 Runtime Modeling 3 AXON DATA ORCHESTRATION STRATEGY
SCALE-SIM[35] developed an analytical model for systolic array run- In the conventional orchestration, it takes (𝑅 + 𝐶 − 2) cycles for
time, which is applicable for all the three dataflows discussed above. the operands to reach the farthest PE. The Axon data orchestration
According to the model, for the operand matrices of dimensions 𝑆𝑅 ×𝑇 shown in Fig. 3 improves this portion of the runtime by feeding
and 𝑇 × 𝑆𝐶 respectively the runtime is as follows: the operand matrices to the systolic array through the PEs on the
principal diagonal which we call feeder PEs (semi-transparent thick
𝜏 = 2𝑆𝑅 + 𝑆𝑐 + 𝑇 − 2 (1) arrows). Once data is inserted into the systolic array, data movement
where 𝑆𝑅 and 𝑆𝐶 are the spatial dimensions along which computation inside the array is orchestrated according to Fig. 3 (solid-thin arrows)
is mapped, and 𝑇 represents the corresponding temporal dimension. to perform the matrix multiplication. The feeder PEs except the two
Axon: A novel systolic array architecture for improved run time and energy efficient GeMM and Conv operation with on-chip im2col

Figure 5: Axon data orchestration in the rectangular systolic

array. The columns that do not have any PEs on the principal
Figure 3: a) Axon’s in-array data orchestration. Thick semi-
diagonal will be fed through PEs at the bottom of the array
transparent arrows indicate data movement into the systolic
with zero padding based on the distance. Notice third column
array from buffers and thin solid arrows indicate data move-
is fed with zero-padded by one and the fourth column is fed
ment inside the array among the PEs. The same colors on PEs
with zero-padded by two.
represent operands’ arrival at the same cycle whereas PEs on
the principal diagonal receive the operands on the first cycle Dataflow Systolic array Axon
directly from the buffers. b) im2col implementation. Note that, OS 2𝑀 + 𝐾 + 𝑁 − 2 max(𝑀, 𝑁 ) + 𝑀 + 𝐾 − 1
each MUX allows feeder PEs to receive data either from buffer WS 2𝐾 + 𝑀 + 𝑁 − 2 max(𝑀, 𝐾) + 𝐾 + 𝑁 − 1
or from immediate PE on the diagonal. IS 2𝐾 + 𝑀 + 𝑁 − 2 max(𝑁 , 𝐾) + 𝐾 + 𝑀 − 1
Table 2: Runtime for SA and Axon
is changed to 𝑚𝑎𝑥 (𝑅, 𝐶) − 1. For a square systolic array(𝑅 = 𝐶), it
is simply (𝑅 − 1) which is half of the (2𝑅 − 2) for the conventional
systolic array. The improvement for non-square arrays is smaller but
is always greater than 1. Table. 2 summarizes the runtime for various
dataflows w.r.t their operand matrix shape considering 𝑆𝑅 = 𝑅 and
𝑆𝐶 = 𝐶. Fig. 6 shows that for any shape of the array (𝑅, 𝐶) the factors
that determine the time it takes for the operands to reach the farthest
PE are always lower in Axon data orchestration. For example, in a
systolic array of shape (256, 256) the compute time required to reach
the farthest PE is reduced from 510 cycles to 255 cycles. However,
the overall runtime improvement for a scale-out design with smaller
arrays is limited by two other factors (i.e. Temporal dimension length
and Readout time) according to Amdahl’s law.
Figure 4: Simple 3 × 3 GeMM example that validates Axon data 3.2 Axon’s hardware support for im2col
orchestration. Partial products labeled with the same colors
During im2col, IFMAP is converted to conv windows depending
indicate that they are generated in the same cycle. All operands
on the shape of the FILTER. Each conv window is responsible for
are fed to SA through PE on the principal diagonal.
generating one element of OFMAP. In the example shown in Fig. 7,
corner PEs transfer the operands in two directions, unlike other PEs Filter and IFMAP are of shape 3 × 3 and 6 × 6 respectively. Thus
on the array. Filter elements are allowed to propagate towards both OFMAP shape is 4 × 4. In other words, there are in total 16 conv
adjacent upward and downward PEs whereas IFMAP elements are windows corresponding to 16 elements of the 4 × 4 OFMAP. In the
allowed to propagate towards both adjacent right and left PEs. Other example only 4 conv windows correspond to 1𝑠𝑡 row of OFMAP has
PEs in the array propagate the operand in the same direction as they been shown. Note that in conv windows, there are 18 unique elements
receive data. Note that in this way, we do not need any extra hardware. and the remaining 18 elements have been repeated (50% repetition)
We only need to rearrange the in-array dataflow direction and feed in a pattern that can be formulated in terms of the FILTER length,
the array from the PEs on the principal diagonal. Moreover, Unlike 𝑛. The number of common elements in consecutive conv windows
conventional SA Axon does not need to stream the operand matrices is 𝑛(𝑛 − 1), which is 3 × 2 or 6 for the demonstrated example. Also
in a skewed manner which increases PE utilization. Fig. 4 shows a toy between each adjacent conv window, there is a cyclic pattern with a
example to demonstrate the Axon data orchestration through a two period of 𝑛 where there are 𝑛 − 1 common elements. This repetition
3 × 3 matrix multiplication. While systolic arrays are often square, the becomes more prominent for large FILTER and IFMAP shapes which
Axon data orchestration can also be extended for rectangular systolic results in excessive memory traffic and a need for either a large on-
arrays and reduce the runtime as shown in Fig. 5. For the columns chip memory or expensive DRAM access. In Axon, we leverage this
that do not have any PEs on the principal diagonal, operands are repetition pattern to reuse the elements directly from the PEs instead
fed from the bottom PE of the column with spatially skewed data of DRAM access or having a large on-chip memory.
similar to a conventional systolic array which ensures accurate data Fig. 3(b) demonstrates the proposed Axon’s simple im2col hard-
orchestration for the PEs. ware support where each PE on the principal diagonal can be fed
with IFMAP elements either directly from IFMAP SRAM buffers or
3.1 Runtime modeling from the adjacent top PE on the principal diagonal using multiplexers
The 2𝑛𝑑 and 3𝑟𝑑 terms in the runtime equation presented by Samajdar (MUX), where the control signal is 0 for 1 cycle and 1 for the other
et al [35] remain unchanged in this orchestration while the 1𝑠𝑡 term (𝑛 − 1) cycles. With this configuration, during the 1𝑠𝑡 cycle, all the
 Md Mizanur Rahaman Nayan, Ritik Raj, Gouse Basha Shaik, Tushar Krishna, Azad J Naeemi
Department of Electrical and Computer Engineering
Georgia Institute of Technology, USA

Figure 8: a) IS/WS PE for Axon, b) Partial sums are coordinated

using bypassing at the same stage apart from the feeder PE in
the column to avoid partial sum data corruption with different
Figure 6: Runtime factor (cycles) responsible for feeding output elements. Identical arrow color represents same cycles
operand to the farthest PE. 𝑓1 (.) represents conventional sys- during the dataflow.
tolic array, 𝑓2 (.) represents Axon data orchestration. conventional design. So the only change in hardware is the addition
of an interconnect to allow bi-directional propagation of data. The
PE also integrates the zero-gating approach presented in[38] where
MAC operation is simply skipped if zero exists in either IFMAP or
FILTER operand. This helps to reduce power during sparse GEMM.
4.2 Input/Weight Stationary dataflow
Axon data orchestration is straightforward to implement for OS
dataflow. IS and WS dataflows; however, raise the two following
Figure 7: Im2col on a) FILTER of shape (3x3) and b) IFMAP challenges:
of shape (6x6). After im2col each filter is flattened like c) and 4.2.1 Preloading. From Fig.3 we observe that data is fed from the
each conv window is flattened as shown in d). (Only 4 conv buffer to the systolic array through the diagonal elements and the flow
windows have been shown related to producing 1𝑠𝑡 row of the is in both directions (top-bottom, left-right). Unlike OS, we require
output). preloading weight/input for WS/IS dataflow. The loading takes 𝑆𝑅
cycles. Loading from the buffer to the array according to the proposed
4 elements (the rightmost element from each row of IFMAP conv
architecture becomes faulty because of dataflow in both directions.
window matrix in Fig. 7(d)) will be loaded from the SRAM buffer
To solve the problem, we have utilized an output interconnect which
through feeder PEs of the array. In the 2𝑛𝑑 cycle, only the row with
is used to propagate the output vertically (Fig.8(a) yellow route).
convWin0 will load the next element from SRAM buffer while all
Two additional 2-to-1 MUXes are required to determine the target
the other three conv windows will get their 2𝑛𝑑 elements from the
buffer based on the dataflow. Note that the in-array dataflow direction
adjacent top feeder PEs via mux i.e. the feeder PE of convWin1 will
during computation remains unchanged like the OS dataflow.
get element from the feeder PE of convWin0. Similarily the feeder
PE of convWin2 will get the element from convWin1’s feeder PE
and so on. In the 3𝑟𝑑 cycle, convWin1, convWin2, convWin3 will
get their element from the top adjacent feeder PE just like 2𝑛𝑑 cycle.
The same dataflow will repeat from the 4𝑡ℎ cycle since the FILTER
length is 3). Hence, Axon can readily exploit the reuse and just load
the PEs with elements from immediate feeder PE instead of loading
from memory for (𝑛 − 1) cycles out of 𝑛 periodic cycles. Thus, we
can avoid the high memory traffic imposed by software-based im2col
with a minimal hardware overhead unlike hardware support pro- Figure 9: Axon Unified PE for OS, IS, and WS.
posed in prior works. The proposed novel data orchestration makes 4.2.2 Partial sum synchronization. Another challenge that comes
this simple scheme possible. In our approach data is loaded in an from implementing the Axon data orchestration in WS/IS is partial
ordered fashion to the feeder PEs from memory in contrast to the sum synchronization. In a conventional systolic array, the operand el-
conventional orchestration where the data is skewed. ements propagate from top to bottom and left to right and the partial
4 PE DESIGN FOR AXON sum is also calculated and propagated downward. In each stage, a par-
tial sum gets added to the previous stage’s total sum along a column
PEs are commonly designed based on the dataflow[45], and the design and finally leaves the array from the bottom row PEs as the output.
of PEs is slightly different for IS and WS than the OS. Fig. 1 depicts Thus, it ensures that the correct partial sums are added in the right
the standard design for PEs of OS and IS/WS. For Axon data orches- order. However, in the proposed architecture the operand elements
tration, the only change in the design of the PEs is in terms of data propagate in both directions; hence, the partial sums corresponding
propagation. We will discuss the design of PEs for each dataflow espe- to the same output element are generated in parallel. They must be
cially for WS/IS due to their design challenges for Axon architecture collected for accurate output generation before leaving the array. To
in the following section. address the issue we use the bypass and add approach which adds
4.1 Output Stationary dataflow two portions of the final output elements separated by the PEs on the
PE design of Axon OS dataflow is identical to the conventional design principal diagonal on that column. The PEs on the principal diagonal
as depicted in Fig.1. In Axon, PE on principle diagonal except the ones propagate operands in both directions but propagate their output
at the two corners propagate data in both directions, unlike the PE in in only one direction (either to the bottom or to the top). Fig.8(b)
Axon: A novel systolic array architecture for improved run time and energy efficient GeMM and Conv operation with on-chip im2col

depicts the bypassing for IS/WS dataflow where the dashed arrows
represent the direction of partial sums propagation and the solid
arrows indicate the direction of the operand propagation. Remember
that in IS/WS one operand (either IFMAP or FILTER operands) is
preloaded and kept stationary where the other operand is allowed
to propagate. Bypassing allows to avoid output being corrupted and
ensures output is collected without pushing stalls.
4.3 Axon Unified PE design for OS, IS/WS
dataflow Figure 10: Implemented Axon with im2col support Specifica-
tions (left). Floorplan (post PnR) of the chip.
In this section, we introduce a unified PE design as depicted in
Fig. 9 that is programmable to switch the data to any of the three for conventional SA and 0.9931𝑚𝑚 2 for Axon. The slight reduction
dataflows for the Axon data orchestration. Two Multiplexers (MUX1 in the area is due to buffer sharing between two adjacent PEs of the
and MUX2) are used to select the target buffer based on WS/IS feeder PE on the principal diagonal. The Input and weight buffers
dataflow during preloading. They forward data from the buffer to of PEs can be shared between two PEs horizontally and vertically
the input/weight buffer through the output path (yellow route). For separated by PE on principal diagonal respectively as they receive
the WS/IS dataflows, MUX3 will forward the previous partial sum the same data in the same cycle and are at the same distance from
to the adder and then forward the final sum to the output buffer to the feeder PE. After adding Im2col hardware support, the Si area
propagate to another PE’s MUX to be summed in the next cycle. For becomes increases by only 0.2% ( total area of 0.9951𝑚𝑚 2 ). The total
the OS, the black route will be active. Through MUX3 and Psum, power increases by 1.6% (59.98mW) with im2col support compared
the previous partial sum buffered in Psum will be summed with the to SA (59.88mW) which indicates that power change is insignificant
newly generated partial sum, and finally, through MUX4 the result is compared to the conventional SA.
written to the output buffer.

Workload M K N Workload M K N
TFO 31999 84 1024 NCF1 256 2048 256
TF1 84 4096 1024 DB0 1024 50000 16
GNMTO 128 4096 2048 DB1 35 2560 4096
GNMT1 2048 32 4096 Resnet50_0_conv2d 64 147 62500
GPT3_0 (matmul0) 1024 1024 80 Resnet50_1_conv2d 512 4608 676
GPT3_1 (matmul1) 1024 2560 7680 YOLO_v3_0_conv2d 64 288 42436
GPT3_2 (addmm) 1024 2560 10240 YOLO_v3_1_conv2d 128 576 10404
GPT3_3 (lmhead) 1024 2560 50257 GEMM_0 128 10 128
NCF0 2048 128 1 GEMM_1 2048 10 2048
GEMM_2 1024 1024 128
GEMM_3 64 2560 2560

Table 3: Values of M, K, and N for different workloads

5 EVALUATION
In this section, we evaluate Axon architecture against conventional
Figure 11: Memory access reduction using proposed on chip
systolic array and configurable multi-directional systolic array(CMSA)
HW support for im2col for different IFMAP and kernel shapes
[46]. CMSA adds an additional datapath to the systolic array to im-
adopted from SOTA neural networks
prove computing efficiency. We used the analytical model adopted
from SCALEsim[35] for SA runtime calculation and the analytical
5.2 Results
model from the CMSA paper. For im2col hardware support we com- 5.2.1 Comparison with SA. Fig. 12 depicts the performance compar-
pared with Sauria[15]. Sauria has a data feeder to support on-the-fly ison of the systolic array with Axon for GEMM and Convolution
im2col. TSMC 45nm PDK and ASAP 7nm PDK[11] have been used for workloads as summarized in Table. 3 for various array shapes. We
the RTL synthesis and PnR. We used Synopsys VCS and DCSHELL have normalized the runtime (cycles) w.r.t corresponding systolic ar-
for RTL design and functional verification. We evaluated the speedup ray runtime. On average, we observe Axon offers 1.47× speedup over
of Axon architecture over the baselines for GEMV, DW-Convolution, the SA for an array shape of 64 × 64. For larger arrays Axon performs
GEMM, and Convolution workloads. Workloads from Transformers even better for most workloads. However, for some workloads (e.g.
(e.g. GPT3)[2, 4], CNN(e.g. RESNET, Efficientnet, Mobilenet and YOLO NCF0 and DB0) for which the runtime is limited by the temporal
V3), Conformer and Sparse GEMM have been used[17, 19, 21, 34, 40]. dimension (e.g. NCF0 and DB0), scaling up doesn’t help to improve
Table. 3 summarizes the M, K, and N values corresponding to GEMM performance. We achieve an average of 1.76× speedup over conven-
and Conv (mapped to GEMM) workloads. For comparison, we used tional systolic array for the array shape of 256 × 256. Fig. 14 depicts
scale-up to calculate the run time for the experimental demonstra- the performance comparison on depthwise convolution(DW-Conv)
tion. The run-time improvement in scale-up due to the proposed and general matrix-vector multiplication (GEMV) workloads between
orchestration will be reflected linearly in the scale-out as well. SA and Axon. It shows that Axon is suited for low AI (Arithmetic
5.1 Hardware implementation Intensity) operations like GEMV and DW-Conv because of its bidirec-
We designed and verified Axon architecture with OS dataflow of tional data propagation. We achieve an average speedup of 1.8×. We
shape 16 × 16 that includes the proposed im2col hardware support have used the simple integrated zero gating technique presented in
and zero gating module. The OS dataflow is used because it offers [15] to lower the power consumption by leveraging the sparsity and
high IFMAP and filter reuse opportunities[14]. We used a simplified were able to achieve a 5.3% total power reduction for the case of 10%
version of the FPnew floating-point unit from the open-source parallel sparsity. Fig.11 demonstrates the capacity of the proposed on-chip
ultra-low power (PULP) platform[32] to implement FP16 MAC unit. im2col to reduce memory access for various convolution workload
Fig. 10 summarizes the specifications. We found 0.9992𝑚 2 of Si area shapes. We observe that the memory access can be reduced by more
 Md Mizanur Rahaman Nayan, Ritik Raj, Gouse Basha Shaik, Tushar Krishna, Azad J Naeemi
Department of Electrical and Computer Engineering
Georgia Institute of Technology, USA

Figure 12: Runtime improvement evaluation on GEMM and

Conv workloads Figure 13: PE utilization rate improvement over conventional
systolic array for CMSA and Axon architecutre.
than 60% by the on-chip im2col hardware for workloads generally
used in SOTA neural networks. The hardware overhead for the im2col
is < 1% found by performing PnR of a 16 × 16 array. We calculate
the energy consumption reduction due to lowering DRAM access for
Resnet50 and YOLOv3 models. Memory access (for conv layer only)
reduced from 261.2𝑀𝐵 to 153.5𝑀𝐵 for Resnet50 and from 2540𝑀𝐵 to
1117𝑀𝐵 for YOLOv3. Considering LPDDR3 memory where energy
cost is 120pJ/byte as found by [6] we find that the inference energy
is reduced by 12𝑚𝐽 for Resnet50 and 170𝑚𝐽 for YOLOv3. We consid-
ered 32-bit-wide LPDDR3 DRAM memory at 800𝑀𝐻𝑧 with a 6.4𝐺𝐵/𝑠
maximum bandwidth and found about 1.25× speedup due to lower
memory traffic enabled by im2col support which is comparable as
reported by [15]. But the proposed im2col area overhead is only 0.2%
where the feeder network of [15] is 4%.
Figure 14: Runtime improvement evaluation on DW Conv and
5.2.2 Comparison with CMSA. Besides SA, We have also compared MV workloads
PE utilization rate (UR) improvement over conventional SA for Axon
with Configurable multidirectional systolic array (CMSA) architecture
proposed by Xu et al[46]. Fig.13 illustrates the comparison where UR
has been calculated for array shape of 128 × 128. We observe that
the utilization rate improvement varies for Axon and CMSA based
on the workload shape and size. Axon outperforms CMSA by an
average of 27% in terms of the utilization rate improvement. It should
be noted that in some workloads the improvement remains small for
(a) (b)
both cases (e.g. GPT3 matmul1, GPT3 addmm, GPT3 mhead). This is
Figure 15: Power and area comparsion with sauria SA im2col.
because for those particular workloads the utilization rate is already
a) 45nm node b) 7nm node
high (average 91%) for conventional SA.
5.2.3 Energy and Area Comparison. To observe the silicon footprint aspect ratios that arise in DNNs. Our work is orthogonal and can be
and power overhead we used 45nm and 7nm advanced technology applied over them. For hardware support of im2col, SPOTS [39] sup-
node for synthesis over different array shapes as demonstrated in port sparsity with on-the-fly im2col but require a large intermediate
Fig.15a) and b) respectively. We have compared with sauria[15] as it buffer. USCA [28] and SAURIA [15] support convolution lowering
includes hardware im2col support like ours. We find that Axon has by using a dedicated data feeder. In contrast, Our work offer im2col
an average 3.93% less area and 4.5% less power consumption over through 2to1 mux with exploiting data reusability from adjacent PE
Sauria because Axon uses 2to1 mux instead of Sauria’s data feeder buffer.
registers and counters for im2col support.
7 CONCLUSION
6 RELATED WORK We present a novel data orchestration in a systolic array that out-
Samajdar et al[36] proposes a reconfigurable systolic array called performs conventional systolic array and state-of-the-art in terms
SARA to improve mapping flexibility as well as increase data reusabil- of run time in all three dataflows (OS, IS, and WS). The Axon archi-
ity through bypass wires. In MAERI[27] a fabric to support arbitrary tecture offers faster computation (upto 2× speedup) with very small
dataflows has been proposed. DRACO[22] has been proposed to opti- power overhead thus making the design more energy efficient. The
mize memory bound DNN workload from algorithm front. AI-MT[3] proposed architecture of Axon unified PE can leverage workloads’ di-
is designed to maximize the accelerator’s computational resources verse shapes to further improve the runtime. We also propose im2col
and memory bandwidth by pairing compute-intensive and memory- hardware support enabled by the Axon data orchestration. Axon
intensive tasks from different networks, and thus allowing for their im2col hardware support can relax the high memory bandwidth re-
parallel execution. COSA [43] is developed to support hybrid data quirement by 60% during convolution workloads posed by software
reuse by analyzing the computational characteristics of attention im2col with significantly lower hardware (0.2%) and power (1.6%)
mechanism. However, all these works focus on supporting diverse overhead.
Axon: A novel systolic array architecture for improved run time and energy efficient GeMM and Conv operation with on-chip im2col

REFERENCES [24] Norman P. Jouppi, Doe Hyun Yoon, Matthew Ashcraft, Mark Gottscho, Thomas B.
[1] Bahar Asgari, Ramyad Hadidi, Hyesoon Kim, and Sudhakar Yalamanchili. 2019. Jablin, George Kurian, James Laudon, Sheng Li, Peter Ma, Xiaoyu Ma, Thomas
ERIDANUS: Efficiently Running Inference of DNNs Using Systolic Arrays. IEEE Norrie, Nishant Patil, Sushma Prasad, Cliff Young, Zongwei Zhou, and David
Micro 39, 5 (2019), 46–54. [Link] Patterson. 2021. Ten Lessons From Three Generations Shaped Google’s TPUv4i
[2] Vaswani Ashish. 2017. Attention is all you need. Advances in neural information : Industrial Product. In 2021 ACM/IEEE 48th Annual International Symposium on
processing systems 30 (2017), I. Computer Architecture (ISCA). 1–14. [Link]
[3] Eunjin Baek, Dongup Kwon, and Jangwoo Kim. 2020. A Multi-Neural Network [25] Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal,
Acceleration Architecture. In 2020 ACM/IEEE 47th Annual International Symposium Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. 2017.
on Computer Architecture (ISCA). 940–953. [Link] In-datacenter performance analysis of a tensor processing unit. In Proceedings of
00081 the 44th annual international symposium on computer architecture. 1–12.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla [26] Hsiang-Tsung Kung. 1982. Why systolic architecture? Design Research Center,
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Carnegie-Mellon University.
2020. Language models are few-shot learners. Advances in neural information [27] Hyoukjun Kwon, Ananda Samajdar, and Tushar Krishna. 2018. Maeri: Enabling
processing systems 33 (2020), 1877–1901. flexible dataflow mapping over dnn accelerators via reconfigurable interconnects.
[5] Nagadastagiri Challapalle, Sahithi Rampalli, Makesh Chandran, Gurpreet Kalsi, ACM SIGPLAN Notices 53, 2 (2018), 461–475.
Sreenivas Subramoney, John Sampson, and Vijaykrishnan Narayanan. 2020. PSB- [28] Wenjian Liu, Jun Lin, and Zhongfeng Wang. 2019. USCA: A unified systolic con-
RNN: A Processing-in-Memory Systolic Array Architecture using Block Circulant volution array architecture for accelerating sparse neural network. In 2019 IEEE
Matrices for Recurrent Neural Networks. In 2020 Design, Automation & Test in Eu- International Symposium on Circuits and Systems (ISCAS). IEEE, 1–5.
rope Conference & Exhibition (DATE). 180–185. [Link] [29] Zhi-Gang Liu, Paul N. Whatmough, and Matthew Mattina. 2020. Systolic Tensor
2020.9116469 Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN Inference.
[6] Karthik Chandrasekar, Christian Weis, Yonghui Li, Sven Goossens, Matthias Jung, IEEE Computer Architecture Letters 19, 1 (2020), 34–37. [Link]
Omar Naji, Benny Akesson, Norbert Wehn, and Kees Goossens. [n. d.]. DRAMPower: 2020.2979965
Open-source DRAM Power & Energy Estimation Tool. [Link] [30] Zhi-Gang Liu, Paul N Whatmough, and Matthew Mattina. 2020. Systolic tensor
Accessed: 2024-08-17. array: An efficient structured-sparse GEMM accelerator for mobile CNN inference.
[7] Yu-Hsin Chen, Joel Emer, and Vivienne Sze. 2016. Eyeriss: A spatial architecture IEEE Computer Architecture Letters 19, 1 (2020), 34–37.
for energy-efficient dataflow for convolutional neural networks. ACM SIGARCH [31] Wenyan Lu, Guihai Yan, Jiajun Li, Shijun Gong, Yinhe Han, and Xiaowei Li. 2017.
computer architecture news 44, 3 (2016), 367–379. Flexflow: A flexible dataflow accelerator architecture for convolutional neural
[8] Yu-Hsin Chen, Tien-Ju Yang, Joel Emer, and Vivienne Sze. 2019. Eyeriss v2: A networks. In 2017 IEEE International Symposium on High Performance Computer
flexible accelerator for emerging deep neural networks on mobile devices. IEEE Architecture (HPCA). IEEE, 553–564.
Journal on Emerging and Selected Topics in Circuits and Systems 9, 2 (2019), 292–308. [32] Stefan Mach, Fabian Schuiki, Florian Zaruba, and Luca Benini. 2020. FPnew: An
[9] Hyungmin Cho. 2021. Risa: A reinforced systolic array for depthwise convolutions open-source multiformat floating-point unit architecture for energy-proportional
and embedded tensor reshaping. ACM Transactions on Embedded Computing Systems transprecision computing. IEEE Transactions on Very Large Scale Integration (VLSI)
(TECS) 20, 5s (2021), 1–20. Systems 29, 4 (2020), 774–787.
[10] Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashin- [33] A NVIDIA. 2017. NVIDIA deep learning accelerator (NVDLA).
sky. 2021. Nvidia a100 tensor core gpu: Performance and innovation. IEEE Micro [34] Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement. arXiv
41, 2 (2021), 29–35. preprint arXiv:1804.02767 (2018).
[11] Lawrence T Clark, Vinay Vashishtha, Lucian Shifren, Aditya Gujja, Saurabh Sinha, [35] Ananda Samajdar, Jan Moritz Joseph, Yuhao Zhu, Paul Whatmough, Matthew
Brian Cline, Chandarasekaran Ramamurthy, and Greg Yeric. 2016. ASAP7: A 7-nm Mattina, and Tushar Krishna. 2020. A systematic methodology for characterizing
finFET predictive process design kit. Microelectronics Journal 53 (2016), 105–115. scalability of dnn accelerators using scale-sim. In 2020 IEEE International Symposium
[12] Jason Cong and Jie Wang. 2018. PolySA: Polyhedral-based systolic array auto- on Performance Analysis of Systems and Software (ISPASS). IEEE, 58–68.
compilation. In 2018 IEEE/ACM International Conference on Computer-Aided Design [36] Ananda Samajdar, Eric Qin, Michael Pellauer, and Tushar Krishna. 2022. Self
(ICCAD). IEEE, 1–8. adaptive reconfigurable arrays (SARA) learning flexible GEMM accelerator config-
[13] Saptarsi Das, Arnab Roy, Kiran Kolar Chandrasekharan, Ankur Deshwal, and uration and mapping-space using ML. In Proceedings of the 59th ACM/IEEE Design
Sehwan Lee. 2020. A systolic dataflow based accelerator for CNNs. In 2020 IEEE Automation Conference. 583–588.
International Symposium on Circuits and Systems (ISCAS). IEEE, 1–5. [37] Surya Selvam, Vinod Ganesan, and Pratyush Kumar. 2021. FuSeConv: Fully
[14] Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. 2020. Model compression Separable Convolutions for Fast Inference on Systolic Arrays. In 2021 Design,
and hardware acceleration for neural networks: A comprehensive survey. Proc. Automation & Test in Europe Conference & Exhibition (DATE). 651–656. https:
IEEE 108, 4 (2020), 485–532. //[Link]/10.23919/DATE51398.2021.9473985
[15] Jordi Fornt, Pau Fontova-Musté, Martí Caro, Jaume Abella, Francesc Moll, Josep [38] Haikuo Shao, Jinming Lu, Meiqi Wang, and Zhongfeng Wang. 2023. An Efficient
Altet, and Christoph Studer. 2023. An energy-efficient gemm-based convolution Training Accelerator for Transformers With Hardware-Algorithm Co-Optimization.
accelerator with on-the-fly im2col. IEEE Transactions on Very Large Scale Integration IEEE Transactions on Very Large Scale Integration (VLSI) Systems (2023).
(VLSI) Systems (2023). [39] Mohammadreza Soltaniyeh, Richard P Martin, and Santosh Nagarakatte. 2021.
[16] Hasan Genc, Seah Kim, Alon Amid, Ameer Haj-Ali, Vighnesh Iyer, Pranav Prakash, SPOTS: An accelerator for sparse CNNs leveraging general matrix-matrix multipli-
Jerry Zhao, Daniel Grubb, Harrison Liew, Howard Mao, et al. 2021. Gemmini: cation. arXiv preprint arXiv:2107.13386 (2021).
Enabling systematic deep-learning architecture evaluation via full-stack integration. [40] Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for
In 2021 58th ACM/IEEE Design Automation Conference (DAC). IEEE, 769–774. convolutional neural networks. In International conference on machine learning.
[17] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui PMLR, 6105–6114.
Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. 2020. Con- [41] Swagath Venkataramani, Vijayalakshmi Srinivasan, Wei Wang, Sanchari Sen, Jintao
former: Convolution-augmented transformer for speech recognition. arXiv preprint Zhang, Ankur Agrawal, Monodeep Kar, Shubham Jain, Alberto Mannari, Hoang
arXiv:2005.08100 (2020). Tran, et al. 2021. RaPiD: AI accelerator for ultra-low precision training and inference.
[18] Cong Guo, Yangjie Zhou, Jingwen Leng, Yuhao Zhu, Zidong Du, Quan Chen, Chao In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture
Li, Bin Yao, and Minyi Guo. 2020. Balancing Efficiency and Flexibility for DNN (ISCA). IEEE, 153–166.
Acceleration via Temporal GPU-Systolic Array Integration. In 2020 57th ACM/IEEE [42] Bo Wang, Sheng Ma, Zhong Liu, Libo Huang, Yuan Yuan, and Yi Dai. 2022. SADD:
Design Automation Conference (DAC). 1–6. [Link] A Novel Systolic Array Accelerator with Dynamic Dataflow for Sparse GEMM in
9218732 Deep Learning. In IFIP International Conference on Network and Parallel Computing.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual Springer, 42–53.
learning for image recognition. In Proceedings of the IEEE conference on computer [43] Zhican Wang, Gang Wang, Honglan Jiang, Ningyi Xu, and Guanghui He. 2023.
vision and pattern recognition. 770–778. COSA: Co-Operative Systolic Arrays for Multi-head Attention Mechanism in Neu-
[20] Xin He, Subhankar Pal, Aporva Amarnath, Siying Feng, Dong-Hyeon Park, Austin ral Network using Hybrid Data Reuse and Fusion Methodologies. In 2023 60th
Rovinski, Haojie Ye, Yuhan Chen, Ronald Dreslinski, and Trevor Mudge. 2020. ACM/IEEE Design Automation Conference (DAC). IEEE, 1–6.
Sparse-TPU: Adapting systolic arrays for sparse matrices. In Proceedings of the 34th [44] Di Wu and Joshua San Miguel. 2022. uSystolic: Byte-Crawling Unary Systolic Array.
ACM international conference on supercomputing. 1–12. In 2022 IEEE International Symposium on High-Performance Computer Architecture
[21] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, (HPCA). 12–24. [Link]
Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Effi- [45] Rui Xu, Sheng Ma, Yang Guo, and Dongsheng Li. 2023. A Survey of Design and
cient convolutional neural networks for mobile vision applications. arXiv preprint Optimization for Systolic Array-Based DNN Accelerators. Comput. Surveys 56, 1
arXiv:1704.04861 (2017). (2023), 1–37.
[22] Nandan Kumar Jha, Shreyas Ravishankar, Sparsh Mittal, Arvind Kaushik, Dipan [46] Rui Xu, Sheng Ma, Yaohua Wang, Xinhai Chen, and Yang Guo. 2021. Configurable
Mandal, and Mahesh Chandra. 2020. DRACO: Co-Optimizing Hardware Utilization, multi-directional systolic array architecture for convolutional neural networks.
and Performance of DNNs on Systolic Accelerator. In 2020 IEEE Computer Society ACM Transactions on Architecture and Code Optimization (TACO) 18, 4 (2021), 1–24.
Annual Symposium on VLSI (ISVLSI). 574–579. [Link] [47] Rui Xu, Sheng Ma, Yaohua Wang, Yang Guo, Dongsheng Li, and Yuran Qiao. 2021.
2020.00088 Heterogeneous systolic array architecture for compact cnns hardware accelerators.
[23] Liancheng Jia, Liqiang Lu, Xuechao Wei, and Yun Liang. 2020. Generating Systolic IEEE Transactions on Parallel and Distributed Systems 33, 11 (2021), 2860–2871.
Array Accelerators With Reusable Blocks. IEEE Micro 40, 4 (2020), 85–92. https:
//[Link]/10.1109/MM.2020.2997611