.................................................................................................................................................................................................................

USING DATAFLOW TO OPTIMIZE

ENERGY EFFICIENCY OF DEEP NEURAL
NETWORK ACCELERATORS
.................................................................................................................................................................................................................

THE AUTHORS DEMONSTRATE THE KEY ROLE DATAFLOWS PLAY IN OPTIMIZING ENERGY

EFFICIENCY FOR DEEP NEURAL NETWORK (DNN) ACCELERATORS. THEY INTRODUCE BOTH A

SYSTEMATIC APPROACH TO ANALYZE THE PROBLEM AND A NEW DATAFLOW, CALLED

ROW-STATIONARY, THAT IS UP TO 2.5 TIMES MORE ENERGY EFFICIENT THAN EXISTING

Yu-Hsin Chen DATAFLOWS IN PROCESSING A STATE-OF-THE-ART DNN. THIS ARTICLE PROVIDES

Massachusetts Institute of GUIDELINES FOR FUTURE DNN ACCELERATOR DESIGNS.

Technology
Joel Emer ...... Recent breakthroughs in deep
neural networks (DNNs) are leading to an
use high parallelism to achieve high process-
ing throughput. However, this processing
industrial revolution based on AI. The super- also requires a significant amount of data
Nvidia and Massachusetts ior accuracy of DNNs, however, comes at movement: each MAC performs three reads
the cost of high computational complexity. and one write of data access. Because moving
Institute of Technology General-purpose processors no longer deliver data can consume more energy than the
sufficient processing throughput and energy computation itself,1 optimizing data move-
Vivienne Sze efficiency for DNNs. As a result, demands ment becomes key to achieving high energy
for dedicated DNN accelerators are increas- efficiency.
Massachusetts Institute of ing in order to support the rapidly growing Data movement can be optimized by
use of AI. exploiting data reuse in a multilevel storage
Technology The processing of a DNN mainly com- hierarchy. By maximizing the reuse of data in
prises multiply-and-accumulate (MAC) oper- the lower-energy-cost storage levels (such as
ations (see Figure 1). Most of these MACs are local scratchpads), thus reducing data accesses
performed in the DNN’s convolutional to the higher-energy-cost levels (such as
layers, in which multichannel filters are con- DRAM), the overall data movement energy
volved with multichannel input feature maps consumption is minimized.
(ifmaps, such as images). This generates par- In fact, DNNs present many data reuse
tial sums (psums) that are further accumu- opportunities. First, there are three types
lated into multichannel output feature maps of input data reuse: filter reuse, wherein
(ofmaps). Because the MAC operations have each filter weight is reused across multiple
few data dependencies, DNN accelerators can ifmaps; ifmap reuse, wherein each ifmap
.......................................................

12 Published by the IEEE Computer Society 

0272-1732/17/$33.00 c 2017 IEEE
pixel is reused across multiple filters; and
convolutional reuse, wherein both ifmap pix- Input feature maps
(ifmaps) Output feature maps
els and filter weights are reused due to the C (ofmaps)
sliding-window processing in convolutions. Filters M
Second, the intermediate psums are reused C
through the accumulation of ofmaps. If not H
Partial E
accumulated and reduced as soon as possi- R sums
(psums)
ble, the psums can pose additional storage 1 1 1
pressure. R H E
. . .
A design can exploit these data reuse . . .
. . .
opportunities by finding the optimal MAC
operation mapping, which determines both C C M

the temporal and spatial scheduling of the

MACs on a highly parallel architecture. R E
H
Ideally, data in the lower-cost storage levels is M
R N
reused by as many MACs as possible before N E
replacement. However, due to the limited H
amount of local storage, input data reuse
(ifmaps and filters) and psum reuse cannot Figure 1. In the processing of a deep neural network (DNN), multichannel
be fully exploited simultaneously. For exam- filters are convolved with the multichannel input feature maps, which then
ple, reusing the same input data for multiple generate the output feature maps. The processing of a DNN comprises
MACs generates psums that cannot be accu- many multiply-and-accumulate (MAC) operations.
mulated together and, as a result, consume
extra storage space. Therefore, the system
energy efficiency is maximized only when the
Because state-of-the-art DNNs come in a
mapping balances all types of data reuse in a
wide range of shapes and sizes, the corre-
multilevel storage hierarchy.
sponding optimal mappings also vary. The
The search for the mapping that maxi-
question is, can we find a dataflow that accom-
mizes system energy efficiency thus becomes
modates the mappings that optimize data
an optimization process. This optimization
movement for various DNN shapes and sizes?
must consider the following factors: the data
In this article, we explore different DNN
reuse opportunities available for a given
dataflows to answer this question in the con-
DNN shape and size (for example, the num-
text of a spatial architecture.2 In particular, we
ber of filters, number of channels, size of fil-
will present the following key contributions:3
ters, and feature map size), the energy cost of
data access at each level of the storage hier-  An analogy between DNN accelera-
archy, and the available processing parallelism tors and general-purpose processors
and storage capacity. The first factor is a func- that clearly identifies the distinct
tion of workload, whereas the second and aspects of operation of a DNN accel-
third factors are a function of the specific erator, which provides insights into
accelerator implementation. opportunities for innovation.
Because of implementation tradeoffs, pre-  A framework that quantitatively eval-
vious proposals for DNN accelerators have uates the energy consumption of dif-
made choices on the subset of mappings that ferent mappings for different DNN
can be supported. Therefore, for a specific shapes and sizes, which is an essential
DNN accelerator design, the optimal map- tool for finding the optimal mapping.
ping can be selected only from the subset of  A taxonomy that classifies existing
supported mappings instead of the entire dataflows from previous DNN accel-
mapping space. The subset of supported erator projects, which helps to under-
mappings is usually determined by a set of stand a large body of work despite
mapping rules, which also characterizes the differences in the lower-level details.
hardware implementation. Such a set of map-  A new dataflow, called Row-Stationary
ping rules defines a dataflow. (RS), which is the first dataflow to
.............................................................

MAY/JUNE 2017 13
 ..............................................................................................................................................................................................
TOP PICKS

tecture. Similar to the role of an ISA or

memory consistency model, the dataflow
Compilation Execution defines the mapping rules that the mapper
DNN shape and size must follow in order to generate hardware-
(Program) Processed compatible mappings. Later in this article,
data
we will introduce several previously pro-
Dataflow, ... posed dataflows.
(Architecture)
Other attributes of a DNN accelerator,
Mapper DNN accelerator
(Processor)
such as the storage organization, also are
(Compiler)
analogous to parts of the general-purpose
Implementation
details processor architecture, such as scratchpads or
(μArch) virtual memory. We consider these attributes
Mapping Input
(Binary) data
part of the architecture, instead of microarch-
itecture, because they may largely remain
invariant across implementations. Although,
Figure 2. An analogy between the operation of DNN accelerators (roman similar to GPUs, the distinction between
text) and that of general-purpose processors (italicized text). architecture and microarchitecture is likely to
blur for DNN accelerators.
Implementation details, such as those that
optimize data movement for superior determine access energy cost at each level of
system energy efficiency. It has also the storage hierarchy and latency between
been verified in a fabricated DNN processing elements (PEs), are analogous to
accelerator chip, Eyeriss.4 the microarchitecture of processors, because a
mapping will be valid despite changes in
We evaluate the energy efficiency of the these characteristics. However, they play a
RS dataflow and compare it to other data- vital part in determining a mapping’s energy
flows from the taxonomy. The comparison efficiency.
uses a popular state-of-the-art DNN model, The mapper’s goal is to search in the map-
AlexNet,5 with a fixed amount of hardware ping space for the mapping that best opti-
resources. Simulation results show that the mizes data movement. The size of the entire
RS dataflow is 1.4 to 2.5 times more energy mapping space is determined by the total
efficient than other dataflows in the convolu- number of MACs, which can be calculated
tional layers. It is also at least 1.3 times more from the DNN shape and size. However,
energy efficient in the fully connected layers only a subset of the space is valid given the
for batch sizes of at least 16. These results mapping rules defined by a dataflow. For
will provide guidance for future DNN accel- example, the dataflow can enforce the follow-
erator designs. ing mapping rule: all MACs that use the
same filter weight must be mapped on the
An Analogy to General-Purpose Processors same PE in the accelerator. Then, it is the
Figure 2 shows an analogy between the oper- mapper’s job to find out the exact ordering of
ation of DNN accelerators and general- these MACs on each PE by evaluating and
purpose processors. In conventional computer comparing the energy efficiency of different
systems, the compiler translates a program valid ordering options.
into machine-readable binary codes for exe- As in conventional compilers, performing
cution; in the processing of DNNs, the map- evaluation is an integral part of the mapper.
per translates the DNN shape and size into a The evaluation process takes a certain map-
hardware-compatible mapping for execu- ping as input and gives an energy consump-
tion. While the compiler usually optimizes tion estimation based on the available
for performance, the mapper especially opti- hardware resources (microarchitecture) and
mizes for energy efficiency. data reuse opportunities extracted from the
The dataflow is a key attribute of a DNN DNN shape and size (program). In the next
accelerator and is analogous to one of the section, we will introduce a framework that
parts of a general-purpose processor’s archi- can perform this evaluation.
............................................................

14 IEEE MICRO
Evaluating Energy Consumption
Finding the optimal mapping requires evalu- CPU iFIFO/oFIFO PE array

Accelerator chip
ation of the energy consumption for various GPU
mapping options. In this article, we evaluate Off-chip
energy consumption based on a spatial archi- DRAM
tecture,2 because many of the previous Global
buffer
designs can be thought of as instances of such
an architecture. The spatial architecture (see
Figure 3) consists of an array of PEs and a iFIFO/oFIFO PE array
multilevel storage hierarchy. The PE array (zoom in)
provides high parallelism for high through-
put, whereas the storage hierarchy can be RF RF RF

pFIFO

pFIFO

pFIFO
used to exploit data reuse in a four-level setup
(in decreasing energy-cost order): DRAM,
global buffer, network-on-chip (NoC, for
inter-PE communication), and register file
(RF) in the PE as local scratchpads.
Figure 3. Spatial array architecture comprises an array of processing
In this architecture, we assume all data
elements (PEs) and a multilevel storage hierarchy, including the off-chip
types can be stored and accessed at any level
DRAM, global buffer, network-on-chip (NoC), and register file (RF) in the PE.
of the storage hierarchy. Input data for the
The off-chip DRAM, global buffer, and PEs in the array can communicate
MAC operations—that is, filter weights and
with each other directly through the input and output FIFOs (the iFIFO and
ifmap pixels—are moved from the most
oFIFO). Within each PE, the PE FIFO (pFIFO) controls the traffic going in and
expensive level (DRAM) to the lower-cost
out of the arithmetic logic unit (ALU), including from the RF or other storage
levels. Ultimately, they are usually delivered
levels.
from the least expensive level (RF) to the
arithmetic logic unit (ALU) for computation.
The results from the ALU—that is, psums—
generally move in the opposite direction. Normalized energy cost
The orchestration of this movement is deter-
mined by the mappings for a specific DNN Computation 1 MAC at ALU 1× (Reference)
shape and size under the mapping rule con- RF (0.5 to 1.0 Kbytes) 1×
straints of a specific dataflow architecture.
Given a specific mapping, the system NoC (1 to 2 mm) 2×
Data access
energy consumption is estimated by account- Global buffer
6×
(>100 Kbytes)
ing for the number of times each data value
DRAM 200×
from all data types (ifmaps, filters, psums) is
reused at each level of the four-level memory
hierarchy, and weighing it with the energy Figure 4. Normalized energy cost relative to the computation of one MAC
cost of accessing that specific storage level. operation at ALU. Numbers are extracted from a commercial 65-nm
Figure 4 shows the normalized energy con- process.
sumption of accessing data from each storage
level relative to the computation of a MAC at each PE can hold only one ifmap pixel at a
the ALU. We extracted these numbers from a time. The mapping first reads an ifmap pixel
commercial 65-nm process and used them in from DRAM to the global buffer, then from
our final experiments. the global buffer to the RF of each PE
Figure 5 uses a toy example to show how a through the NoC, and reuses it from the RF
mapping determines the data reuse at each for four MACs consecutively in each PE. The
storage level, and thus the energy consump- mapping then switches to MACs that use
tion, in a three-PE setup. In this example, we other ifmap pixels, so the original one in the
have the following assumptions: each ifmap RF is replaced by new ones, due to limited
pixel is used by 24 MACs, all ifmap pixels capacity. Therefore, the original ifmap pixel
can fit into the global buffer, and the RF of must be fetched from the global buffer again
.............................................................

MAY/JUNE 2017 15
 ..............................................................................................................................................................................................
TOP PICKS

valid mapping space based on how they han-

dle data. As a result, we can classify them into
iFIFO/oFIFO PE array a taxonomy.

RF RF RF  The Weight-Stationary (WS) data-

DRAM

pFIFO

pFIFO

pFIFO
Global flow keeps filter weights stationary in
buffer each PE’s RF by enforcing the follow-
ing mapping rule: all MACs that use
Memory level
Buffer level the same filter weight must be
NoC level mapped on the same PE for process-
ing serially. This maximizes the con-
RF level volutional and filter reuse of weights
Ifmap pixel in the RF, thus minimizing the
data movement Processing other data . . .
energy consumption of accessing
weights (for example, work by Srimat
Chakradhar and colleagues6 and
time
Vinayak Gokhale and colleagues7).
Figure 6a shows the data movement
of a common WS dataflow imple-
Figure 5. Example of how a mapping determines data reuse at each storage mentation. While each weight stays
level. This example shows the data movement of one ifmap pixel going in the RF of each PE, the ifmap pixels
through the storage hierarchy. Each arrow means moving data between are broadcast to all PEs, and the gen-
specific levels (or to an ALU for computation). erated psums are then accumulated
spatially across PEs.
 The Output-Stationary (OS) data-
when the mapping switches back to the
MACs that use it. In this case, the same flow keeps psums stationary by accu-
ifmap pixel is reused at the DRAM, global mulating them locally in the RF. The
buffer, NoC, and RF for 1, 2, 6, and 24 mapping rule is that all MACs that
times, respectively. The corresponding nor- generate psums for the same ofmap
malized energy consumption of moving this pixel must be mapped on the same
ifmap pixel is obtained by weighing these PE serially. This maximizes psum
numbers with the normalized energy num- reuse in the RF, thus minimizing
bers in Figure 4 and then adding them energy consumption of psum move-
together (that is, 1  200 þ 2  6 þ 6  2 ment (for example, work by Zidong
þ 24  1 ¼ 248). For other data types, the Du and colleagues,8 Suyog Gupta
same approach can be applied. and colleagues,9 and Maurice Pee-
This analysis framework can be used not men and colleagues10). The data
only to find the optimal mapping for a spe- movement of a common OS dataflow
cific dataflow, but also to evaluate and com- implementation is to broadcast filter
pare the energy consumption of different weights while passing ifmap pixels
dataflows. In the next section, we will spatially across the PE array (see
describe various existing dataflows. Figure 6b).
 Unlike the previous two dataflows,
which keep a certain data type sta-
A Taxonomy of Existing DNN Dataflows tionary, the No-Local-Reuse (NLR)
Numerous previous efforts have proposed dataflow keeps no data stationary
solutions for DNN acceleration. These locally so it can trade the RF off for a
designs reflect a variety of trade-offs between larger global buffer. This is to mini-
performance and implementation complex- mize DRAM access energy consump-
ity. Despite their differences in low-level tion by storing more data on-chip
implementation details, we find that many of (for example, work by Tianshi Chen
them can be described as embodying a set of and colleagues11 and Chen Zhang
rules—that is, a dataflow—that defines the and colleagues12). The corresponding
............................................................

16 IEEE MICRO
 mapping rule is that at each process-
ing cycle, all parallel MACs must Ifmap pixel (l) Filter weight (W) Psum (P)
come from a unique pair of filter and Weight-Stationary (WS) dataflow
channel. The data movement of the
Global buffer
NLR dataflow is to single-cast weights,
P8 I8 P0
multicast ifmap pixels, and spatially
accumulate psums across the PE array
(see Figure 6c). W0 P7 W1 P6 W2 P5 W3 P4 W4 P3 W5 P2 W6 P1 W7 PE

The three dataflows show distinct data (a)

movement patterns, which imply different Output-Stationary (OS) dataflow
tradeoffs. First, as Figures 6a and 6b show, Global buffer
the cost for keeping a specific data type sta- I7 W7
tionary is to move the other types of data
more. Second, the timing of data accesses
P0 P1 P2 P3 P4 P5 P6 P7 PE
also matters. For example, in the WS data- I6 I5 I4 I3 I2 I1 I0
flow, each ifmap pixel read from the global (b)
buffer is broadcast to all PEs with properly No-Local-Reuse (NLR) dataflow
mapped MACs on the PE array. This is more
Global buffer
efficient than reading the same value multiple
times from the global buffer and single-cast- W0 P8 I0 W1P9 W2 I1 W3 W4 I2 W5 W6P0 I3 W7P1

ing it to the PEs, which is the case for filter P6 P4 P2

weights in the NLR dataflow (see Figure 6c). PE
P7 P5 P3
Other dataflows can make other tradeoffs. In
(c)
the next section, we present a new dataflow
that takes these factors into account for opti-
mizing energy efficiency. Figure 6. Dataflow taxonomy. (a) Weight Stationary. (b) Output Stationary.
(c) No Local Reuse.

An Energy-Efficient Dataflow  The ordering of these MACs enables

Although the dataflows in the taxonomy
the use of a sliding window for ifmaps,
describe the design of many DNN accelera-
as shown in Figure 7.
tors, they optimize data movement only for a
specific data type (for example, WS for Convolutional and psum reuse opportu-
weights) or storage level (NLR for DRAM). nities within a row primitive are fully
In this section, we introduce a new dataflow, exploited in the RF, given sufficient RF stor-
called Row-Stationary (RS), which aims to age capacity.
optimize data movement for all data types in Even with the RS dataflow, as defined by
all levels of the storage hierarchy of a spatial the row primitives, there are still a large num-
architecture. ber of valid mapping choices. These mapping
The RS dataflow divides the MACs into choices arise both in the spatial and temporal
mapping primitives, each of which comprises assignment of primitives to PEs:
a subset of MACs that run on the same PE in
a fixed order. Specifically, each mapping 1. One spatial mapping option is to
primitive performs a 1D row convolution, so assign primitives with data rows
we call it a row primitive, and intrinsically from the same 2D plane on the PE
optimizes data reuse per MAC for all data array, to lay out a 2D convolution
types combined. Each row primitive is (see Figure 8). This mapping fully
formed with the following rules: exploits convolutional and psum
reuse opportunities across primitives
 The MACs for applying a row of fil- in the NoC: the same rows of filter
ter weights on a row of ifmap pixels, weights and ifmap pixels are reused
which generate a row of psums, must across PEs horizontally and diago-
be mapped on the same PE. nally, respectively; psum rows are
.............................................................

MAY/JUNE 2017 17
 ..............................................................................................................................................................................................
TOP PICKS

becomes a larger 1D row convolution,

Filter row Ifmap row Psum row which exploits these cross-primitive
A B C ∗ a b c d e = x y z data reuse opportunities in the RF.
4. Another temporal mapping choice
Time
arises when the PE array size is too
MAC1 MAC2 MAC3 MAC4 MAC5 MAC6 MAC7 MAC8 MAC9
small, and the originally spatially
Filter weight: A B C A B C A B C
mapped row primitives must be tem-
x + x + x x + x + x x + x + x
Ifmap pixel: a b c a b c a b c
porally folded into multiple process-
ing passes (that is, the computation is
II II II
Psum:
serialized). In this case, the data reuse
x y z
opportunities that are originally spa-
tially exploited in the NoC can be
Figure 7. Each row primitive in the Row-Stationary (RS) dataflow runs a 1D temporally exploited by the global
row convolution on the same PE in a sliding-window processing order. buffer to avoid DRAM accesses,
given sufficient storage capacity.
As evident from the preceding list, the RS
dataflow provides a high degree of mapping
Row 1 Row 2 Row 3
flexibility, such as using concatenation, inter-
PE1 PE4 PE7
leaving, duplicating, and folding of the row
Row 1 ∗ Row 1 Row 1 ∗ Row 2 Row 1 ∗ Row 3
primitives. The mapper searches for the exact
PE2 PE5 PE8
amount to apply each technique in the opti-
Row 2 ∗ Row 2 Row 2 ∗ Row 3 Row 2 ∗ Row 4 mal mapping—for example, how many fil-
PE3 PE6 PE9 ters are interleaved on the same PE to exploit
Row 3 ∗ Row 3 Row 3 ∗ Row 4 Row 3 ∗ Row 5 ifmap reuse—to minimize overall system
energy consumption.
∗ = ∗ = ∗ =

Dataflow Comparison
Figure 8. Patterns of how row primitives from the same 2D plane are In this section, we quantitatively compare the
mapped onto the PE array in the RS dataflow. energy efficiency of different DNN dataflows
in a spatial architecture, including those from
the taxonomy and the proposed RS dataflow.
further accumulated across PEs We use AlexNet5 as the benchmarking DNN
vertically. because it is one of the most popular DNNs
2. Another spatial mapping option available, and it comprises five convolutional
arises when the size of the PE array is (CONV) layers and three fully connected
large, and the pattern shown in (FC) layers with a wide variety of shapes and
Figure 8 can be spatially duplicated sizes, which can more thoroughly evaluate
across the PE array for various 2D the optimal mappings from each dataflow.
convolutions. This not only increases In order to have a fair comparison, we
utilization of PEs, but also further apply the following two constraints for all
exploits filter, ifmap, and psum reuse dataflows. First, the size of the PE array is
opportunities in the NoC. fixed at 256 for constant processing through-
3. One temporal mapping option arises put across dataflows. Second, the total hard-
when row primitives from different ware area is also fixed. For example, because
2D planes can be concatenated or the NLR dataflow does not use an RF, it can
interleaved on the same PE. As Figure allocate more area for the global buffer. The
9 shows, primitives with different corresponding hardware resource parameters
ifmaps, filters, and channels have filter are based on the RS dataflow implementation
reuse, ifmap reuse, and psum reuse in Eyeriss, a DNN accelerator chip fabricated
opportunities, respectively. By concat- in 65-nm CMOS.4 By applying these con-
enating or interleaving their computa- straints, we fix the total cost to implement
tion together in a PE, it virtually the microarchitecture of each dataflow.
............................................................

18 IEEE MICRO
 Filter 1 Ifmap 1 Psum 1
Channel 1 Row 1 ∗ Row 1 = Row 1 Filter 1 Ifmap 1 and 2 Psum 1 and 2

Filter 1 Ifmap 2 Psum 2 Row 1 ∗ Row 1 Row 1 = Row 1 Row 1

Channel 1 Row 1 ∗ Row 1 = Row 1
filter reuse
(a)

Filter 1 Ifmap 1 Psum 1

Channel 1 Row 1 ∗ Row 1 = Row 1 Filter 1 and 2 Ifmap 1 Psum 1 and 2
Filter 2 Ifmap 1 Psum 2 ∗ =
Channel 1 Row 1 ∗ Row 1 = Row 1
Ifmap reuse
(b)

Filter 1 Ifmap 1 Psum 1

Channel 1 Row 1 ∗ Row 1 = Row 1 Filter 1 Ifmap 1 Psum
Filter 1 Ifmap 1 Psum 1 ∗ = Row 1
Channel 2 Row 1 ∗ Row 1 = Row 1

psum reuse (can be further accumulated)

(c)

Figure 9. Row primitives from different 2D planes can be combined by concatenating or interleaving their computation on the
same PE to further exploit data reuse at the RF level. (a) Two-row primitives reuse the same filter row for different ifmap
rows. (b) Two-row primitives reuse the same ifmap row for different filter rows. (c) Two-row primitives from different
channels further accumulate psum rows.

Therefore, the differences in energy efficiency DRAM alone does not dictate energy effi-
are solely from the dataflows. ciency, and optimizing the energy consump-
Figures 10a and 10b show the comparison tion for only a certain data type does not lead
of energy efficiency between dataflows in the to the best system energy efficiency. Overall,
CONV layers of AlexNet with an ifmap batch the RS dataflow is 1.4 to 2.5 times more
size of 16. Figure 10a gives the breakdown in energy efficient than other dataflows in the
terms of storage levels and ALU, and Figure CONV layers of AlexNet.
10b gives the breakdown in terms of data Figure 11 shows the same experiment
types. First, the ALU energy consumption is results as in Figure 10b, except that it is for
only a small fraction of the total energy con- the FC layers of AlexNet. Compared to the
sumption, which proves the importance of CONV layers, the FC layers have no convo-
data movement optimization. Second, even lutional reuse and use much more filter
though NLR has the lowest energy consump- weights. Still, the RS dataflow is at least 1.3
tion in DRAM, its total energy consumption times more energy efficient than the other
is still high, because most of the data accesses dataflows, which proves that the capability to
come from the global buffer, which are more optimize data movement for all data types is
expensive than those from the NoC or RF. the key to achieving the highest overall
Third, although WS and OS dataflows clearly energy efficiency. Note that the FC layers
optimize the energy consumption of accessing account for less than 20 percent of the total
weights and psums, respectively, they sacrifice energy consumption in AlexNet. In recent
the energy consumption of moving other data DNNs, the number of FC layers have also
types, and therefore do not achieve the lowest been greatly reduced, making their energy
overall energy consumption. This shows that consumption even less significant.
.............................................................

MAY/JUNE 2017 19
 ..............................................................................................................................................................................................
TOP PICKS

2.0 2.0
Normalized energy/MAC

Normalized energy/MAC
1.5 RF 1.5
NoC psums
1.0 Buffer 1.0 weights

DRAM pixels
0.5 ALU 0.5

0 0
WS OSA OSB OSC NLR RS WS OSA OSB OSC NLR RS
(a) DNN dataflows (b) DNN dataflows

Figure 10. Comparison of energy efficiency between different dataflows in the convolutional (CONV) layers of AlexNet.5 (a)
Breakdown in terms of storage levels and ALU versus (b) data types. OSA, OSB, and OSC are three variants of the OS dataflow
that are commonly seen in different implementations.3

tinct architectures, it is also possible to come

up with a union architecture that can support
multiple dataflows simultaneously. The ques-
Normalized energy/MAC

2.0
tions are how to choose a combination of
1.5
psums
dataflows that maximally benefit the search
weights
for optimal mappings, and how to support
1.0 these dataflows with the minimum amount of
pixels
hardware implementation overhead.
0.5
This article has also pointed out how the
0
concept of DNN dataflows and the mapping
WS OSA OSB OSC NLR RS of a DNN computation onto a dataflow can be
DNN dataflows viewed as analogous to a general-purpose pro-
cessor’s architecture and compiling onto that
Figure 11. Comparison of energy efficiency between different dataflows in architecture. We hope this will open up space
the fully connected (FC) layers of AlexNet. for computer architects to approach the design
of DNN accelerators by applying the knowl-
edge and techniques from a well-established

R esearch on architectures for DNN

accelerators has become very popular
for its promising performance and wide
research field in a more systematic manner,
such as methodologies for design abstraction,
modularization, and performance evaluation.
applicability. This article has demonstrated For instance, a recent research trend for
the key role of dataflows in DNN accelerator DNNs is to exploit data statistics. Specifically,
design, and it shows how to systematically different proposals on quantization, pruning,
exploit all types of data reuse in a multilevel and data representation have all shown prom-
storage hierarchy for optimizing energy effi- ising results on improving the performance of
ciency with a new dataflow. It challenges con- DNNs. Therefore, it is important that new
ventional design approaches, which focus architectures also take advantage of these find-
more on optimizing parts of the problem, ings. As compilers for general-purpose pro-
and shifts it toward a global optimization cessors can take the profile of targeted
that considers all relevant metrics. workloads to further improve the performance
The taxonomy of dataflows lets us compare of the generated binary, the analogy between
high-level design choices irrespective of low- general-purpose processors and DNN acceler-
level implementation details, and thus can be ators suggests that the mapper for DNN accel-
used to guide future designs. Although these erators might also take the profile of targeted
dataflows are currently implemented on dis- DNN statistics to further optimize the
............................................................

20 IEEE MICRO
generated mappings. This is an endeavor we 11. T. Chen et al., “DianNao: A Small-Footprint
will leave for future work. MICRO High-Throughput Accelerator for Ubiquitous
Machine-Learning,” Proc. 19th Int’l Conf.
.................................................................... Architectural Support for Programming Lan-
References guages and Operating Systems (ASPLOS
1. M. Horowitz, “Computing’s Energy Prob- 14), 2014, pp. 269–284.
lem (And What We Can Do About It),” Proc. 12. C. Zhang et al., “Optimizing FPGA-based
IEEE Int’l Solid-State Circuits Conf. (ISSCC Accelerator Design for Deep Convolutional
14), 2014, pp. 10–14. Neural Networks,” Proc. ACM/SIGDA Int’l
2. A. Parashar et al., “Triggered Instructions: A Symp. Field-Programmable Gate Arrays
Control Paradigm for Spatially-Programmed (FPGA 15), 2015, pp. 161–170.
Architectures,” Proc. 40th Ann. Int’l Symp.
Computer Architecture (ISCA 13), 2013, pp. Yu-Hsin Chen is a PhD student in the
142–153. Department of Electrical Engineering and
3. Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A Computer Science at the Massachusetts
Spatial Architecture for Energy-Efficient Data- Institute of Technology. His research inter-
flow for Convolutional Neural Networks,” Proc. ests include energy-efficient multimedia sys-
ACM/IEEE 43rd Ann. Int’l Symp. Computer tems, deep learning architectures, and com-
Architecture (ISCA 16), 2016, pp. 367–379. puter vision. Chen received an MS in
electrical engineering and computer science
4. Y.-H. Chen et al., “Eyeriss: An Energy-
from the Massachusetts Institute of Tech-
Efficient Reconfigurable Accelerator for
nology. He is a student member of IEEE.
Deep Convolutional Neural Networks,”
Contact him at yhchen@[Link].
Proc. IEEE Int’l Solid-States Circuits Conf.
(ISSCC 16), 2016, pp. 262–263. Joel Emer is a senior distinguished research
5. A. Krizhevsky, I. Sutskever, and G.E. Hinton, scientist at Nvidia and a professor of electrical
“ImageNet Classification with Deep Convo- engineering and computer science at the Mas-
lutional Neural Networks,” Proc. 25th Int’l sachusetts Institute of Technology. His research
Conf. Neural Information Processing Sys- interests include spatial and parallel architec-
tems (NIPS 12), 2012, pp. 1097–1105. tures, performance modeling, reliability analy-
6. S. Chakradhar et al., “A Dynamically Config-
sis, and memory hierarchies. Emer received a
urable Coprocessor for Convolutional Neural
PhD in electrical engineering from the Uni-
Networks,” Proc. 37th Ann. Int’l Symp.
versity of Illinois. He is a Fellow of IEEE.
Computer Architecture (ISCA 10), 2010, pp.
Contact him at jsemer@[Link].
247–257.
Vivienne Sze is an assistant professor in the
7. V. Gokhale et al., “A 240 G-ops/s Mobile Department of Electrical Engineering and
Coprocessor for Deep Neural Networks,” Computer Science at the Massachusetts
Proc. IEEE Conf. Computer Vision and Pat- Institute of Technology. Her research inter-
tern Recognition Workshops (CVPRW 14), ests include energy-aware signal processing
2014, pp. 696–701. algorithms and low-power architecture and
8. Z. Du et al., “ShiDianNao: Shifting Vision system design for multimedia applications,
Processing Closer to the Sensor,” Proc. such as machine learning, computer vision,
ACM/IEEE 42nd Ann. Int’l Symp. Computer and video coding. Sze received a PhD in elec-
Architecture (ISCA 15), 2015, pp. 92–104. trical engineering from the Massachusetts
9. S. Gupta et al., “Deep Learning with Limited
Institute of Technology. She is a senior mem-
Numerical Precision,” Proc. 32nd Int’l Conf.
ber of IEEE. Contact her at sze@[Link].
Machine Learning, vol. 37, 2015, pp.
1737–1746.
10. M. Peemen et al., “Memory-Centric Accelera-
tor Design for Convolutional Neural Networks,” Read your subscriptions through
Proc. IEEE 31st Int’l Conf. Computer Design the myCS publications portal at
[Link]
(ICCD 13), 2013, pp. 13–19.
.............................................................

MAY/JUNE 2017 21