Timeloop: A Systematic Approach to DNN

Accelerator Evaluation
Angshuman Parashar∗ Priyanka Raina∗‡ Yakun Sophia Shao∗ Yu-Hsin Chen∗
Victor A. Ying† Anurag Mukkara † Rangharajan Venkatesan∗ Brucek Khailany∗
Stephen W. Keckler∗ Joel Emer∗†
∗ NVIDIA † Massachusetts Institute of Technology ‡ Stanford University

Abstract—This paper presents Timeloop, an infrastructure for different ways of scheduling operations and staging data on
evaluating and exploring the architecture design space of deep the same architecture, which we call different mappings [7],
neural network (DNN) accelerators. Timeloop uses a concise and resulting in widely varying performance and energy efficiency.
unified representation of the key architecture and implementation
attributes of DNN accelerators to describe a broad space of This results in a requirement to search for a good mapping
hardware topologies. It can then emulate those topologies to within the space of valid mappings, i.e., the mapspace, to fairly
generate an accurate projection of performance and energy characterize the architecture. The need to do this search for
efficiency for a DNN workload through a mapper that finds the each workload compounds the complexity of exploring the
best way to schedule operations and stage data on the specified architecture design space. Prior work on DNN accelerators
architecture. This enables fair comparisons across different archi-
tectures and makes DNN accelerator design more systematic. This have dealt with this problem only within the scope of their
paper describes Timeloop’s underlying models and algorithms in specific designs [6], [13], [22], [26], [27], [32], [35]–[37].
detail and shows results from case studies enabled by Timeloop, Tools for broader design space exploration [19], [27], [34],
which provide interesting insights into the current state of DNN [38] also struggle with challenges such as separation of the
architecture design. In particular, they reveal that dataflow and architecture design space from the mapspace, enumeration of
memory hierarchy co-design plays a critical role in optimizing
energy efficiency. Also, there is currently still not a single mappings in the mapspace and quality of performance and
architecture that achieves the best performance and energy energy efficiency evaluation, making it difficult to explore and
efficiency across a diverse set of workloads due to flexibility compare a wide range of architectures.
and efficiency trade-offs. These results provide inspiration into To address these challenges, we propose Timeloop, an
possible directions for DNN accelerator research.
Index Terms—modeling; accelerator architecture; deep neural infrastructure for evaluating and exploring the architecture
networks; neural network dataflows design space of DNN accelerators. Timeloop has two main
components: a model to provide performance, area and energy
I. I NTRODUCTION projections and a mapper to construct and search through the
mapspace of any given workload on the targeted architecture.
Deep neural networks (DNNs) have emerged as a key To perform an analysis using Timeloop, a user describes an
approach for solving complex problems across a wide range architecture’s organization in a highly configurable template
of domains, including image recognition [21], speech pro- with abstractions for compute units (with configurable preci-
cessing [3], [14], [16], natural language processing [9], lan- sion), memories (in a configurable multilevel hierarchy), and
guage translation [11], and autonomous navigation [23]. To communication links. Architectural constraints, including the
provide high performance and energy efficiency to this class of dataflow and additional hardware attributes such as utilization
compute-intensive algorithms, many DNN hardware accelera- and bandwidth limitations, can be further imposed to define
tors have been proposed [1], [2], [5], [6], [12], [13], [15], [18], the mapspace. Given a workload and an architecture specifi-
[25], [29], [33], [35], [39]. While they share similar hardware cation (i.e., organization and constraints), Timeloop’s mapper
constructs such as a specialized memory hierarchy [30] and an systematically constructs the mapspace, evaluates the quality
array of multiply-accumulate units (MACs), they differ widely of each mapping in the mapspace with its embedded cost
in their underlying dataflows, which define how operations are model, and searches through the mapspace iteratively for an
allowed to be partitioned and scheduled for computation, and optimal mapping. This search is feasible thanks to the model’s
how data can be staged in the memory hierarchy [7]. Dataflows speed (by exploiting the regular structure of DNN workloads)
have been shown to have a significant impact on performance and accuracy (for energy and performance), which has been
and energy efficiency due to the resulting differences in data validated against existing designs.
reuse and hardware utilization [6], [19], [22], [25], [31].
Timeloop makes the following key contributions:
The vast number of possible dataflows and the accom-
panying choices of hardware implementation have created a 1) Timeloop provides a concise and unified way of de-
large architecture design space. In addition, for a given DNN scribing the key attributes of a diverse class of DNN
layer, i.e., a workload, a flexible architecture may permit many architectures and their implementation features as the
 input to a fast and accurate analytical model. This allows
16000
for the exploration of a wide range of architectures.
14000

Number of mappings
2) Timeloop is the first infrastructure to effectively combine
12000
the exploration of a large design space with a mapper
10000
that allows for finding the optimal mapping of any
8000
workload on the targeted architecture. This enables fair
6000
comparisons between different architectures and makes 4000
the DNN accelerator design more systematic and less of 2000
an art. 0
3) Timeloop enables a number of studies that provide inter- 20 40 60 80
Energy/MAC (relative to multiplier energy)
esting insights into the current state of DNN accelerator
Fig. 1. Histogram of the energy efficiency of various mappings of
architecture design. These studies point out the pros VGG conv3 2 on an example architecture.
and cons of various representative architectures in the
literature and provide inspiration for future research into Section V how Timeloop systematically constructs a mapspace
the design of DNN accelerators. and walks through the mappings in it.
We expect to release Timeloop under an open-source license A mapper needs a model. A mapper enumerates the
and welcome contributions from the community. mapspace and then iteratively searches through the space for
the optimal mapping. Thus, it needs a cost function to deter-
II. M OTIVATION mine whether one mapping is better than another. This cost
function must be fast enough to evaluate to allow the mapper to
The architectural design of DNN accelerators tends to be explore a large mapspace in a reasonable amount of time, but
more of an art than science due to the large design space and it must also be accurate, i.e., it must be representative of the
the lack of a systematic approach to explore it. To explore actual behavior of the mapping on the architecture, otherwise
such a design space, we have observed a unique symbiosis the mapper will give misleading results. For example, while
between the need for a model that can describe and emulate DRAM accesses consume a lot of energy, accesses to on-
a wide variety of architectures and the need for a mapper that chip buffers are often just as expensive. In Figure 1, 6,582
can optimally configure the accelerator to get a fair character- mappings have the same minimum DRAM accesses, but vary
ization of the performance and energy efficiency of a specific 11× in energy efficiency. Even though the actual optimal
workload. In this section, we will describe the symbiosis that mapping in this example does have the minimum number of
Timeloop exploits to make it an effective infrastructure for DRAM accesses, it is not guaranteed to be this case across all
exploring the design space of DNN accelerators. architectures and technologies. As another example, we show
A model needs a mapper. Unlike traditional architectures in Section VIII-B that cost ratio changes due to technology
that use an ISA as the hardware/software interface, each DNN trends lead to different optimal mappings, and that the optimal
accelerator uniquely exposes many configurable settings in the mapping in one technology may be sub-optimal in another.
hardware, and its behavior is critically dependent on those
settings. Therefore, instead of using pre-compiled binaries or III. P RIOR WORK
traces, running a DNN workload on an accelerator involves There is an abundance of excellent recent work on DNN
finding a mapping (and corresponding set of configuration accelerators [6], [13], [22], [26], [27], [32], [35]–[37]. Across
settings), which dictates the scheduling of operations and data these works, a large component of the mapping is fixed in
movement, to maximize performance and energy efficiency. hardware with a limited set of loop unrolling (parallelism),
To obtain a fair characterization of the hardware, it is impor- loop tiling and ordering (data reuse) choices exposed to a
tant to select a good mapping. Figure 1 shows a distribution of search algorithm. Thus, they do provide mappers, but the
the energy efficiency of various mappings for the convolutional respective mapspace as well as architectural model are closely
layer VGG conv3 2 on a 1024-MAC architecture similar to tied to the specific attributes of each design and, therefore, are
NVDLA [28]. The 480k mappings shown here are all within unsuitable for broader design space exploration or competitive
5% of the peak performance, but vary nearly 19× in energy assessment.
efficiency. Only one mapping is energy-optimal and 9 others A general framework for studying a broad range of designs
are within 1% of the optimal mapping. The variations arise due must be able to describe the organization of each design,
to different tiling and scheduling alternatives represented by automatically generate the complete mapspace for any de-
the mappings, with the more efficient mappings making better scribed architecture, and find efficient mappings within the
use of buffer capacity, network multicast and loop ordering to space. Otherwise, a comparison between competing architec-
maximize reuse. However, the optimal mapping can and will tures is meaningless. A few recent proposals have attempted
likely change across workloads; an optimal mapping for one this challenging task [19], [27], [34], [38]. Unfortunately, we
architecture may also be a poor (or even invalid) choice on believe some of them suffer from inadequate architectural cost
another architecture. Therefore, mapping is an essential step models [31], [34], [38], with simplistic hardware templates
in evaluating a DNN workload on an architecture. We show in and/or inability to infer different forms of spatial or temporal
 1 for r=[0:R):
2 for s=[0:S):
3 for p=[0:P):
4 for q=[0:Q):
5 for c=[0:C):
6 for k=[0:K):
7 for n=[0:N):
8 Output[p][q][k][n] +=
9 Weight[r][s][k][c] * Input[p+r][q+s][c][n];

Fig. 3. Convolutional layer 7D loop nest.

Fig. 2. Timeloop tool-flow. deterministic, enabling it to compute throughput and access

counts analytically for an architecture. The model determines
reuse. Others fail to comprehend the complete size and scope the counts of various activities, including arithmetic opera-
of the mapspace and/or obfuscate the separation between the tions, memory accesses and network transfers, including the
architecture design space and mapspace [19], [27]. Finally, re- multi-casting of data from a producer to multiple consumers,
cent work on sophisticated compilers for DNN workloads [4], and forwarding of data between units. These access counts
[31] have built-in cost models, though the objective of those are used to determine performance. Combined with the energy
models is simply to provide enough fidelity to guide the opti- cost per access from the energy model, they are also used to
mizer rather than to provide accurate energy and performance determine the energy consumption of the workload.
projections for architecture design-space exploration. Figure 2 shows an overview of Timeloop. The infrastructure
Timeloop provides a rich and general template that is capa- is split into two main components, the mapper and the model,
ble of representing DNN accelerators in a large design space, with a mapping serving as the interface between the two.
and it automatically constructs the complete mapspace for any We describe the mapper in Section V along with the input
specified architecture. We show that popular dataflows such as specifications used to configure Timeloop to represent different
output-stationary or weight-stationary [6], [8] are but specific architectures and workloads. In Section VI, we describe how
instances of a larger set of constraints that can be imposed on the model evaluates each mapping.
the mapspace, limiting the computation schedules and operand
reuse patterns that the architecture can exploit. For each work- V. T IMELOOP I NPUTS AND M APPER
load, Timeloop automatically searches for optimal mappings A. Workload Specification
within these constraints, ultimately providing performance,
Timeloop’s workload format is similar to the form of a
energy and area characterizations using a detailed architectural
single DNN layer. To evaluate a complete network, one can
cost model. Thus, it aims to serve as a super-set of many of
invoke Timeloop sequentially on each layer and accumulate
these prior approaches, and as a robust tool for rapid early-
the results. Each layer has tremendous reuse opportunities, and
stage design space exploration and competitive evaluation of
we leave exploration of cross-layer reuse to future work. In
DNN accelerator architectures.
this paper, we focus our evaluation efforts on convolutional
IV. T IMELOOP OVERVIEW (CONV) layers shown in Figure 3. These layers can be
described as a 7D nested loop over weight tensor’s height
Timeloop’s operation consists of creating a mapspace for a and width (R, S), output tensor’s height and width (P, Q),
given workload on an architecture, exploring that mapspace to number of input channels (C), number of output channels
find the optimal mapping, and reporting the performance and (K), and number of inputs or batch size (N). Matrix-matrix
energy metrics of that optimal mapping. Timeloop needs the multiplications can be expressed as convolutions by setting R,
following inputs to generate a mapspace: S, P and Q to 1, and matrix-vector multiplications can also
1) The shape and parameterization of a workload, e.g., the be expressed by setting R, S, P, Q and N to 1. Therefore,
dimensions of the input, output, and weight tensors used Timeloop also supports fully-connected (FC) layers and re-
in a DNN layer, and the set of operations needed to current neural network (RNN) layers since they are essentially
compute the layer. matrix-vector multiplications. CONV and FC layers account
2) A specification of the architecture’s hardware organiza- for a majority of the computation in DNNs (e.g., 99.25% in
tion (arithmetic units, memories and networks). ResNet50 [17] inference on ImageNet). In general, Timeloop
3) The constraints that the architecture imposes on ways can analyze any workload that can be described as a deep
in which the computation can be partitioned across the loop nest with fixed base and bounds, with operand indexing
hardware substrate and scheduled over time. expressions that are a linear composition of loop indices and
Once the mapspace is constructed, Timeloop evaluates the body iterations that may be freely re-ordered.
performance and energy efficiency of a set of mappings within From the nested loop in Figure 3, we see that loops
the space. This evaluation does not rely on a cycle-accurate have constant base and bound. Convolution operand tensors
simulator; instead, Timeloop exploits the fact that computation (i.e., weights and inputs) and result tensors (i.e., outputs) are
and data-movement patterns in DNN computations are largely indexed using linear combinations of loop indices, and all loop
 arch = { { 1 // === 1D Convolution Workload ===
arithmetic = { name = "GBuf"; 2 for r=[0:R):
instances = 256; sizeKB = 128; 3 for p=[0:P):
word-bits = 16; instances = 1; 4 Output[p] += Weight[r] * Input[r+p];
}; word-bits = 16; 5 p
6 // === Mapping ===
Weights
storage = ({ }, { r
7 // DRAM
name = "RFile"; name = "DRAM"; R
8 for r3=[0:R3):
entries = 256; technology = "DRAM"; 9 for p3=[0:P3):
instances = 256; instances = 1; 10 // GBuf
meshX = 16; word-bits = 16; 11 for r2=[0:R2): Inputs 1
word-bits = 16; }, } ); }; 12 for p2=[0:P2): W=P+R-1

Fig. 4. Eyeriss [6] organization described in Timeloop. In addition to this 13 // Spatial: GBuf->RFile
14 parallel_for r1=[0:R1):
organization spec, mapspace constraints (in Section V-D) are required to 15 parallel_for p1=[0:P1):
capture Eyeriss’ row-stationary dataflow. 16 // RFile
17 for r0=[0:R0):
18 for p0=[0:P0):
iterations are commutative. Each iteration of the loop body is a 19 p = p3*P2*P1*P0 + p2*P1*P0 + p1*P0 + p0;
MAC operation, which we refer to as a point in the operation 20 r = r3*R2*R1*R0 + r2*R1*R0 + r1*R0 + r0;
21 Output[p] += Weight[r] * Input[r+p];
space of a workload. This space consists of all the integer
lattice points enclosed by the 7D hyper-rectangle defined by
the loop bounds. These dimensions also define 4D hyper- Fig. 5. Example mapping of a 1D convolution onto the Eyeriss organization.
rectangles for each of the operand and result tensors, which we Inset: the projections defined by this workload from the operation space to
the input and weight dataspaces.
call dataspaces: the C ∗K ∗R∗S weight tensor, the N ∗K ∗P∗Q
output tensor, and the N ∗ C ∗ (P + R − 1) ∗ (Q + S − 1) input
row-stationary dataflow enforced by the architecture. This is
tensor. Given a set of points in the operation space, Timeloop
done by using mapspace constraints described in Section V-D.
can determine the set of operands and results for those points
To understand mapspace constraints, we must first understand
by projecting the 7D operation points into the 4D dataspace
the structure of a mapping in Timeloop.
dimensions.
C. Mappings
B. Architecture Specification
A mapping describes the way in which the operation space
To describe an architecture in Timeloop, one must specify and the associated dataspaces are split into tiles (chunks)
(a) the hardware organization, i.e., the topology of intercon- at each level of the memory hierarchy and among multiple
nected compute and storage units, and (b) mapspace con- memory instances at each level. We use a loop nest based
straints that limit the set of mappings allowed by the hardware. mapping representation which can describe all facets of a
We describe the organization specification in this section, and mapping in a single unified format. Figure 5 shows a 1D
cover mapspace constraints in Section V-D. convolution workload with input activation of size P and filter
Timeloop uses an organization template that is sufficiently size R followed by an example mapping onto the Eyeriss
parameterizable to model a variety of architectures of interest. organization from Figure 4. To construct the mapping, the 1D
The template is a hierarchical tree of storage elements with convolution loop nest is split into a number of sections (called
a set of arithmetic units (such as MACs) at the leaves and a tiling levels) equal to the the number of storage hierarchy
backing store (such as DRAM) holding all of the workload’s levels, plus the number of levels with spatial fanouts. Each
data at the root. Timeloop supports an arbitrary number tiling level has a loop corresponding to each dimension in the
of memory levels. For each memory level, the number of original workload (though the bound may be 1). The product
instances, entries per instance, bits in each entry, bandwidth, of all the loop bounds belonging to a dimension must be equal
and various other microarchitectural attributes can be specified. to the final (optionally padded) value of the dimension, e.g.,
Interconnection network topology is automatically inferred P = P0 ∗ P1 ∗ P2 ∗ P3 and R = R0 ∗ R1 ∗ R2 ∗ R3.
from the storage hierarchy specification; additional micro- Using the mapping in Figure 5 as an example, we see:
architectural properties can be explicitly specified. Inter-level (1) Loop bounds at each tiling level determine the size of the
networks are either point-to-point or fan-out/fan-in networks tile for each dataspace held at that level, e.g., each RFile owns
capable of multi-casting operands and/or spatially reducing an R0 ∗ P0-sized operation space tile, which projects onto R0,
partial sums (using adder trees) — abilities that have a signif- (P0 + R0 − 1), and P0-sized weight, input, and output tiles,
icant impact on energy efficiency [24]. Intra-level networks respectively. Size of the dataspace tiles is constrained by the
connect peer-instances within a level and may be used to size of the buffer at each level.
forward operands and partial sums between neighbors, eliding (2) parallel_for loops represent spatial partitioning of
expensive accesses to a larger parent storage. tiles across instances of a level, e.g., the parallel_for
Figure 4 shows the organization of Eyeriss [6] specified in loops above distribute the (R0 ∗ R1) ∗ (P0 ∗ P1) operation
the Timeloop format. The configuration uses 256 PEs (each space (and the dataspace projections thereof) across (R1 ∗ P1)
with a MAC unit and a private 256-entry register file), a single instances of the RFile. This specific mapping results in repli-
shared 128KB global buffer, and a backing DRAM. In addition cation of some input data (called a halo) between adjacent
to this organization specification, we also need to model the RFile instances.
 constraints = ({ N dimensions and force unrolling along the S (filter height)
type = "spatial"; { type = "temporal";
target = "GBuf->RFile"; target = "RFile"; dimension. The [Link] permutation constraint forces S and C
factors = "S0 P1 R1 N1"; factors = "R0 S1 Q1"; to be unrolled on the physical X-axis of the mesh of hardware
permutation = "[Link]";}, permutation = "RCP";}); PEs, and Q and K on the Y-axis.
Fig. 6. Row-stationary dataflow used in Eyeriss [6] described as a set of
When applied to a temporal tiling level (for), these
Timeloop mapspace constraints. directives affect tile sizes and data access patterns at that level.
In Figure 6, the R0 S1 Q1 factors force each PE to exhaust
(3) Ordering of loops within a tiling level determines the the entire R (filter-width) dimension as a temporal loop, and
sequence in which sub-tiles will be delivered from that level to allow it to only map one row of filters and one row of output
an inner level during execution, e.g., the GBuf owns an (R0 ∗ activations. Additionally, level bypass constraints (not shown
R1 ∗ R2) ∗ (P0 ∗ P1 ∗ P2)-sized operation space tile. Iterating in the example) dictate whether a level must compulsorily store
over the p3 loop at DRAM results in interesting changes in or bypass a tensor, or whether the mapper is free to explore
each GBuff dataspace tile: the weights remain stationary, the various alternatives.
outputs get replaced each iteration, and the inputs display an Together, the constraints in Figure 6 are sufficient to model
overlapping sliding-window pattern, which means the DRAM a row-stationary dataflow. Once the constraints are set up,
only needs to supply the portion of the tile that is not already Timeloop automatically infers and exploits the various spatio-
present in the GBuf. Inferring these dataspace changes (or temporal reuse opportunities described in [6].
deltas) is a primary function of Timeloop’s access-count model
described in Section VI-A. E. Mapspace Construction and Search
This representation produces a strictly inclusive tile hierar- A mapspace is the set of all legal mappings of a workload
chy, which may not be optimal. When a dataspace has less onto an architecture. All mappings in a mapspace have the
reuse at a level, allowing it to bypass that level opens up same number of tiling levels, but differ in (a) the values as-
the capacity to other dataspaces, enabling larger tile sizes and signed to the loop bounds at each level and (b) the permutation
potentially resulting in a more optimal mapping. Our mapping of loops within each level. To construct the mapspace, we
specification includes a directive (not shown for brevity) for must enumerate all possible factorizations of each workload
specifying which dataspaces are allowed to reside at each level. dimension across levels (we call this the IndexFactorization
The approach used for this 1D convolution example can be sub-space), all possible permutations of loops within a level
easily extended to a higher-dimensional space, such as a 7D (the LoopPermutation sub-space), and all level bypassing al-
CNN layer. This unified mapping representation allows us to ternatives (the LevelBypass sub-space). The Cartesian product
reason about the space of possible mappings (or mapspace) of these sub-spaces gives us an unconstrained mapspace,
for an architecture in a structured, programmatic manner. which can be quite large due to combinatorial explosion, e.g.,
for a 7D CNN on a 4-tiling-level architecture the size is
D. Mapspace Constraints (7!)4 × (24 )3 × size of the Cartesian product of the co-factor
By default, Timeloop assumes that a specified hardware sets for each of the 7 loop bounds. While there are ways
organization is fully flexible, i.e., networks and state machines to prune this space, e.g., permutations do not matter for the
in the hardware are fully programmable, which gives a mapper innermost tiling level, and for factors that are 1, the space is
complete flexibility in partitioning and scheduling arithmetic still large.
operations and data movement across the hardware resources. User-specified constraints are accommodated into the
To keep hardware designs simple and efficient, most real mapspace, shrinking the sizes of the underlying sub-spaces. A
architectures are constrained in various ways and only sup- mapping sampled from the mapspace is therefore guaranteed
port a subset of the complete mapspace. We model these to obey those constraints. Hardware resource constraints, e.g.,
using mapspace constraints, a generalization of the concept of whether a set of tensor tiles at a level fit into the size
dataflows [6] introduced in prior work. To explain Timeloop’s of memory at that level, are checked once a mapping is
constraints, we augment the Eyeriss architecture specification sampled from the mapspace, and the mapping is rejected if
from Section V-B with a list of constraints in Figure 6. Each the constraints cannot be met.
constraint targets a single tiling level in the mapping. A search routine samples a mapping from the pruned-
Factors in a constraint fix values for loop bounds in the and-constrained mapspace, evaluates it using the architecture
mapping (recall mapping in Figure 5). An un-specified factor model (described next in Section VI), and chooses the next
gives Timeloop’s mapper full flexibility in determining an mapping to evaluate based on some heuristic. For our experi-
optimal value. Permutations specify loop ordering within a ments, we currently employ either an exhaustive linear search
tiling level. Left un-specified (or partially specified), they give (for small mapspaces) or a random sampling based heuristic
Timeloop flexibility to reorder loops. (for large mapspaces). More sophisticated search heuristics
When applied to a spatial tiling level (parallel_for), are planned for future work. For most experiments, we use
these directives affect the partitioning of the workload space energy-delay product as the goodness metric, though any of
across a hardware spatial dimension. In Figure 6, the S0 the statistics provided by the model can be trivially used for
P1 R1 N1 factors disallow parallelism along the P, R and this purpose.
 VI. T IMELOOP M ODEL tile shape is an axis-aligned hyper-rectangle within the tensor,
which allows for easy delta calculation.
Timeloop’s architecture model evaluates a mapping by first
analyzing hierarchical tiles of data that represent the mapping
B. Microarchitecture Model
and measuring the transfer of data between them to obtain tile-
access counts. Next, these tile-access counts are used to derive Because of the regular computation and data movement
the access counts to hardware components, which combined patterns exhibited by DNN layers, buffer accesses and network
with the energy per access model give performance and energy. transfers can usually be statically scheduled, largely avoiding
The technology model also provides an area estimate based on irregular and disruptive bank and network conflicts. Therefore,
the architecture specification. given a set of tile-access counts, performance can be projected
using a throughput/bandwidth based approach, and energy can
A. Tile Analysis be estimated using linear transformations of the tile-access
counts. The first transformation step is to determine access
As explained in Section V-C, loop bounds at each tiling level
counts for various microarchitectural structures.
determine the tile for the operation space, and in turn for each
dataspace held at that level. To compute tile-access counts, we Multiplier accesses are the accesses of the innermost tile of
must determine the volumes of data that must move between the operation space.
the tiles over space and time to execute the schedule dictated Buffer accesses and network ingresses are linear functions
by the mapping. To perform this analysis, Timeloop uses a of data transfer counts between levels obtained from tile anal-
data structure called a point set to track tiles of operation and ysis. SRAM buffers can have various aspect ratios and bank
operand/result spaces at each time step and at each instance configurations, and can be ganged together as vectors for area
of a hardware unit (such as a multiplier or a buffer instance). and energy-efficiency; these configurable micro-architectural
Consider two consecutive iterations i and i+1 at any loop in parameters are factored into the access count calculations.
a mapping (from Figure 5). At each iteration, we can determine Partial sums require a read, a write and a temporal accumu-
the point sets required by the complete sub-nest underneath lation operation. Some architectures may elide the first zero-
this loop. We call the set-difference between these point sets read, and may support local accumulation at some buffers.
a delta (see Figure 7). The size of this delta has different Network activity includes traffic between levels (including fan-
connotations for spatial and temporal loops. out and multi-cast traffic), as well as traffic between storage
instances at the same level (if they forward data to each other).
For a temporal (for) loop, an empty
Multi-cast signatures from the tile analysis stage are used to
tile i delta delta indicates perfect reuse over time
determine the spatial locations of child instances targeted by a
(stationarity), requiring no additional
specific parent→child data transfer. The architectural topology
tile i+1
data movement. A non-empty delta—
1 determines the number of hops required for the data to be
indicating no reuse or partial reuse
routed to the destinations. Spatial reduction of output tensors
(such as in a sliding-window pattern)—
Fig. 7. The delta for tile is also derived from network activity.
represents the incremental data that
i + 1 are points in tile i +
must be transferred between levels. Address generators are adders surrounded by state machines
1 that are not in tile i.
For a spatial (parallel_for) used by accelerators to generate read and write addresses for
loop, the deltas represent overlaps between data held at storage elements. The number of invocations to these is equal
adjacent hardware instances (represented by the iterations i to the number of accesses to the storage element. The bit-
and i + 1). An empty delta at the same time step indicates a width of the adders is log2 of the number of vector entries in
multicasting opportunity, while an empty delta at consecutive the storage element; this is used by the energy model.
time steps between adjacent hardware instances indicates a
C. Technology-Specific Area/Energy Models
forwarding opportunity (such as in a systolic array).
To compute all tile access counts, we need to measure and Timeloop supports several user-extensible technology mod-
accumulate deltas over all space (all hardware units) and time els for area and energy modeling. The nominal TSMC 16nm
(the complete execution of the mapping). A naı̈ve but robust FinFET-based model used in most of our case studies (except
way to do this is to effectively simulate the execution of the where noted otherwise) has the following components:
entire loop nest on the architecture. Sadly, this is unacceptably (1) A Memory Model supports the modeling of memory
slow. Fortunately, DNN workloads index into operand and with different sizes, aspect ratios, number of ports and banks.
result tensors in a very regular way, and in most cases with It also supports different memory implementations, including
each loop index variable showing up only once in the indexing DRAM, SRAM and register files. The SRAM and register-
expressions for each tensor. This allows for two optimizations. file component is based on a database of area and energy-per-
First, Timeloop only needs to compute the tiles for the first, access costs that is created by generating and measuring a large
second and last iterations of each loop. The deltas between variety of memory structures with different parameters using
other iterations will be of the same shape and size as between an internal memory compiler in TSMC 16nm. The DRAM
the first and second iterations, so the access costs of these model is based on average pJ/bit access costs and supports
other iterations can be algebraically extrapolated. Second, each LPDDR4, HBM, DDR4 and GDDR5 technologies.
 TABLE I
VALIDATED DNN ACCELERATOR ARCHITECTURES .

NVDLA-derived [28] Eyeriss [8]

Dataflow Weight Stationary Row Stationary
Reduction Spatial Reduction Temporal Reduction
Memory Hierarchy Distributed and Partitioned Buffer Centralized L2 Buffer
Interconnect N/A Multicast/Unicast
Technology 16 nm 65 nm

(2) An Arithmetic Model supports modeling of MACs with

different bit-widths. It is also based on a database of area and
energy per access costs created by synthesizing and measuring
Fig. 8. Energy validation results for NVDLA-derived architecture.
various multiplier and adder designs for different bit-widths
in TSMC 16nm. For bit-widths not in the database, Timeloop
VII. T IMELOOP VALIDATION
scales energy per access either quadratically (for multipliers)
or linearly (for adders). We validate Timeloop across a wide range of workloads
(3) A Wire/Network Model supports the modeling of on- with two state-of-the-art DNN accelerators for which accurate
chip networks with different bit-widths. To model the wire performance and energy characterizations are available.
energy consumed for a network hop, area estimates are used to
determine the wire distance between the two ends of the hop. A. Architectures
This distance is multiplied with a fJ/bit/mm value measured
in TSMC 16nm to determine the per-hop energy. To demonstrate Timeloop’s ability to flexibly evaluate di-
verse DNN architectures, we validate Timeloop against two
representative DNN accelerator architectures that are notably
D. Estimating Performance and Energy
different in scale, organization, dataflow and technology. The
Performance is estimated by calculating the number of architectures’ key attributes are summarized in Table I.
cycles it would take for each hardware component to complete 1) NVDLA-derived Architecture: NVDLA is an open-
the workload in isolation. For multipliers, required cycles source hardware accelerator design for automotive applica-
are equal to the number of MACs in the workload divided tions [28]. To better understand the design trade-offs in
by the number of multipliers. For communication interfaces, DNN accelerators, we designed an in-house, NVDLA-derived
required cycles are equal to the amount of data flowing through architecture in RTL. Similar to NVDLA, it also deploys a
that interface divided by its bandwidth. Buffers, networks and weight-stationary dataflow with spatial reduction. However,
arithmetic units are modeled to be operating in a pipeline. it features a distributed and partitioned weight and input
Thus, the overall latency is the maximum of isolated execution L1 buffer for better local reuse. We use a detailed in-house
cycles across all buffers, networks, and arithmetic units in simulator designed to accurately model this specific architec-
the hardware. This model, which assumes negligible pipeline ture to derive reference performance and energy numbers for
stalls, is reasonable for architectures that use double-buffering validation against Timeloop.
or more sophisticated techniques like buffets [30]. 2) Eyeriss Architecture: Eyeriss is one of the first efficient
Energy consumption is estimated by multiplying the hard- and flexible DNN accelerators [6]. As shown in Table I, it
ware component access count with the energy per access proposes a novel row-stationary dataflow with both a temporal
from the energy model, taking sparsity into account. The reduction datapath, a centralized L2 buffer, and a flexible
total energy consumed by the mapping is the sum of energy network that supports both multicast and unicast. In addition,
consumed by all the components. Eyeriss was designed using a 65nm technology. To validate
Timeloop against Eyeriss results, we use the 65nm model from
E. Extensibility beyond regular workloads/architectures Table IV in [6].
While our throughput-based performance model and linear
B. Workloads
energy model are adequate for regular DNN architectures,
we believe that our approach of separating a high-level tile We validate Timeloop using DNN kernels from the Deep-
analysis from a low-level microarchitecture model allows Bench benchmark suite [10] augmented with additional syn-
for extensibility to more complex architectures as well. Tile thetic DNN kernels with representative configurations. The
analysis produces a compact representation of a mappings DeepBench suite includes 107 DNN workloads capturing com-
data access patterns, which can be fed into a non-linear putation in convolution, matrix-matrix multiply, and matrix-
modeling backend if desired, e.g., one with a stochastic model vector multiply of different dimensions, all of which are
of network conflicts/congestion, or even a full simulation of a commonly used in modern neural networks, e.g., computer
network serving those accesses. vision and speech recognition.
 1.1
1e10 1e10
1 RF RF
2.0 Buffer+Array 2.0 Buffer+Array
0.9
DRAM DRAM
ALU ALU

Normalized Energy
Normalized Energy
0.8 1.5
1.5
0.7

0.6 1.0 1.0

Accuracy

0.5
0.5 0.5
0.4

0.3 0.0 CONV1 CONV2 CONV3 CONV4 CONV5 0.0 CONV1 CONV2 CONV3 CONV4 CONV5
0.2

0.1 (a) Data from [6] (b) Timeloop Model

0
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71
Fig. 10. Normalized energy for AlexNet layers on a 256-PE Eyeriss [6]
architecture employing a row-stationary dataflow.
Workload

Fig. 9. Performance validation results for NVDLA-derived architecture. previously described as examples in Sections V-B and V-D
respectively. We see that Timeloop’s estimation tracks quite
C. Validation Results closely with the energy reported from the prior study.
Figure 8 shows energy validation results for Timeloop
against our NVDLA-derived baseline. The bar on the left VIII. C ASE S TUDIES
shows the energy breakdown from Timeloop, and the bar This section demonstrates Timeloop’s power and flexibil-
on the left represents the baseline energy. All the energy is ity by evaluating a variety of distinct architectures such as
normalized to the total baseline energy. The X-axis represents NVDLA [28], Eyeriss [6] and DianNao [5] over a range of
different workloads from DeepBench. We show a subset applications. We first show how designers can use Timeloop to
of DeepBench workloads in Figure 8 for brevity, though quickly analyze different application behaviors on a given ar-
Timeloop accurately captures the energy behavior of the chitecture and quantify its performance and energy efficiency.
accelerator across the entire benchmark suite, with projections We then present a sampling of case studies that DNN acceler-
for all 107 workloads within 8% of the baseline. ator designers can conduct with Timeloop, e.g., understanding
Figure 9 shows performance validation results for Timeloop the impact of technology scaling on the mapping strategy,
against our NVDLA-derived baseline architecture. The X- optimizing the energy efficiency through tailoring the memory
axis represents a range of synthetic workloads and the Y-axis hierarchy for the dataflow, and systematically comparing and
plots accuracy measured as the cycles reported by Timeloop quantifying the differences between existing DNN accelerator
divided by reference cycles. Accuracy ranges from 78% to architectures.
99% with a mean of 95% across all workloads. Recall that
Timeloop uses a throughput based performance model that A. Case Study: Workload Characterization
assumes minimal pipeline disruptions from fills and drains. We first demonstrate how DNN accelerator designers can
Despite this assumption, accuracy is high (90% to 99%) on use Timeloop to characterize an architecture over a suite of
all but six outlier workloads, in this case because the hardware workloads and derive a host of insightful statistics. Specifi-
uses buffets [30], which provides efficient overlap of compute cally, we use Timeloop to evaluate all DeepBench workloads
and data-transfers with minimal additional storage. This can running on the NVDLA architecture. Figure 11 shows a
also achieved via double-buffering but at the cost of twice the detailed characterization of each workload with its optimal
amount of storage. mapping found using Timeloop. The Y-axis on the left shows
The lower accuracy (78% to 88%) for six outlier workloads the normalized total energy over the MAC energy, while
is a consequence of sub-optimal data layout and transfer orders the one on the right shows the algorithmic reuse of each
in the hardware. For example, in one case, the sequence of benchmark, defined as the number of MACs divided by the
addresses used by the hardware address-generator filling the minimum number of DRAM accesses, i.e., the total size of
Input Buffer was different from the sequence used by the inputs, weights and outputs. We also show the MAC utilization
address-generator reading from the buffer, causing unexpected on the top where we achieve close-to-1 utilization in most of
pipeline stalls. We belatedly realized that these are sub- cases except the ones with shallow input (C < 64) and output
optimal configurations of the hardware, and believe that better (K < 16) channels as NVDLA spatially maps C and K to its
configurations would show better correlation with Timeloop’s MACs. We observe that energy is dominated by DRAM for
projections. workloads with low reuse. On the contrary, workloads with
Figure 10 shows results for the validation experiment high reuse are more affected by the efficiency of on-chip
against the Eyeriss architecture. Specifically, our objective components. This study shows how Timeloop allows DNN
here is to recreate the experiment and results in Figure 10 accelerator designers to quickly evaluate a DNN accelerator
of the Eyeriss paper [6], with AlexNet [21] as the workload. across a wide range of application without having to write
The architecture’s organization and mapspace constraints were RTL or even a C++ model.
 1
MAC Utilization
0
4.5 MAC Registers Accumulation Buffer Weight Input Buffer DRAM Reuse 700
62.2
4
55.3 600

Reuse (MAC/Memory Accesses)

Normalized Energy per MAC

3.5
48.4
500
3
41.5
2.5 400
34.6
2
27.6 300
1.5
20.7
200
1
13.8
100
0.5
6.9
00 0
67
21
94
51
91
86
56
58
32
48

88
36
97
33
14
87
20
53
92
77
54
31
2
96
80
7
83
84

27
37
8
46
19
11
50
68
4
65
12
52
35
66
22
69
18
70
17
71
72
105
106

101
100
102

104

103
10
59
15
78
42
75

47
49
39
57
89

90
29
81
40
26

44
76
93
45
38
0
1
79
95
99
82
85
98
34
30
13
55
9
25
41
60
5
3
73
43
28
74
16
61
24
62
23
63
64
10 59 15 78 42 7510247 49 39 57 8910490 29 81 40 2610344 76 93 45 38 0 1 79 95 99 82 85 98 34 30 13 55 9 25 41 60 5 3 73 43 28 74 16 61 24 62 23 63 64
DeepBench Workload Index

Fig. 11. Energy/MAC breakdown for DeepBench workloads sorted by algorithmic reuse.

B. Case Study: Impact of Technology C. Case Study: Memory Hierarchy Optimization

Technology differences change the relative energy and area The Eyeriss architecture we have modeled so far uses a 256-
costs of different components, e.g., arithmetic and memory, entry register file (RF) in each PE as the innermost storage
which may in turn change the optimal mapping of a workload level, which is shared across all dataspaces and contributes
on an architecture. This study evaluates Timeloop’s Eyeriss a significant amount of energy consumption, as shown in
configuration with different technology models and highlights Figure 12. In this study, we explore two designs that aim
the importance of a mapper in the evaluation of DNN architec- to reduce the RF energy consumption by (1) adding an
tures. Specifically, we compare energy efficiency and different additional one-entry register at the innermost storage level, and
mapping choices using both Eyeriss’ original 65nm energy (2) partitioning the shared RF into separate ones for inputs,
model and our 16nm energy model. weights and partial sums. As the row-stationary dataflow has
Figure 12(a) shows the energy breakdown of Eyeriss archi- high temporal locality in the RF for inputs and partial sums,
tecture running AlexNet layers with 65nm and 16nm models. the RFs for inputs and partial sums are allocated with 12 and
Energy is normalized to the total energy of each layer. In 16 entries, respectively, to take advantage of a small capacity
both configurations, we use the same optimal mapping from to reduce their access cost. The rest of the entries are allocated
65nm (65map). We see that technology change causes a for the weight. This is inspired by how Eyeriss is actually
re-distribution of energy between hardware components. In implemented in [8], which is slightly different from the model
addition, we see that the optimality of mappings does not in [6].
trivially carry over across different technologies—the optimal Figure 13 shows the normalized energy for various work-
mapping for 65nm is sub-optimal for the 16nm model, as loads (with batch size of 1) of the three Eyeriss variants.
illustrated in Figure 12(b). We can reduce the overall energy For each layer, the bars from left to right are Eyeriss with:
up to 22% by re-evaluating the mapspace with the 16nm model (1) a shared RF, (2) a shared RF and an additional register,
and finding the optimal mapping for that technology (16map). and (3) a partitioned RF. The result shows that the memory
MAC DRAM Buffer + Array RF MAC DRAM Buffer + Array RF
Both using 65map Both using 16nm 470
100% 7E+10
460
6E+10
Normalized Energy/MAC
Normalized Energy

80%
Energy Breakdown

5E+10 450
60% Shared RF
4E+10 440
Shared RF + Register
3E+10
40% 40 Partitioned RF
2E+10
20% 30
1E+10
0 20
0%
CONV1-65map
CONV1-16map
CONV2-65map
CONV2-16map
CONV3-65map
CONV3-16map
CONV4-65map
CONV4-16map
CONV5-65map
CONV5-16map
CONV1-65nm
CONV1-16nm
CONV2-65nm
CONV2-16nm
CONV3-65nm
CONV3-16nm
CONV4-65nm
CONV4-16nm
CONV5-65nm
CONV5-16nm

10

0
1 2 3 4 5 6 7 8 0 8 3 9 9
NV NV NV NV NV FC FC FC _1 _4 _9 h_ _6
CO CO CO CO CO t_ t_ t_ ch ch ch nc ch
(a) (b) t_ t_ t_ t_ t_ x Ne x Ne x Ne Ben Ben Ben pBe Ben
e e e e e e e e p p p e p
ex
N
ex
N
ex
N
ex
N
ex
N Al Al Al e e e De e
Al Al Al Al Al De De De De

Fig. 12. (a) Energy breakdown of the same mapping with different technolo-
gies, i.e., 65nm and 16nm. (b) Normalized energy of the same technology Fig. 13. Normalized energy per MAC for three different Eyeriss designs. For
16nm with different mapping choices. 65map is the optimal mapping with each layer, the three bars from left to right are Eyeriss with (1) a shared RF,
65nm model, while 16map is the optimal mapping with 16nm model. (2) a shared RF and an additional register, and (3) a partitioned RF.
 1.0 500
Normalized Performance

Normalized Energy/MAC
0.5
100

0.1
10

0 0
V1 V2 V3 V4 V5 FC
6
FC
7
FC
8
_1
0
_4
8
_9
3
h_
9
_6
9 V1 V2 V3 V4 V5 FC
6
FC
7
FC
8
_1
0
_4
8
_9
3
h_
9
_6
9
ON ON ON ON ON t_ t_ t_ ch ch ch nc ch ON ON ON ON ON t_ t_ t_ ch ch ch nc ch
_C _C _C _C _C Ne Ne Ne en en en Be en _C _C _C _C _C Ne Ne Ne en en en Be en
N et N et N et N et N et l ex l ex l ex pB pB pB ep pB N et N et N et N et N et l ex l ex l ex pB pB pB ep pB
A A A e e e e e A A A e e e e e
ex ex ex ex ex De De De D De ex ex ex ex ex De De De D De
Al Al Al Al Al Al Al Al Al Al

(a) Normalized Performance (b) Normalized Energy/MAC

Fig. 14. Performance and Energy Efficiency Comparisons of NVDLA, DianNao, and Eyeriss.

hierarchy optimizations in (2) and (3) reduce the total energy the fact that most of the energy in Eyeriss is consumed by
consumption in all the workloads. The improvement is more the RF in each PE instead of shared buffers. As a result, its
pronounced in convolutional layers, which show an over 40% energy scales approximately with the number of PEs.
reduction in energy. This case study shows that the co-design The case studies in this section demonstrate Timeloop’s
of dataflow and memory hierarchy implementation is crucial capability for modeling and fairly comparing a wide range
to the energy efficiency of an architecture; Timeloop can of architectures on a diverse set of workloads. The results of
effectively guide this design space exploration. such studies should be useful to inform the design of future
DNN accelerator architectures.
D. Case Study: Modeling of Existing Architectures
In this study, we use Timeloop to model and compare IX. C ONCLUSIONS AND F UTURE W ORK
the performance and energy efficiency of three representative We presented Timeloop, an infrastructure for modeling and
architectures from prior work: NVDLA, DianNao, and Eyeriss. evaluating a rich design space of DNN accelerator architec-
Since NVDLA has more PEs (1024 PEs) than the nominal tures. We demonstrated how to describe DNN architectures
configurations of DianNao and Eyeriss (256 PEs), we also using a generic template, characterize workloads such as
model scaled variants of DianNao and Eyeriss with 1024 PEs convolutional layers, comprehensively describe mappings us-
each. Increasing the number of PEs scales the multipliers, ing a loop-nest-style representation, and efficiently enumerate
buffers and network, but results in each architecture occupying the space of feasible mappings by combining workload and
a different silicon area from NVDLA (because of organiza- architectural constraints. We also described computationally
tional differences). Therefore, we then adjust the buffer sizes inexpensive and modular techniques for modeling the data
to align the final area with NVDLA. Figure 14 shows the transfers and arithmetic for a given mapping, which gives
normalized performance and energy of the three architectures. energy and performance estimates with enough speed and
First, NVDLA demonstrates higher performance and energy accuracy to rapidly explore architectural trade-offs, while
efficiency compared to the other two architectures except including the important step of using an appropriate mapping
for AlexNet CONV1 and DeepBench workload 10, both of for each accelerator design. Overall, this provides a practical
which have shallow input channels (C). As both NVDLA and and powerful automated tool for understanding the complex
DianNao map input channels spatially across PEs, when the trade-offs in DNN accelerator design across a wide range of
network has smaller number of input channels, not all the PEs workloads and helps navigate those trade-offs in a systematic
can be utilized during its execution. In contrast, Eyeriss shows fashion.
a consistent performance across different workloads thanks Future work includes modeling inter-layer relationships to
to its flexible mapping scheme. Overall, this demonstrates find globally-optimal solutions for full networks (as in [2]).
that there is no single architecture in this experiment that is Also, although Timeloop already accounts for the energy
universally better than the others. savings due to sparsity, future work includes modeling archi-
In addition, the scaled version of DianNao demonstrates tectures that save both time and energy [1], [15], [29], [39].
a higher energy efficiency and better performance across all Other future work includes modeling accelerators for other
the workloads than its default version. This is mainly due to domains like graph analytics [40] and sparse tensor algebra
the additional spatial reuse of inputs and the efficient spatial [20].
reduction of partial sums in the larger PE array, which greatly The generality of Timeloop opens the door to a variety of re-
amortizes the cost of accessing the on-chip buffers. search directions in accelerator architecture design, especially
On the contrary, while the performance of Eyeriss improves for accelerators with mapping dependent behavior. We are in
with more PEs, its energy efficiency stays relatively the same the process of releasing the infrastructure to the community
between the scaled and default configurations. This is due to under an open-source license.
 ACKNOWLEDGMENTS [13] Mingyu Gao, Jing Pu, Xuan Yang, Mark Horowitz, and Christos
Kozyrakis. Tetris: Scalable and efficient neural network acceleration
We thank Michael Pellauer and Christopher W. Fletcher for with 3d memory. In Proceedings of the International Conference
on Architectural Support for Programming Languages and Operation
sharing their insights on dataflows over numerous discussions. Systems (ASPLOS), 2017.
We also appreciated the valuable insights William J. Dally [14] A. Graves and J. Schmidhuber. Framewise Phoneme Classification With
provided. Finally, we thank the anonymous reviewers for their Bidirectional LSTM and Other Neural Network Architectures. In Neural
Networks, 2005.
helpful and insightful comments. [15] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark
Horowitz, and Bill Dally. EIE: Efficient Inference Engine on Com-
Distribution Statement A. This research was, in part, funded pressed Deep Neural Network. In Proceedings of the International
by the U.S. Government. The views and conclusions contained Symposium on Computer Architecture (ISCA), pages 243–254, June
2016.
in this document are those of the authors and should not be in- [16] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen,
terpreted as representing the official policies, either expressed R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y. Ng. Deep
or implied, of the U.S. Government. Speech: Scaling Up End-To-End Speech Recognition. [Link]
abs/1412.5567, 2014.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep
R EFERENCES residual learning for image recognition. 2016 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pages 770–778,
2016.
[1] Jorge Albericio, Patrick Judd, Tayler Hetherington, Tor Aamodt, Na-
[18] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav
talie Enright Jerger, and Andreas Moshovos. Cnvlutin: Ineffectual-
Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden,
Neuron-Free Deep Convolutional Neural Network Computing. In
Al Borchers, Rick Boyle, Pierre luc Cantin, Clifford Chao, Chris
Proceedings of the International Symposium on Computer Architecture
Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben
(ISCA), pages 1–13, June 2016.
Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland,
[2] Manoj Alwani, Han Chen, Michael Ferdman, and Peter Milder. Fused- Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert
layer CNN accelerators. In Proceedings of the International Symposium Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexan-
on Microarchitecture (MICRO), 2016. der Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen
[3] Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris
Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean,
Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi
Fougner, Tony Han, Awni Hannun, Billy Jun, Patrick LeGresley, Libby Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick,
Lin, Sharan Narang, Andrew Ng, Sherjil Ozair, Ryan Prenger, Jonathan Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir
Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snel-
Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and ham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory
Zhenyao Zhu. Deep Speech 2: End-To-End Speech Recognition in Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard
English and Mandarin. [Link] 2015. Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter
[4] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie performance analysis of a tensor processing unit. In Proceedings of the
Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis International Symposium on Computer Architecture (ISCA), pages 1–12,
Ceze, Carlos Guestrin, and Arvind Krishnamurthy. TVM: An automated 2017.
end-to-end optimizing compiler for deep learning. In 13th USENIX [19] Liu Ke, Xin He, and Xuan Zhang. Nnest: Early-stage design space ex-
Symposium on Operating Systems Design and Implementation (OSDI ploration tool for neural network inference accelerators. In Proceedings
18), pages 578–594, Carlsbad, CA, 2018. USENIX Association. of the International Symposium on Low Power Electronics and Design,
[5] Tianshi Chen, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji ISLPED ’18, pages 4:1–4:6, New York, NY, USA, 2018. ACM.
Chen, and Olivier Temam. DianNao: A Small-footprint High-throughput [20] Fredrik Kjolstad, Stephen Chou, David Lugato, Shoaib Kamil, and
Accelerator for Ubiquitous Machine-learning. In Proceedings of the Saman Amarasinghe. The Tensor Algebra Compiler. In Proc. OOPSLA,
International Conference on Architectural Support for Programming 2017.
Languages and Operation Systems (ASPLOS), pages 269–284, March [21] A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet Classification
2014. with Deep Convolutional Neural Networks. In Proceedings of the
[6] Yu-Hsin Chen, Joel Emer, and Vivienne Sze. Eyeriss: A Spatial International Conference on Neural Information Processing Systems
Architecture for Energy-Efficient Dataflow for Convolutional Neural (NIPS), December 2012.
Networks. In Proceedings of the International Symposium on Computer [22] Hyoukjun Kwon, Ananda Samajdar, and Tushar Krishna. Maeri:
Architecture (ISCA), pages 367–379, June 2016. Enabling flexible dataflow mapping over dnn accelerators via reconfig-
[7] Yu-Hsin Chen, Joel Emer, and Vivienne Sze. Using Dataflow to urable interconnects. In Proceedings of the Twenty-Third International
Optimize Energy Efficiency of Deep Neural Network Accelerators. IEEE Conference on Architectural Support for Programming Languages and
Micro’s Top Picks from the Computer Architecture Conferences, 37(3), Operating Systems, ASPLOS ’18, pages 461–475, New York, NY, USA,
May-June 2017. 2018. ACM.
[8] Yu-Hsin Chen, Tushar Krishna, Joel Emer, and Vivienne Sze. Eyeriss: [23] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep Learning.
An Energy-efficient Reconfigurable Accelerator for Deep Convolutional Nature, 521:436–444, May 2015.
Neural Networks. In Proceedings of the International Solid State [24] C. Lee, Y.S. Shao, J-F Zhang, A. Parashar, J. Emer, S.W. Keckler,
Circuits Conference (ISSCC), February 2016. and Z. Zhang. Stitch-x: An accelerator architecture for exploiting
[9] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and unstructured sparsity in deep neural networks. In SysML Conference,
P. Kuksa. Natural Language Processing (Almost) From Scratch. In 2018.
[Link], 2011. [25] Wenyan Lu, Guihai Yan, Jiajun Li, Shijun Gong, Yinhe Han, and
[10] DeepBench. [Link] Xiaowei Li. Flexflow: A flexible dataflow accelerator architecture for
[11] G. Diamos, S. Sengupta, B. Catanzaro, M. Chrzanowski, A. Coates, convolutional neural networks. In Proceedings of the International
E. Elsen, J. Engel, A. Hannun, and S. Satheesh. Persistent RNNs: Symposium on High-Performance Computer Architecture (HPCA), 2017.
Stashing Recurrent Weights On-Chip. In ICML, June 2016. [26] Yufei Ma, Yu Cao, Sarma Vrudhula, and Jae-sun Seo. Optimizing loop
[12] Zidong Du, Robert Fasthuber, Tianshi Chen, Paolo Ienne, Ling Li, Tao operation and dataflow in FPGA acceleration of deep convolutional
Luo, Xiaobing Feng, Yunji Chen, and Olivier Temam. ShiDianNao: neural networks. In Proceedings of the ACM/SIGDA International
Shifting Vision Processing Closer to the Sensor. In Proceedings of the Symposium on Field-Programmable Gate Arrays, 2017.
International Symposium on Computer Architecture (ISCA), pages 92– [27] Mohammad Motamedi, Philipp Gysel, Venkatesh Akella, and Soheil
104, June 2015. Ghiasi. Design space exploration of FPGA-based deep convolutional
 neural networks. In 21st Asia and South Pacific Design Automation 278, June 2016.
Conference (ASP-DAC), 2016. [34] Hardik Sharma, Jongse Park, Divya Mahajan, Emmanuel Amaro,
[28] Nvidia. NVDLA Open Source Project, 2017. Joon Kyung Kim, Chenkai Shao, Asit Mishra, and Hadi Esmaeilzadeh.
[29] Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, From high-level deep neural models to FPGAs. In Proceedings of the
Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Stephen W Keck- International Symposium on Microarchitecture (MICRO), 2016.
ler, and William J Dally. SCNN: An accelerator for compressed-sparse [35] Yongming Shen, Michael Ferdman, and Peter Milder. Maximizing CNN
convolutional neural networks. In Proceedings of the International accelerator efficiency through resource partitioning. In Proceedings of
Symposium on Computer Architecture (ISCA), 2017. the International Symposium on Computer Architecture (ISCA), 2017.
[30] Michael Pellauer, Yakun Sophia Shao, Jason Clemons, Neal Crago, Kar- [36] Kodai Ueyoshi, Kota Ando, Kentaro Orimo, Masayuki Ikebe, Tetsuya
tik Hegde, Rangharajan Venkatesan, Stephen W. Keckler, Christopher W. Asai, and Masato Motomura. Exploring optimized accelerator design
Fletcher, and Joel Emer. Buffets: An efficient and composable storage for binarized convolutional neural networks. In International Joint
idiom for explicit decoupled data orchestration. In Architectural Sup- Conference on Neural Networks (IJCNN), 2017.
port for Programming Languages and Operating Systems (to appear),
ASPLOS ’19, 2019. [37] Kaiyi Yang, Shihao Wang, Jianbin Zhou, and Takeshi Yoshimura.
[31] Benoı̂t Pradelle, Benoı̂t Meister, M. Baskaran, Jonathan Springer, and Energy-efficient scheduling method with cross-loop model for resource-
Richard Lethin. Polyhedral Optimization of TensorFlow Computation limited cnn accelerator designs. In IEEE International Symposium on
Graphs. In Workshop on Extreme-scale Programming Tools (ESPT), Circuits and Systems (ISCAS), 2017.
November 2017. [38] Tien-Ju Yang, Yu-Hsin Chen, Joel Emer, and Vivienne Sze. A method to
[32] Atul Rahman, Sangyun Oh, Jongeun Lee, and Kiyoung Choi. Design estimate the energy consumption of deep neural networks. In Asilomar
space exploration of FPGA accelerators for convolutional neural net- Conference on Signals, Systems and Computers, 2017.
works. In Design, Automation & Test in Europe Conference & Exhibition [39] Shijin Zhang, Zidong Du, Lei Zhang, Huiying Lan, Shaoli Liu, Ling Li,
(DATE), 2017. Qi Guo, Tianshi Chen, and Yunji Chen. Cambricon-X: An Accelerator
[33] Brandon Reagen, Paul Whatmough, Robert Adolf, Saketh Rama, for Sparse Neural Networks. In Proceedings of the International
Hyunkwang Lee, Saekyu Lee, Jose Miguel Hernandez Lobato, Gu- Symposium on Microarchitecture (MICRO), October 2016.
Yeon Wei, and David Brooks. Minerva: Enabling Low-Power, High- [40] Yunming Zhang, Mengjiao Yang, Riyadh Baghdadi, Shoaib Kamil,
Accuracy Deep Neural Network Accelerators. In Proceedings of the Julian Shun, and Saman Amarasinghe. GraphIt-A High-Performance
International Symposium on Computer Architecture (ISCA), pages 267– DSL for Graph Analytics. arXiv preprint arXiv:1805.00923, 2018.