Neural Networks 45 (2013) 4–26

Contents lists available at SciVerse ScienceDirect

Neural Networks
journal homepage: [Link]/locate/neunet

2013 Special Issue

Design of silicon brains in the nano-CMOS era: Spiking neurons,

learning synapses and neural architecture optimization
Andrew S. Cassidy a,1 , Julius Georgiou b , Andreas G. Andreou a,b,∗
a
Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
b
Department of Electrical and Computer Engineering, University of Cyprus, Nicosia 1678, Cyprus

article info abstract

Keywords: We present a design framework for neuromorphic architectures in the nano-CMOS era. Our approach
Silicon brains
to the design of spiking neurons and STDP learning circuits relies on parallel computational structures
Neuromorphic engineering
where neurons are abstracted as digital arithmetic logic units and communication processors. Using this
Silicon neurons
Learning in silicon approach, we have developed arrays of silicon neurons that scale to millions of neurons in a single state-
FPGA neural arrays of-the-art Field Programmable Gate Array (FPGA). We demonstrate the validity of the design methodology
through the implementation of cortical development in a circuit of spiking neurons, STDP synapses, and
neural architecture optimization.
© 2013 Elsevier Ltd. All rights reserved.

1. The computer and the brain units abstract the function of neurons while synapses abstract
the connections between neurons. The strength of the synaptic
The brain is a massively parallel and efficient information connections in networks of such units is determined through a
processing system, with a radically different computational learning algorithm. A two volume edited book-set by the ‘‘Parallel
architecture from present day computers. Characteristics of neural Distributed Research Group’’ (McClelland, Rumelhardt, & Group,
computation include event based processing, fine-grained parallel 1987; Rumelhart, McClelland, & Group, 1987) defined the research
computational units, robustness and redundancy, as well as agenda in the field of connectionist architectures and neural
adaptation and learning, all done under severe constraints of size, networks in the decades that followed. At about the same time,
weight, and energy resources. This computational architecture Carver Mead’s book ‘‘Analog VLSI and Neural Systems’’ (Mead,
excels at lower-level sensory information processing such as 1989) inspired a new generation of scientists and engineers to
vision, and sensor–motor integration as well as cognitive tasks explore hardware implementation of neural models in state-of-
such as speech and language understanding. the-art silicon integrated circuit technology. The book had a dual
Over the last half century computer scientists, architects and objective: (i) to create a new design discipline for collective
engineers have envisioned building computers that match the computational systems using analog VLSI subthreshold CMOS
parallel processing capabilities of biological brains. Fifty years integrated circuit technology and (ii) to promote a synthetic
ago, the fathers of computer science Alan Turing (Turing, 1952) approach in the understanding of biology and the human brain.
and John von-Neumann (Neumann, 1958) looked to the brain for This was the birth of neuromorphic design as an engineering
inspiration in order to advance the science of computing. discipline.
Twenty-five years ago, the connectionist movement emerged
as an alternative approach to artificial intelligence for solving the 1.1. Neuromorphic engineering: the formative years
hard problems in perception and cognition. The central doctrine
in the connectionist movement is that the cognitive abilities of ‘‘Neuromorphic’’ electronic systems, a term coined by Carver
the brain are a result of a highly interconnected network of Mead in the late 1980s, describes systems that perform artificial
simple processing units. These simple non-linear computational computation based on the principles of neurobiological circuits. In
the following two decades, inspired by Mead’s pioneering work
(Mead, 1990) and colleagues at Caltech, a large number of CMOS
∗ Corresponding author at: Department of Electrical and Computer Engineering, neuromorphic chip designs have been reported in the literature.
Johns Hopkins University, Baltimore, MD 21218, USA. These spanned a wide range of designs from analog VLSI models
E-mail addresses: andrewca@[Link] (A.S. Cassidy), julio@[Link] of neurons (Arthur & Boahen, 2010; Hsin, Saighi, Buhry, & Renaud,
(J. Georgiou), andreou@[Link] (A.G. Andreou). 2010; Saighi, Bornat, Tomas, Le Masson, & Renaud, 2010; Yu,
1 Now with IBM Research, Almaden, USA. Sejnowski, & Cauwenberghs, 2011) to silicon retina architectures
0893-6080/$ – see front matter © 2013 Elsevier Ltd. All rights reserved.
[Link]
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 5

(Boahen & Andreou, 1992; Mahowald, 1992), and retinomorphic merges a CNN–UM type processor and an imager (Carmona et al.,
vision systems (Boahen, 1996), to attention circuits (Horiuchi 1998; Dominguez-Castro et al., 1997). This system, while analog
& Koch, 1999), and biomorphic imagers (Culurciello, Etienne- internally, has a digital interface with on-chip 7-bit A/D and D/A
Cummings, & Boahen, 2003) that abstract biology at a lower converters, improving the programmability and simplifying the in-
level. Other mixed-mode designs (Andreou, Meitzler, Strohben, terface to digital computers (Cembrano et al., 2004).
& Boahen, 1995; Pardo, Dierickx, & Scheffer, 1998) and (Etienne- Programmable analog VLSI circuits and systems aimed at large-
Cummings, Kalayjian, & Donghui, 2001) have also implemented scale model simulation have also been under development in
silicon retinas and focal plane processing architectures that include the last decade. The Neurogrid architecture in Kwabena’s group
processing beyond gain control and spatio-temporal filtering, (Arthur & Boahen, 2010; Choudhary et al., 2012; Silver, Boahen,
including polarization sensing (Andreou & Kalayjian, 2002; Wolff Grillner, Kopell, & Olsen, 2007), the IFAT architecture (Goldberg,
& Andreou, 1995). Most of the above bio-inspired sensors have Cauwenberghs, & Andreou, 2001a; Vogelstein, Mallik, Culurciello,
limited programmability as they employ analog computational Cauwenberghs, & Etienne-Cummings, 2007), the PAX platform
circuits at the focal plane. (Renaud et al., 2010) and the FACETS wafer-scale computational
The shortcomings of non-programmable analog architectures infrastructure (Bruederle et al., 2011) are notable projects in this
motivated the exploration of analog vision chip architectures direction.
with programmable functionality (Serrano-Gotarredona, Andreou,
& Linares-Barranco, 1999; Serrano-Gotarredona et al., 2009). 1.2. Neuromorphic engineering: the nano-CMOS Era
Programmable architectures for associative memory (Boahen,
Pouliquen, Andreou, & Jenkins, 1989; Pouliquen, Andreou, & In 1986, Mead’s group at Caltech was employing bulk CMOS
Strohben, 1997), pattern classification (Genov & Cauwenberghs, technology with λ between 2.5 micron and 0.7 micron (p. 59
2001; Karakiewicz, Genov, & Cauwenberghs, 2007) and audition of Mead, 1989). A quick review of our own publications and
(Kumar, Himmelbauer, Cauwenberghs, & Andreou, 1998; Stanace- laboratory notebooks from that period, reveals that we were
vic & Cauwenberghs, 2005) have also been reported in the litera- fabricating chips in 4 micron Silicon On Sapphire (SOS)–CMOS
ture. technology and in 3 micron p-well bulk CMOS. Alas! Twenty
Programmable architectures have also been advanced by the five years later, with foundry CMOS technologies at the 45 nm
adoption of a standard interface between chips known as Address and 22 nm nodes, the neuromorphic engineering community
Event Representation or (AER) in short. The time-multiplexed AER
at large has not been able to capitalize on the benefits of the
bus (Boahen, 2000; Lin & Boahen, 2009; Mahowald, 1992; Sivilotti,
(×10 000) improvements in digital MOS transistor area density to
1991) is a popular interconnect method for neuromorphic systems.
engineer brain like structures and cognitive machines that match
Spike events from multiple channels are time-multiplexed onto
the effectiveness and energetic efficiency of the human brain. With
a digital AER bus, transmitted, and decoded at the destination
the exception of the event-based, asynchronous vision sensors
onto individual channels. Throughout this proposal, we use the
(Lichtsteiner, Posch, & Delbruck, 2008) and subsequent design
terms spikes, events, and spike events interchangeably. AER has
(Posch, Matolin, & Wohlgenannt, 2011), the goals of endowing
been used by many analog and digital spiking neural arrays,
modern computer systems with industrial-strength robust bio-
as well as to communicate events from off-chip neuromorphic
inspired sensoria or tackling the challenge of silicon cognition have
sensors and even in 3D CMOS technology (Harrison, Özgün, Lin,
been unrealized. And even though our lack of knowledge about
Andreou, & Etienne-Cummings, 2010). The European Union project
the inner workings of brain function and behavior has contributed
CAVIAR ([Link] demonstrated a
to this chasm and is limiting us today, matching the information
board-level vision system architecture communicating using the
processing capabilities of biological neural structures in state-of-
AER protocol (Serrano-Gotarredona et al., 2009). Variants of AER
to improve the efficiency of the protocol have also been proposed the-art silicon technology has remained an elusive dream despite
(Georgiou & Andreou, 2006, 2007). A probabilistic approach to AER the stunning advances in microelectronics.
has been exploited to perform computations in the address domain Even more elusive has been our quest to understand how to
(Goldberg, Cauwenberghs, & Andreou, 2001b). achieve the energy efficiency seen in biological brains. One would
Learning in silicon has also been pursued intensively in the have thought that the research activities in the last two decades
analog VLSI neuromorphic community. The early work by Dio- would have brought us closer to both a deeper understanding of
rio and colleagues (Diorio, Hasler, Minch, & Mead, 1996, 1997), brain function as well as to commercially-viable brain-inspired
the Field Programmable Analog Arrays (Sivilotti, 1991) and the information technology at the scale. However, this is not the case.
research program of Hasler (Hall, Twigg, Gray, Hasler, & Ander- Many of the analog VLSI neuromorphic systems rely on analog
son, 2005) paved the way to floating gate MOS transistors in con- devices and as such, scaling the density of these components
figurable learning chips. Other designs employ dynamic circuits (mostly MOS transistors and capacitors) did not follow Moore’s
for implementing learning in analog VLSI with excellent results law. Furthermore, the majority of neuromorphic hardware was
on small systems (Bartolozzi & Indiveri, 2007; Indiveri, Chicca, & based on traditional ‘‘analog’’ circuit models of neurons and
Douglas, 2004; Mahowald, 1992). This work has continued with synapses, a technology that does not offer flexibility in component
encouraging results for hardware models that abstract higher-level models, nor in their level of description; an aspect which impedes
functions such as stimulus specific adaptation (Mill, Sheik, Indiveri, rapid advances.
& Denham, 2011) and working memory using attractor dynamics Mead advocated using analog transistor physics to perform
(Giulioni et al., 2011). neural computation, directly mimicking the currents in neuron
Abstracting biology at a higher level, the Cellular Non-linear/ ion channels (Mead, 1990), and speculated that an energy
Neural Networks (CNN) approach (Chua & Yang, 1988) offered an- savings of approximately 104 could be gained over comparable
other paradigm for an analog visual processor with programming traditional digital approaches. However the power dissipation of
capabilities. In CNN architectures, information processing is im- neuromorphic systems did not benefit from technology scaling
plemented through the evolution of a continuous-time non-linear either and our best circuits today hover between 10 and 100 nW
dynamical network with nearest neighborhood connectivity. The per computational cell. Each cell has typically one or two single
CNN–UM (Universal Machine) is one of the earliest systems (Roska pole circuits with two or three current branches biased in the nano-
& Chua, 1993) that implemented CNN programmable functional- ampere current level. Even though one could argue the power
ity on a chip. Another example of CNN hardware implementation dissipation is manageable locally, the energy cost to send the
6 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

digital representation of the state from one chip to another on rapidly prototype digital systems makes them an attractive plat-
the same board is high. It takes less than a femto-Joule of energy form for bio-inspired systems exploration in the nano-CMOS era.
to move one bit worth of charge through the source to the drain The design of small systems and applications of digital Spiking
of an MOS transistor in deep sub-micron CMOS technology, one Neural Networks (SNNs) have already been reported in FPGAs
pico-Joule to move one bit of information across a 1 cm die, and (Belhadj, Tomas, & Bornat, 2009; Cardoso, Diniz, & Weinhardt,
almost one hundred pico-Joules to move it from one die to another! 2010; Cassidy, Denham, Kanold, & Andreou, 2007; Koickal, Gou-
(Cassidy & Andreou, 2012). This is a poor utilization of energy and veia, & Hamilton, 2009; La Rosa et al., 2005; Pearson et al., 2007;
a direct result of the limitations of two dimensional integration Rice, Bhuiyan, Taha, & Smith, 2009).
and the use of macroscopic components to interconnect chips. Research towards the engineering of custom large-scale digital
Optical interconnects, while efficient at distances measured in bio-inspired integrated circuits has also begun with encouraging
kilometers, have not been very helpful at short distances from a results. SpiNNaker is a System on a Chip (SoC), a massively-
power dissipation point of view; this may change in the years to parallel digital neuromorphic computing architecture (Khan et al.,
come with advances in silicon photonics and CMOS integration. In 2008) based on an 18 core symmetric chip-multiprocessor where
a two dimensional array of cells that is typical in neuromorphic each core is an ARM968. A SpiNNaker computer will consist of
electronic systems (feature maps), additional energy costs are a million microprocessor cores interconnected via a switching
accrued in going from the two dimensional representation of data network fabric. APIs and software have already been developed for
to a one dimensional stream on the periphery of the die for inter- the SpiNNaker system (APT-Group, 2011a,b,c), hence an excellent
chip transmission.
platform to explore bio-inspired algorithms and architectures
Without exception, all of the analog VLSI systems reported in
for cognitive computing. Another digital bio-inspired system
the literature, small or large, rely on substantial digital commu-
architecture for energy-aware cognitive computing is currently
nication infrastructures for functionality, ranging from single chip
being developed by IBM under the SyNAPSE project (Arthur et al.,
micro-controllers (Goldberg et al., 2001a,b), to arrays of FPGAs
2012; Merolla et al., 2011; Modha et al., 2011).
(Bruederle et al., 2011; Choudhary et al., 2012; Renaud et al., 2010;
Complementary to advances in large-scale hardware architec-
Vogelstein et al., 2007) to even single board computers (Fasnacht
tures have been the advances in large-scale software simulation
& Indiveri, 2011).
and modeling environments such as Brian (Goodman & Brette,
Unlike biological nervous systems, constrained by the limita-
tions of 2D CMOS technology, networks of electronic components 2009), Nengo (Anderson & Eliasmith, 2004; Eliasmith & Stewart,
such as switches, capacitors and short wires in VLSI integrated cir- 2011; Eliasmith et al., 2012), and Compass (Modha et al., 2011;
cuits are only weakly connected to each other i.e. each component Preissl et al., 2012). The close coupling of these software envi-
is connected to only a few other components, often in a pipeline ronments to the SpiNNaker architecture (Galluppi, Davies, Furber,
structure. Architectures of modern processors and memories are Stewart, & Eliasmith, 2012), Neurogrid (Choudhary et al., 2012),
designed specifically to meet the constraints of limited connectiv- FACETS wafer-scale system (Bruederle et al., 2011), and SyNAPSE
ity in modern two dimensional integrated circuits, which feature a project (Modha et al., 2011) is likely to facilitate the widespread ac-
dozen or so metallization layers. ceptance of custom architectures for large-scale simulations (Silver
We are now at a juncture point: an era where digital transistors et al., 2007) as an alternative to high performance computing (de
are nearly ‘‘free’’, and billion-transistor integrated Systems on a Garis, Shuo, Goertzel, & Ruiting, 2010; Markram, 2011).
Chip (SoC) are now commonplace. Capitalizing on the dramatic In this paper we address the challenging task of engineering
advances in the scaling of the MOS transistor, analog silicon silicon brains in the nano-CMOS era, with billions of transistors
retinas (Boahen & Andreou, 1992) and mixed-signal sensors and thousands of processors on a chip. We take a systematic
(Lichtsteiner et al., 2008; Posch et al., 2011) are being displaced approach, revisiting Marr’s three levels of description and argue
by all digital architectures. Sophisticated digital sensor arrays that the computational theory of parallel processing under
and computational sensors are currently being developed to physical constraints provides the theoretical underpinnings for
extract and quantify the subtle and intricate information from both understanding the brain and a new era of neuromorphic
natural scenes in visible and infra-red wavelengths (Lin, Pouliquen, engineering. We present a scalable neural architecture with
Andreou, Goldberg, & Rizk, 2012; Lin, Pouliquen, Goldberg, computationally efficient structures for spiking silicon neurons
Rizk, & Andreou, 2011). Cellular neural network architectures and learning synapses. Finally we employ a cost function based
and computational imagers have been reported in standard formulation of parallel processing under the physical constraints
digital CMOS (Federico, Mandolesi, Julian, & Andreou, 2008; of speed, energy, and area to optimize the neural architecture.
Mandolesi, Julian, & Andreou, 2004) as well as experimental Experimental verification of the computational structures for the
3D CMOS technologies (Mandolesi, Julian, & Andreou, 2006). optimization of the neural array, spiking neurons and Spike Timing
Stream processor architectures for convolutional neural networks Dependent Plasticity (STDP) learning is done using state-of-the-art
(ConvNets) have also been recently reported in the literature field programmable gate arrays (FPGAs).2
(Camuñas-Mesa et al., 2012; Pham et al., 2012). Digital FPGA based
bio-inspired architectures have also been reported in the literature,
2. Revisiting Marr’s vision: employing new eyes
for example recent work by Kestur et al., and references therein
(Kestur et al., 2012).
In this nano-CMOS era, the engineering of large-scale neuro- In the previous section, we have argued that our progress
morphic systems aimed at silicon cognition must also be carried towards synthetic brain like systems with cognitive capability
on at a different level of abstraction. Instead of using analog tran- matching human performance and energetic efficiency is seriously
sistors to emulate the biophysics of neurons, we must move to a hampered by difficulties in expressing brain-related functional
higher level of abstraction, using digital transistors to perform the system-level models and algorithmic constructs at an appropriate
arithmetic equivalent to the behavior of a neuron. Combined with
high-density digital memories and high-speed digital communica-
tions interconnects, this paradigm will enable the implementation 2 The Introduction in this paper is not meant to be a comprehensive review of
of large-scale, flexible silicon neural arrays. hardware implementations of neural systems over the last quarter century. The
State-of-the-art Field Programmable Gate Arrays (FPGAs) are interested reader is referred to a recent survey for a comprehensive overview (Misra
often at the forefront of technological advances and the ability to & Saha, 2010).
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 7

Computational
Theory Parallel Processing Under Physical Constraints

Brain Multiprocessor
Architectonics Architecture

Abstract
Network Fine-Grained
Computational Parallelism
Structures Computation

Probabilistic
Spikes Event Based
EPSP/IPSPc Information
Processing

Learning Synapses
LTDP/STDP Circuits that
Learn and Adapt
Physical
Computational
Laminar/
Structures Columnar
Organization Nano and 3D CMOS

Brains Silicon Brains

Fig. 1. Biological and silicon brains: layered levels of abstraction and information processing structures. Note that even though the abstract and physical layers are shown
in a hierarchy, they are done so only for the sake of clarity. In particular, the layered description for the physical computational structures in the brain are simplified in this
diagram. Churchland and Sejnowski offer a more accurate diagram for layers for the different physical scales in their seminal paper (Churchland & Sejnowski, 1988).

level of abstraction that can hide the details of their implementa- the Human Representation and Processing of Visual Information’’
tion. What we need is a multi-scale framework where information proposes a framework that has three levels of description to help us
processing structures at different layers in the two levels of ab- understand visual processing in biological systems and to engineer
straction have different interpretations of the processing element’s machines that see.
type, capability, and complexity. Any approach that relies on a sin- The three levels of organization as proposed by Marr are:
gle monolithic level of abstraction is bound to be slave to the un- • Computational theory: What is the goal of the computation and
derlying limitations imposed by that level. For example, on analog the logical strategy needed to carry it out?
neuromorphic chips, models of higher-level cognitive function can • Representation and algorithm: How can the computation be
only be expressed in terms of networks of analog neurons, with any implemented and what input/output representations are
other level of description is required to be implemented ad hoc on needed?
the host digital computer or a micro-controller. Thus the prevailing • Hardware implementation: What is the physical realization of
research approaches in neuromorphic engineering offer neither an the algorithm and the architecture?
effective basis for exploring system-level models of brain function,
We paraphrase Proust and argue that what we need is ‘‘new
nor a practical foundation for future brain-inspired cognitive com-
vision’’ and Marr’s vision on levels of organization (Marr, 1982),
puting technology.
provides the foundation for a fresh perspective on parallel
To address the challenges of engineering large-scale silicon processing, ‘‘the computer and the brain’’.
brains, we need a fresh perspective, a new view, one that sys- At the level of computational theory, we suggest that the
tematically allows to abstract brain computation into synthetic general theme of parallel processing under physical constraints could
structures. Quoting Marcel Proust, ‘‘Fundamental discoveries do not provide the theoretical foundation for understanding information
necessarily rely on exploring new landscapes, but on employing new processing in both the brain and in massively parallel computing
eyes’’. systems. Thus a principled approach that links the algorithmic
Conventional wisdom and the prevailing viewpoint in the aspects and costs of parallel processing to the constraints imposed
scientific community suggest that complex information processing by the physical hardware in the physical structure could provide a
systems, natural or synthetic, are best considered at multiple fundamental foundation for further understanding the brain and a
levels of organization. In the broadly defined task of ‘‘engineering powerful principle for engineering and architecting future brain-
silicon brains’’, we must have a consistent framework to address inspired computing machinery. This will be further elaborated in
scientific challenges and exploit technological opportunities at the following sections of this paper (Sections 3 and 6).
different levels of description (see Fig. 1). David Marr in the preface At the level of representation, in neural computation, the
of his seminal book ‘‘Vision: A Computational Investigation into temporal dynamics of spiking neurons encode information. Spikes
8 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

in biology or digital events in silicon systems can encode graded the two and delta (2D+delta) dimensional patches on the surface
(analog) signals in time while at the same time employing the of the cortex (V 1) (Das, 2000). An important insight into this
robustness of binary signaling. At this level we design abstract challenging problem emerged out of experimental work in the
computational structures optimized for minimum energy that visual cortex of the cat where it was shown that maps in the
exploit spike-event based representations to compute likelihoods visual cortex are optimized for uniform coverage (Stepanyants,
in graphical probabilistic models of inference and cognition. Such Hof, & Chklovskii, 2002; Swindale, Shoham, Grinvald, Bonhoeffer,
a probabilistic event based approach that was first proposed and & Hübener, 2000). Through a process of development and self-
used in engineered systems in (Goldberg et al., 2001a,b) provides organization, brains find an optimal solution in the presence
a principled description of event generation rules that maximize of two conflicting requirements: spatial coverage and stimulus
the information transfer, while limiting the number of energy representation. More specifically, to maximize coverage and
expensive events (spikes) (Laughlin & Sejnowski, 2003; Schreiber, parallel processing, every location in the physical space must be
Machens, Herz, & Laughlin, 2002), that need to be communicated mapped to all possible combinations of stimulus features. At the
between successive layers in a neural architecture. Using the same time, this must be done under constraints imposed by the
digital abstraction, neural computation can readily take on a physical structure; i.e. the wiring length of axons must be kept as
variety of computational models including the Leaky Integrate short as possible to minimize both the metabolic costs (energy)
and Fire and dynamical Izhikevich neural models, as elaborated and the time to respond (delay). The result is a smoothness of
in Section 4. In this work, models of learning take on the STDP mapping or locality of reference; i.e. neurons with similar stimulus
representation, a Hebbian learning rule, described in Section 5. response properties lie in close proximity on the cortical surface,
At the level of implementation, the physics and chemistry im- with local discontinuities arising from the multi-feature mapping.
posed by the underlying substrate define the constraints for the Similar self-organizing principles are found in the structural and
micro-architectural elements, whether real neurons in biological morphological organization of neurons in the brain, taking into
tissue or silicon neurons in nano-CMOS technologies. It is at this account both the functional and the underlying physical and
level of description that the information processing elements of bi- chemical constraints of the computational machinery (Chklovskii,
ological tissue are likely to differ substantially in form and function Mel, & Svoboda, 2004; Varshney, Sjöström, & Chklovskii, 2006;
from those implemented in nano-CMOS technology. At this level, Wen & Chklovskii, 2008; Wen, Stepanyants, Elston, Grosberg, &
attempts to draw analogies in the physical implementations are at Chklovskii, 2009).
best superficial. So long as we are aware of the fundamental dif-
ferences in the underlying substrates, we are assured to stay away
3.1. From brain architectonics to silicon neural architectures
from the perilous paths that impeded our progress towards the en-
gineering of large-scale silicon brains and cognitive machines over
the last two decades. Parallel processing silicon neural arrays must be optimized with
In Sections 4 and 5, we detail multiplexed digital circuits, stor- respect to two performance objectives: speed (inverse of delay)
ing the neural state in dense local memory and multiplexing neural and energy. Motivated by the need for a design methodology
computation on shared arithmetic logic units. This approach uses to address architectural space exploration in multi-scale parallel
the high frequency (relative to biological computation) of silicon architectures, we have derived a simple objective function (Cassidy
circuits to efficiently implement the massive parallelism required & Andreou, 2012) to link parallel processing in an information
for implementing large-scale neural computational architectures. processing system of N units, to the costs of energy and delay,
The proposed holistic approach towards the engineering of silicon the traditional metrics in integrated circuits (Mead & Conway,
brains forms the foundation for a new research direction in brain- 1979). The foundation of our model is a cost function formulation
inspired architectonics. It relies on a combination of multi-scale of Amdahl’s law (Amdahl, 1967), that employs parameterized
abstraction from algorithms to representation and the architec- models of computation and communication to represent the
ture. At each level, the computational structures rely on consistent characteristics of processors, memories, and communication
contracts between the levels of abstraction and the layers of de- networks. The interaction of these micro-architectural elements
scription. within a parallel processing framework defines global system
performance in terms of energy–delay cost.
Starting from Amdahl’s original intuitive argument, we can
3. Parallel processing under physical constraints
derive a generalized cost function that combines a delay cost
function and an energy cost function according to the energy–delay
Biological information processing systems employ dynamic
product, to obtain a generalized objective function JED that links the
matter and learning at all levels in an amazing network of
gains from parallel processing to delay and energy costs (Cassidy &
complex structures of different scales, from the nano to the
micro and macro. In the human brain, a physical structure with Andreou, 2012).
approximately 100 billion neurons and 100 trillion synapses, The equation below is an extension of the work in Cassidy and
parallel processing must be the norm rather than the exception. Andreou (2012) to address parallel processing in an asymmetric
Indeed parallel distributed processing is found at all levels of brain multiprocessor architecture such as the one necessary for brain-
function from molecules to networks to social behavior. inspired multiprocessor systems. Using a summation over the
Some insights into the brain’s organization and parallel processor types p of different performance, the objective function
processing function can be gained by considering the way the JED can be written as:
visual representation of the natural world is organized through
 
cortical maps. Early work by Hubel and Wiesel, suggested stimulus
 K −1
 
modalities are mapped in orthogonal dimensions: the ice-cube Fj 
JED =  −1 
model for stimulus representation in V 1 (Hubel & Wiesel, 1977). −1
M −1
 j=0 P
 
 
However, this simple and elegant idea does not scale beyond Njp Gijp Dijp
two stimulus features and into natural visual environments that p=0 i=0

involve a plethora of stimulus features. What is needed is a process  γ

K −1 P −1 M −1
 Fj   
by means of which multiple features such as orientation, spatial × Njhp Gijhp Eijhp . (1)
frequency, ocular dominance, and so on, can be mapped into j =0
NjA h∈{A,I } p=0 i =0
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 9

In the inner summations, each of the algorithm fractions Fj , are

subdivided into constituent cost components. Gijp is the fraction
of Fj that incurs the ith cost component Dijp or Eijh . Fj and Gij are
 K −1 M −1
fractions, such that j=0 Fj must equal 1 and i=0 Gijp must
equal 1. The ijth delay is Dij and the ijth energy cost is Eijh for the jth
fraction of the algorithm and the active or idle unit, h ∈ {A, I }. The
fraction Fj is divided by NjA units in duration, but multiplied by NjA +
NjI computational units operating in parallel. Njp is the number of
units of type P assigned to the jth fraction of the algorithm. The
innermost summation in the energy term is the average Energy-
Per-operation of a computational unit h ∈ {Active or Idle} :
 M −1
EPIph = i=0 Gihp Eihp .
In the outer summation of the delay term, Nj in the denominator
reflects the speedup in delay obtained by parallelizing the
algorithm over N units. In the middle summation of the energy
term, Njh is the number of active or idle units during the jth phase Fig. 2. Spiking neural array architecture.
of the algorithm. However, while delay is reduced by a factor of N,
energy expended is increased by a factor of N, since there are now
N computational units running in parallel. This results in the Nj in
the inner summation.
Adding an exponential weighting parameter γ to the energy
side of the equation allows energy and delay to be unequally
weighted. In the realm of energy efficient design, two metrics
are typically used for design evaluation: the energy–delay prod-
uct ED and energy–delay squared ED2 . The energy–delay product
equally weights the contribution of delay and energy, while en-
ergy–delay squared doubly weights the contribution of delay in or-
der to emphasize performance over energy savings. In our model,
using γ = 1 results in the standard energy–delay product, while Fig. 3. Multiplexed spiking neuron block diagram.
with γ = 0.5, the contribution of delay is twice as large as the con-
tribution of energy, analogous to the energy–delay squared metric. multiplexing data from multiple sources onto a single shared bus.
Efficient parallel computational architectures are developed Events traverse the architecture in Fig. 2 in a clockwise direction.
by minimizing this cost function in order to maximize the Each incoming event looks up a synaptic weight value. That weight
performance of the architecture. The cost function is minimized is sent to the appropriate neuron in the array based on the event
by: maximizing system parallelism and minimizing the energy address. When a neuron generates an event, it is tagged with the
and delay costs of operations within the constituent cores. In address of the generating neuron. The re-mapper translates the
Section 6 we will use the theoretical framework described in this generating address into a set of destination addresses, which are
section to explore the architectural space of spiking neural arrays sent back into the neuron array.
implemented in FPGAs. However, prior to that, we must define The high-level architecture is derived from our earlier work
the organization of the spiking neural array as well as the basic (Cassidy & Andreou, 2008; Cassidy et al., 2007). One of the differ-
computational structures for spiking neurons, the interconnect ences in the architecture presented in this paper is that the synap-
fabric and local learning machinery. tic weight RAM and the re-mapper RAM are both implemented in
SRAM external to the FPGA. The latter modification allows utiliza-
4. Spiking neural arrays tion of a much higher total RAM capacity as compared to internal
to the FPGA. In addition, this leverages the increasing density ben-
In this section we present an architecture for implementing efits of commercial SRAM technology. The targeted system in this
large-scale arrays of digital neurons. Although our architecture work is the Nallatech FSB accelerated computing platform (Nallat-
targets both FPGAs and standard cell ASICs in the current work, ech, 2008), with an FSB expansion module based on a Xilinx Virtex
we report results from commonly available high-end FPGAs that 5 SX240T FPGA (Xilinx, 2011) and two GSI Technologies 36 Mb QDR
have the advantage of reprogrammability. ASICs on the other SRAMs (GSI, 2011).
hand, have higher density and operational performance as well Each neuron in the array is a computational engine (or
as low power operation, with allocation of resources (RAM and core) composed of local SRAM and an arithmetic pipeline for
logic) custom to the task at hand. However, FPGAs are amenable computation. One key feature of the neuron, introduced in our
to reconfiguration hence suitable for the experimental work on earlier work, is neural state multiplexing (Cassidy & Andreou,
architecture exploration that is discussed in the next section. 2008). In this scheme, the computation for multiple neurons is
Furthermore, advances in software tools allow the compilation of multiplexed onto a single physical neuron. This requires two local
hardware design using high-level design languages (Cardoso et al., memories, one to store the state for the neurons that are not being
2010). currently computed (the state cache), and one to align input events
with the proper timeslot in the frame (the input aligner cache).
4.1. Spiking neural array architecture The multiplexed neuron block diagram is shown in Fig. 3. The
state cache is a dual port RAM so that the pre-computation neuron
A system-level block diagram of the spiking neuron array and state is read out of the cache at the same time as the post-
its interfaces is depicted in Fig. 2. Spike events enter and exit the computation neuron state is written back into the cache. The input
system from the left side of the diagram. Events are communicated aligner cache has two banks of dual port RAM. The two banks
using the AER (Address Event Representation) protocol, a means of implement a ping-pong buffer to decouple writing new events
10 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

Fig. 4. Frame processing example.

from reading out current events. New events are written into one 4.2.1. Izhikevich neuron model
bank while current events are read out from the other. Writing The dynamics of the Izhikevich spiking neuron model are de-
new events is a read–modify–write operation using both ports of fined by two coupled differential equations, and a reset condition.
the RAM bank in order to prevent events in the buffer from being The variable v represents the voltage across the neural membrane
overwritten. Reading out current events also uses both ports of the and u is a slow variable, representing membrane recovery.
RAM bank. The current value is read out of one port, while the other
port clears the RAM location (one clock cycle delayed) removing it v ′ = 0.04v 2 + 5v + 140 − u + I (2)
from the circular buffer. u = a(bv − u).
′
(3)
Frame-based processing of multiplexed neurons effectively
time multiplexes several neuron state computations onto a single The synaptic input is I and a, b are parameters controlling the dy-
neural arithmetic pipeline. Every physical neuron instantiated namical behavior of the neural model. The reset condition is de-
in the array computes the independent state of M multiplexed fined by:
neurons, where M is the frame size. Fig. 4 illustrates the frame
v

←c
processing paradigm for M = 8 multiplexed neurons. if v ≥ +30 mV, then (4)
The input state for frame T 0 is read out of the state cache
u ←u+d
and the input aligner simultaneously, and sent to the arithmetic where c , d are parameters controlling the neural reset behavior.
pipeline. The results from the arithmetic pipeline are available An example of the complex behavior achievable with the IZH
after a few clock cycles and are written back into the state model is tonic bursting. (Refer to Izhikevich, 2003, 2004 for 19
cache. All computations are fully pipelined, so that operations are other neural behaviors produced by the IZH model.) Fig. 5(a) shows
performed every clock cycle and there are no stalls in the datapath. the membrane voltage (‘v ’ parameter) for a burst of neural spikes.
Computation on frame T 1 begins exactly one cycle after frame The spike firing threshold is 30 mV, depicted by the green line.
T 0 enters the pipeline. The maximum frame size is constrained The dynamics of the ‘v ’ variable versus the ‘u’ variable is plotted
only by the on-chip memory required to store the neural state in Fig. 5(b), showing the same burst of spikes in v –u phase space.
variables. The frame size also affects the computational speedup Phase space is an effective method for visualizing the system
of the system, in an inversely proportional relationship. If the dynamics. The nullclines are shown in blue (‘v ’ is a parabola and
frame size doubles, the speedup is halved. In order to maintain ‘u’ a sloping line). The phase space diagram depicts the system
full-rate processing, the minimum frame size should be greater starting at t = 100 with a step input and ending at t = 200. From
than the pipeline depth (five clock cycles). The neuron blocks are the initial rest potential (v = −75 mV), the system moves in the
logically arranged in a linear array and the communication fabric positive ‘v ’ direction until it crosses the threshold (green vertical
is comprised of a tree arbiter which multiplexes events onto the line) and fires a spike. Upon firing a spike, the system is reset into
shared AER bus. an unstable region, again moving in the positive ‘v ’ direction, firing
another spike. As this bursting behavior continues, the ‘u’ variable
4.2. Dynamical spiking silicon neurons increases until the system is finally reset into a region that moves
the system away from the firing threshold (inside the parabola).
The design space of spiking neuron models spans many decades The system will slowly move along the ‘v ’ nullcline, until it is
of research, beginning with Hodgkin–Huxley’s pioneering work in once again swept into the firing region, generating tonic bursting
1952 (Hodgkin & Huxley, 1952). Their detailed approach models behavior.
neuron behavior using four differential equations to represent the The first implementation of the IZH model in an FPGA was
membrane dynamics and the non-linear conductances of three reported in La Rosa et al. (2005). From the little detail reported
types of ion channels. While this detailed approach produces a in the publication describing the work, a single neuron was
biophysically accurate model, it is computationally intensive and implemented, running at 1 MHz. This is far from sufficient for
requires the estimation of a large number of free parameters. large-scale neural simulation acceleration or scalable neural array
Since then, numerous models have been made in order to reduce implementation. Another design uses the Izhikevich model for
the complexity of the model. Typically however, reducing the inspiration, implementing a similar dynamical neuron in analog
model complexity also reduces the biophysical accuracy and VLSI (Wijekoon & Dudek, 2006). The fast and slow state variable
the number of neural behaviors that can be reproduced by paradigm is employed (analogous to ‘v ’ and ‘u’ in IZH), creating a
the model. The Leaky Integrate and Fire (LIF) neural model is neuron that exhibits oscillatory and bursting behavior. However,
popular because of its relative simplicity, however, it cannot this well principled design suffers from the disadvantages common
reproduce many complex neural behaviors. Izhikevich reviewed to all analog VLSI approaches (see Section 1).
ten models including Hodgkin–Huxley and LIF, comparing them We implemented the IZH model using fixed point arithmetic.
with his own approach (Izhikevich, 2003, 2004). His simplified Numeric values were constrained to an 18-bit representation,
approach (hereafter denoted as ‘‘IZH’’), contains only two coupled based on the 18 × 18 fixed point multiplier cores in the FPGAs.
differential equations and yet is able to reproduce many complex We choose a 10.8 fixed point representation as a balance between
neural behaviors. We adopt the IZH model as a computationally dynamic range (10-bit integer portion) and precision (8-bit
efficient model for neurons requiring complex dynamical behavior. fractional portion).
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 11

tonic burst -- membrane voltage

40
50

20
0
v

u
0
-50

-20
-100
80 100 120 140 160 -100 -50 0 50
time (ms) v
(a) Tonic burst—membrane voltage. (b) v –u phase space.

Fig. 5. Dynamical system description for the Izhikevich neuron model. (For interpretation of the references to colour in this figure legend, the reader is referred to the web
version of this article.)

Table 1
Device utilization: Xilinx Spartan XC3S1500.
Resource Percent utilization (%) Total available

Slice FF’s: 64 26,624

4-LUTs: 78 26,624
2 kB RAMs: 34 32
18 × 18 mults: 100 32

directly through the tree from input operands to resulting output.

The arithmetic trees maximize the parallelism in time (pipelining)
and space (parallel arithmetic units). The computations in each
Fig. 6. IZH neuron block diagram.
arithmetic tree are fully pipelined to support full-rate dataflow
processing. The pipeline is kept full by the frame based processing
In addition, significant implementation advantages can be of blocks of multiplexed neuron state.
gained if powers of two arithmetic can be used for multiplication We made three specific optimizations for the algorithm. First,
and division. With this motivation, we modified Eq. (2) by the constant coefficients 4 and 32 1
in Eq. (5) are implemented as
multiplying the coefficients by 0.78125, for an approximate static shift operations (2’s complement arithmetic). Second, since
powers of two representation of two coefficients in the equation, multipliers are a scarce resource in FPGAs, the multiplication of
(0.78125 × 0.04 = 1/32 and 0.78125 × 5 = 3.91 ≈ 4). the parameter ‘a’ in Eq. (6) is implemented using a shift and
1 2 add/subtract operation. This limits the resolution of the values that
v[n + 1] = v[n] + v [n] + 4v[n] ‘a’ can take on, however, it is efficiently implemented in logic.
32
+ 109.375 − u[n] + I [n] (5) Third, the multiplication operations in both pipelines (Eqs. (5) and
(6)) share the same physical multiplier, via time multiplexing. As
u[n + 1] = u[n] + a(bv[n] − u[n]). (6) a result, the net throughput of the pipelines is halved (i.e. if the
These equations model the same system behavior as Eqs. (2) and FPGA runs at 80 MHz, new results are generated by the pipelines
(3) above, however, the units have effectively been modified. In every 40 MHz). This tradeoff is also made in order to optimize the
addition, the constants a, b, c , d must also be slightly modified utilization of the scarce multiplier resource.
from the values of the original IZH model. Lastly, the differential We implemented the silicon spiking neural array in a Xilinx
equations are implemented in discrete time. Spartan XC3S1500 FPGA (Xilinx, 2011), hosted on an Opal Kelly
Using this approach, we created a multiplexed neuron as a XEM-3010 FPGA integration module (Opal-Kelly, 2012). The
base element for creating large-scale neural arrays. The physical integration module has a USB 2.0 interface to a host PC. High-level
neuron implements the arithmetic computations required for the control and interface to the design is through MatLab or Visual
IZH neural model, while local memories buffer the state of multiple C++.
neurons between operations, in a time-division multiplexed The device utilization is summarized in Table 1, and is roughly
manner. A block diagram of an individual physical IZH neuron is balanced for each resource (with the exception of 2 kB RAMs). Ul-
shown in Fig. 6. The neuron state computations are performed timately, the number of physical neurons that can be instantiated
in the ‘v ’ and ‘u’ arithmetic pipelines. Frame-based processing of is limited by the number of 18 × 18 multiplier cores available on
multiplexed neurons is supported by the two dual port memories, the device. The number of multiplexed neurons that can be multi-
‘‘v -store’’ and ‘‘u-store’’, as well as the input alignment block. The plexed onto each physical neuron is limited by the amount of dis-
dual port memories buffer the multiplexed neuron state between tributed memory available (LUT resource).
frame iterations. The input alignment block aligns asynchronous In the XC3S1500 FPGA, we have 32 physical neurons with 8
input events with their proper timeslot in the frame. multiplexed neurons each, for a total of 256 neurons in the array.
The IZH neuron behavior described in Eqs. (4)–(6) is imple- With a state of the art FPGA, the number of neurons can be
mented using two parallel arithmetic pipelines, one for the ‘v ’ dy- increased by at least two orders of magnitude. A current generation
namics, and one for the ‘u’ dynamics. These pipelines are shown Xilinx XC7VX980 FPGA has 3600 multipliers and 54 Mb of internal
in detail in Fig. 7. The arithmetic operations in Eqs. (5) and (6) SRAM. Including multiplexed neurons, these devices enable on the
are assigned to arithmetic functional units and arranged accord- order of one million independent dynamical neurons per chip (see
ing to the standard algebraic order of operations. The data flows Section 6).
12 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

(a) ‘v ’ pipeline. (b) ‘u’ pipeline.

Fig. 7. IZH neuron arithmetic pipelines.

tonic spiking
tonic bursting
50
50
membrance potential (V)

membrance potential (V)

0 0

-50 -50

-100 -100
0 100 200 300 400 0 100 200 300 400
time (ms) time (ms)
(a) Tonic spiking. (b) Tonic bursting.

phasic spiking phasic bursting

50 50
membrance potential (V)
membrance potential (V)

0 0

-50 -50

-100 -100
0 100 200 300 400 0 100 200 300 400
time (ms) time (ms)
(c) Phasic spiking. (d) Phasic bursting.

Fig. 8. Dynamical IZH neuron behavior.

Although the IZH neural array operates at 80 MHz in the Xilinx Table 2
Fixed point model parameters (TS = Tonic Spiking, TB = Tonic Bursting, PS = Phasic
Spartan XC3S1500 FPGA, the multipliers are time multiplexed
Spiking, PB = Phasic Bursting).
between the ‘v ’ and ‘u’ arithmetic pipelines, so that the effective
processing rate is 40 MHz. A 40 MHz clock period is 25 ns, far faster a b c d I

than the neurobiological timescale. Since multiplexed neurons are TS: 1

64
0.156250 −50.508 6.2500 10.9375
processed in frames, it takes 8 clock cycles or 200 ns to process TB: 1
64
0.234375 −39.063 3.9062 0.58594
one frame of 8 neurons. Then, assuming a biologically realistic PS: 1
64
0.273438 −50.508 6.2500 11.7188
simulation timescale of 1 ms per clock cycle, our silicon neural PB: 1
64
0.273438 −42.969 1.1719 0.78125
array simulates spiking neural networks 5000 times faster than
biological real time.
The IZH neural array was designed and simulated in VHDL, bursting generate activity only at the onset of the step input. The
prior to mapping onto the FPGA. Cycle and bit accurate simulations parameters for the four test cases are shown in Table 2. They were
of the VHDL (ModelSim) capture the system functionality. Fig. 8 obtained from (Izhikevich, 2004) and modified for our fixed point
depicts the dynamical behavior of a single IZH neuron in four implementation.
different test cases, tonic spiking, tonic bursting, phasic spiking, An estimate of the computational power of the IZH neural ar-
and phasic bursting. All four test cases are in response to a step ray is as follows. Eqs. (4)–(6) are implemented with 16 fixed point
input (at 100 ms). Tonic spiking and bursting have persistent arithmetic operations: 2 multiplications, 9 additions/subtractions,
activity for the duration of the step input, while phasic spiking and 1 compare operation, and 4 shift operations. All of the operations
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 13

are fully pipelined and parallel. Thus running at 40 MHz, each

pipeline pair computes: 16 × 40 MHz = 640 MOPS. There are 32
pipelines in the array, so the FPGA is performing: 32 × 640 M =
20.48 GOPS. This is substantial performance for a medium sized
FPGA. Also note that 20.48 GOPs is sustained performance—not
maximum performance as is reported by many performance mea-
sures. The FPGA sustains 20.48 GOPS of computation, guaranteed
by full data flow processing. This computational measure also only
includes the arithmetic computations in the design. The routing,
spike generation, and other array operations are performed en-
tirely in parallel on the FPGA, but require additional computational Fig. 9. LIF neuron block diagram.
cycles if run on a microprocessor.
Table 3
4.2.2. Leaky integrate and fire neuron model (LIF) Device utilization: Xilinx Spartan XC3S1500.
The membrane voltage dynamics of the LIF neuron model is Resource Percent utilization (%) Total available
defined by the following equations. The equation for synaptic Slice FF’s: 28 26,624
integration is: 4-LUTs: 44 26,624
2 kB RAMs: 34 32
Ns

vs (n) = wi xi (n) (7)
i =1 The programmable relative refractory value increases the spike
where vs is the synaptic input contribution to the membrane threshold for time tabs_r , decreasing the probability of firing
potential, Ns is the number of synapses, wi are the synaptic weights, another spike. Another programmable value sets the exponential
(positive wi for excitatory synapses, and negative wi for inhibitory decay of the accumulator, emulating a shunting leak current in the
synapses), and xi (n) ∈ {0, 1} denotes the arrival of a presynaptic neural membrane.
spike on input i at time n. The neuron firing dynamics are defined We implemented the silicon spiking neural array in a Xilinx
by: Spartan XC3S1500 FPGA (Xilinx, 2011), hosted on an Opal Kelly
XEM-3010 FPGA integration module (Opal-Kelly, 2012). The initial
vrst if vs (n) > vth

design operates at 50 MHz. It was implemented in approximately
v(n) = or to < tabs_r (8) 4000 lines of VHDL, requiring only 3 weeks of design and debug
v(n − 1) + vs (n) − vL (τ ) otherwise time. The device utilization is summarized in Table 3.
where v(n) is the membrane potential, v(n − 1) is the membrane A modular approach has led to subsequent redesigns with
potential at the previous time instant, vrst is the reset potential, improved clock frequency to 100 MHz with only a few days
vth is the threshold potential, vL (τ ) is the exponentially decreasing of work. The 100 MHz clock period is 10 ns, far faster than
leak voltage with time constant τ , to is the time since the last the biological neural spike timescale. If we assume a biological
output spike event, and tabs_r is the absolute refractory period. The simulation timescale of 1 ms per clock, then we can simulate
relative refractory function is defined as: neural networks at 100,000 times faster than real time. Or, using
an appropriate multiplexing scheme, the speedup could be used
vth = vth + vrel_r (to ) (9)
to simulate a greater number of neurons on a biological timescale.
where vrel_r is an additional potential added to the threshold This tradeoff as well as the scaling of this architecture to truly large-
potential. This models the decreased probability of firing a spike scale neuromorphic systems is detailed extensively in Section 6.
shortly after an output spike has been generated. vrel_r is set to a To compare the performance of our architecture with a general
constant at the time of the output spike event and then decreases purpose processor, we created a simple simulation modeling
linearly with time until it reaches zero. spatio-temporal receptive field convolutions in the auditory cortex
Using the same neural array architecture, we can substitute using spiking neurons. The task consisted of 32 integrate and
in different neuron models, changing the biophysical-level of fire neurons with 16 synapses each, computing the convolutions
emulation as well as the implementation complexity. The LIF over 100,000 timesteps. The average input spike rate per neuron
neuron is a simplification over the IZH model. Its reduced was 70.4 spikes per second and average output spike rate of 11.1
complexity of arithmetic implementation and single state variable spikes per second. We coded the simulation in low-level C for the
means that area to implement each neuron (physical and
processor, and we also ran the exact same task on our FPGA array
multiplexed) is minimized, and the dependence on the FPGA
of 32 neurons with the same parameters and input dataset.
multiplier is also eliminated. This enables the overall neural array
The single core 2.13 GHz Pentium 4 processor ran this
to scale to even greater neuron densities.
simulation in 45.5 ms (mean of 8 trials). The array of 32 FPGA
As shown in the LIF neuron block diagram (Fig. 9), there are
integrate and fire neurons, running at 100 MHz, computes the
three primary operations in the arithmetic pipeline, an exponential
same task in exactly 1.0 ms, a 45× speedup over the single core,
leak on the membrane potential, synaptic integration, and firing.
The fire operation compares the computed value of the membrane general purpose processor. This speedup is due to the parallelism
potential with a threshold. If the value is above the threshold, in the neural array (N = 32) as well as the special purpose micro-
a spike event is generated, and the membrane potential is reset architecture of the neural engines. Every operation is pipelined,
to vrst . including control, datapath arithmetic, and memory operations.
A block diagram of an individual physical LIF neuron is shown The neural architecture and analysis presented in this section
in Fig. 9. The membrane potential of the integrate and fire also supports other neural models, such as the Izhikevich or Mi-
neuron is implemented as a 16-bit digital accumulator. When the halas–Niebur (Mihalas & Niebur, 2009) neurons, by substituting
accumulator exceeds a programmable threshold, a spike is output into the arithmetic pipeline the appropriate computations. With-
from the block and the accumulator is reset. The accumulator out loss of generality, in the remainder of the architecture-level
begins integrating again after the absolute refractory period. discussion, we use the LIF neural model.
14 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

5. Computational structures for STDP learning

Endowing large-scale neuromorphic systems with integrated

learning enables them to adopt new behaviors or adapt to new
stimuli. Furthermore, this opens possibilities for developmental
approaches where the system evolves to optimally solve problems
under relevant constraints. Thus, considerable research effort is di-
rected towards developing learning capabilities for neuromorphic
and other synthetic intelligent systems.
One learning method that has garnered substantial interest
recently is Spike Timing Dependent Plasticity (STDP) (Bi & Poo,
1998; Markram, Gerstner, & Sjöström, 2011; Song & Abbott, 2000),
a biologically-based, Hebbian reinforcement learning rule. This Fig. 10. General STDP modification function.
particular learning rule is attractive from a hardware perspec-
tive because it exploits the robustness of discrete value (digital)
representations employed in long distance communication while
preserving analog information in the time dimension. In this
paradigm, inputs that contribute to a neuron firing are strength-
ened, while inputs that do not contribute are weakened. Contribu-
tion to firing is determined by the time of an incident (presynaptic)
spike relative to the time of the neuron firing a (postsynaptic)
spike. Indeed, many approaches have adopted STDP, using analog
long-term storage (Bartolozzi & Indiveri, 2007; Bofill-i Petit & Mur-
ray, 2004; Chicca et al., 2003; Indiveri et al., 2004; Indiveri, Chicca,
& Douglas, 2006; Koickal et al., 2009; Schemmel, Grubl, Meier, &
Mueller, 2006) and floating gate (Liu & Mockel, 2008; Ramakrish-
nan, Hasler, & Gordon, 2012) circuits. The recent work by Bamford,
Murray, and Willshaw (2012) provides an excellent overview in
the state-of-the-art for analog hardware implementations. Digital
implementation of the STDP rule has also been developed and re- Fig. 11. Block diagram: baseline digital STDP.
ported in the literature (Belhadj et al., 2009; Cassidy, Andreou, &
Georgiou, 2011; Cassidy et al., 2007). spike times are stored in a circular buffer along with the synaptic
A key characteristics to large-scale learning is the scalability index. When the neuron generates a postsynaptic spike, the time
of the learning circuit. The size and complexity of a single of the postsynaptic spike is compared with the stored presynaptic
synaptic learning circuit is multiplied by the number of learning spike times. The time difference is used to address the look-
synapses in the system. In a system with 103 –106 neurons and up table (LUT) storing the STDP modification function (Fig. 10).
2–3 orders of magnitude more synapses, the silicon area required The LUT outputs 1w , the value to modify the synaptic weight.
to implement learning can be significant. We show that two key A ‘read–modify–write’ block receives the 1w and modifies the
innovations enable scalable learning circuits: multiplexing of the appropriate synaptic weight based on the index stored in the
learning circuit (Cassidy et al., 2007) and low complexity of the circular buffer.
learning circuit (Cassidy, Andreou et al., 2011). Naturally, the best
improvement in size and complexity over baseline performance is 5.3. Low-complexity learning circuits
obtained by using both techniques together, as shown here.
A low-complexity digital implementation of the STDP learning
5.1. The STDP learning rule rule begins with the observation that weight update functions
can be built from the convolution of digital signals. Fig. 12 shows
the simplest case, a convolution of a rectangle function with a
The STDP learning rule is an unsupervised method that updates
single pulse (the synchronous digital equivalent to an impulse
synaptic weights based on the time of a presynaptic input spike
function). By combining the output of multiple rectangle-impulse
relative to the time that an output spike event is generated.
convolutions, the STDP function can take on many shapes of
The concept is that synapses that contribute to the generation
varying amplitude, width, and curvature, as shown for example
of an output spike event should be strengthened, while non-
in Fig. 14. The convolution of rectangle functions to create a
contributing synapses (i.e., those whose input spikes occur after
triangular STDP function is shown in Fig. 16. In this case, the
the output spike is generated) should be weakened. The STDP
convolution of two rectangle functions creates a pyramid shape.
modification function (Fig. 10), shows the change in synaptic
The pyramid must be bisected in the center in order to create
weight based on the relative arrival time of a presynaptic input
the negative weight update. This bisection is accomplished by
spike on a particular synapse. The output spike is generated at time
selecting between the increment and decrement convolutions,
zero in the plot. If a presynaptic spike arrives just before an output
depending on whether the presynaptic spike occurs prior to or
spike is generated, then the weight is increased by the amount 1w .
following the postsynaptic spike.
If a presynaptic spike arrives just after an output spike is generated,
The STDP learning rule updates synaptic weights based on
then the weight is decreased. the relative spike timing of presynaptic and postsynaptic spikes.
Implementing this learning rule in silicon requires measurement
5.2. Multiplexed learning circuits of relative spike timing and a feedback path to modify the synaptic
weights.
The functional block diagram of a multiplexed implementation Our minimum complexity implementation encodes this update
of digital STDP is shown in Fig. 11. In this approach, the presynaptic function using combinational digital logic. Fig. 13 gives an
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 15

Fig. 15. Block diagram: STDP combinational encoding—II.

Fig. 12. Timing diagram: STDP modification function—I.

Fig. 16. Timing diagram: STDP modification function—III.

postsynaptic spike is also sent to the STDP encoding block. Inside

the STDP encoding block, a shift register and two OR gates create
Fig. 13. Block diagram: minimum complexity digital STDP, encoding—I.
rectangle functions for increment and decrement. The postsynaptic
spike is a single cycle pulse. The rectangle functions and the pulse
are convolved using an AND gate, creating the STDP modification
function shown in Fig. 12.
More complex STDP modifications, as depicted in Figs. 14 and
16, are created by changing the combinational logic and shift reg-
isters in the STDP encoding block. Fig. 15 shows the combinational
encoding block for the STDP modification function shown in Fig. 14.
This block directly replaces the block in Fig. 13 labeled ‘‘STDP fx
combinational encoding’’. In this case, a second, smaller rectan-
gle function is created by replicating the OR gates that aggregate
the shift register values. We combine the two increment signals to
form a pulse one or two cycles wide (depending on whether one or
two rectangle convolutions are non-zero). Essentially we are pulse
width encoding the amplitude of the STDP modification (1w ). The
pulse width encoded signal is decoded by the up/down counter
which increments (or decrements) once every clock cycle that the
increment (or decrement) signal is asserted.
The third approach convolves rectangle functions as shown
in Figs. 16 and 17. The rectangle function for the postsynaptic
Fig. 14. Timing diagram: STDP modification function—II.
spike is also created using a shift register-OR gate combination. An
additional function, the I/D sel block, selects between incrementing
overview of a complete neuron including synapses and learning. and decrementing the synaptic weight based on the binary
In this implementation, synaptic weights are stored using a signed decision of whether the presynaptic spike came before or after
binary up/down counter capable of representing both excitatory the postsynaptic spike. This decision is also encoded using basic
and inhibitory synapses. (The learning circuit can also be used with combinational logic. We contrast these low-complexity STDP
more traditional RAM-based synapses by substituting a RAM with implementations with a baseline implementation reported earlier
the synaptic weights and a read–modify–write operation instead of in Cassidy et al. (2007).
the up/down counter.) When a presynaptic spike arrives, the value
of the synaptic weight is sent to the digital neuron. In addition, 5.4. Extension to multiple synapses
the presynaptic spike is also sent to a shift register in the STDP
encoding block. The neuron integrates the synaptic input, and if The small, low-complexity STDP circuits presented thus far
it exceeds its firing threshold, it emits a postsynaptic spike. The have been for a single synapse–neuron pair. With a modest amount
16 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

Table 4
Utilization: Xilinx Spartan XC3S1500.
Approach Slice FF’s 4-LUTs

Baseline 159 371

Comb. encoding I 35 14
Comb. encoding II 37 17
Comb. encoding III 39 16

Comb. encoding I—FIFO, 1 46 24

Comb. encoding I—FIFO, 2 48 26
Comb. encoding I—FIFO, 3 56 42

Fig. 17. Block diagram: STDP combinational encoding—III.

Fig. 20. Synaptic weight distribution histogram after STDP learning. Baseline
system, 32 synapses. Single trial.

improvement. For additional comparison, the approaches given in

Belhadj et al. (2009) require block RAMs, embedded multipliers, as
Fig. 18. Multiple synapse extension, first order approach.
well as 28× to 480× more LUTs than our largest approach (Comb.
Encoding I—FIFO, 3).

5.5. Experiments and results

We tested the performance of the STDP learning algorithm

by performing a balanced excitation experiment, based on the
experiment run by Song et al. (see Figs. 2a and 2b in Song & Abbott,
2000). In this experiment, 32 synapses from a single neuron start
with a uniform positive weight distribution. Each synapse is driven
by an independent Poisson spike train input with the same average
rate. When STDP is enabled, the synapses converge to a steady state
condition with a bimodal distribution of excitatory and inhibitory
weights.
Fig. 19. Multiple synapse extension, second and third order approaches. The baseline implementation replicated the bimodal distribu-
tion as shown in Fig. 20. The results from the combinational en-
of additional circuitry, one STDP circuit can be used by all of coding approaches are shown in Figs. 21–23. Each histogram plot
the synapses in a neuron, by storing the synapse address of the shows the synaptic weight distribution after STDP learning. The
presynaptic spike and then sending it out with the increment or plots are the aggregate results of eight trials using different stim-
decrement update signals. This first order approach using a single uli for each trial. It is apparent that even the simple scheme (ap-
FIFO is shown in Fig. 18. However, this approach only updates proach I) is capable of producing the expected bimodal distribution
one synapse for each postsynaptic event. By using two FIFOs, as (Fig. 21). However, it is also apparent that approach III produces
shown in Fig. 19, up to two synapses per postsynaptic event can the best3 shaped distribution, which is unsurprising given the best
be updated, one increment and one decrement. By also adding modification function shape shown in Fig. 16.
counters that flush all of the addresses in the FIFOs, this can be The STDP functions for this experiment are not symmetric, as is
extended to update an arbitrary number of synapses for each depicted in Figs. 12, 14, 16. Instead, the widths of the increment and
postsynaptic event. decrement shift registers (rectangle functions in the convolution)
are given in Table 5. These parameters were used to obtain the
We synthesized each of these STDP approaches, targeting a
results shown for this experiment.
Xilinx Spartan XC3S1500 FPGA (Xilinx, 2011). For comparing area
Finally, we use the STDP learning rule implementation to model
results, each approach uses a time window of 32 clock cycles,
the effects of ocular dominance during cortical column formation
16 prior and 16 following the postsynaptic spike. The resource
in the visual cortex. This experiment reproduces the results of
utilization for the single synapse circuit, as well as the multiple
Kanold and Shatz (2006) while employing silicon spiking neurons,
synapse extension, are summarized in Table 4. The low-complexity
synapses, and STDP learning implemented in the FPGA. Fig. 24
STDP approaches are over 4× smaller than the baseline approach
in flip-flops and over 20× smaller in terms of LUTs. The multiple
synapse extension using FIFOs uses additional resources, but is still
2.8× smaller in flip-flops and 8.8× smaller in LUTs. Synthesizing 3 ‘‘Best’’ refers to the distribution closest to Fig. 2b in Song and Abbott (2000), the
the design for a 130 nm CMOS ASIC technology shows similar area experiment that we are replicating.
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 17

Fig. 24. Ocular dominance experiment: cortical column.

Fig. 21. Synaptic weight distribution histogram after STDP learning. Combinational
encoding I system, 32 synapses.

Fig. 25. Cortical column case 3.

column growth. Specifically, there are four cases for different input
conditions:
Fig. 22. Synaptic weight distribution histogram after STDP learning. Combinational 1. No subplate, th0 > th1: thalamic inputs (th0, th1) die away.
encoding II system, 32 synapses.
2. No subplate, th0 = th1: thalamic inputs (th0, th1) die away.
3. Subplate, th0 > th1: th0 strengthened, th1 and subplate die
away.
4. Subplate, th0 = th1: th0 or th1 strengthened, th1 or th0 and
subplate die away.
Given a proper single set of system parameters (STDP τ , spike fire
thresholds, leak current, and relative refractory period), all four
cases can be met depending on the particular input conditions
present. For example, results for case 3 are shown in Fig. 25. In
this case, the input from th0 is stronger than th1, resulting in a
strengthening of the L4-0 synaptic weight and a weakening of the
L4-1 synaptic weight. As the L4-0 and L4-1 synapses differentiate,
the subplate neuron synapse becomes less important and goes to
zero. Results for the other three cases are similar.

Fig. 23. Synaptic weight distribution histogram after STDP learning. Combinational
6. Neural architecture optimization
encoding III system, 32 synapses.

Table 5 Now given the preceding representations and implementations,

Balanced excitation experiment parameters. we return to Marr’s level of computational theory. Given the phys-
Approach Dec. width Inc. width ical constraints of delay, energy, and area, how can we maximize
the performance of these parallel computational architectures?
Comb. encoding I 16 3
Comb. encoding II-L1 16 4 In this section, we perform a constrained optimization in order
Comb. encoding II-L2 8 0 to find optimal architectural parameters. Our goal is to find the
Comb. encoding III 16 8 optimal number of physical neurons (processors), as well as the
allocation of processor and memory resources (expressed in units
depicts the cortical column formation scenario. There are two of area) given a fixed total area on the die and the finite communi-
neurons, a layer 4 neuron N0 and a subplate neuron N1. There cation bandwidths. Our analysis follows the approach we outlined
are two thalamic inputs th0 and th1, one originating from each in earlier work for traditional chip-multiprocessors (Cassidy & An-
eye. These inputs connect to the subplate neuron as well as the dreou, 2009, 2012). In the present work, processors are the neuron
layer 4 neuron. In addition to the subplate neuron connection, the engines and memory is the internal RAM (state cache and input
layer 4 neuron also receives spontaneous input. It is hypothesized aligner cache). Without loss of generality, throughout this work we
that the subplate neuron plays an important role during cortical employ the LIF neural model.
18 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

few hundred MHz. In this work, we use a 200 MHz clock in a Xilinx
Virtex 5 FPGA. The first-order speedup over real-time is:
CLK 200 MHz
xRT = = (14)
NCR 1 kHz
where NCR is the neural computation rate. NCR is the inverse
of the neural simulation timestep (i.e. for a 1 ms simulation
timestep, NCR = 1 kHz). By multiplexing neural state on physical
neurons, we tradeoff speedup over real-time for system area.
In the multiplexing case, the speedup over real-time is (using a
multiplexed frame length of FL = 1024):
     
CLK 1 200 MHz 1
xRT = = . (15)
FL NCR 1024 1 kHz
Thus, the maximum frame length FL in order to operate at or above
real-time is 200,000 given a neural computation rate NCR of 1 kHz.
Fig. 26. Frame length FL vs. neuron engines N.

6.1.3. External RAM sizing

6.1. Physical constraints The size of the synapse RAM is:

Using the architecture outlined in Fig. 2, we begin the discussion SYNP_RAM (Bytes) = W · P · Q (16)
of physical constraints, examining the tradeoffs between area and where P number of synapses per neurons and W is the synaptic
delay. weight size defined above. This external RAM is the dominant
factor in determining the total number of synapses in the system,
6.1.1. Physical vs. multiplexed neurons as well as the number of synapses per neuron. The size of the re-
The total number of neurons in the system (Q ) is the product mapper RAM is a function of fanout FO (destinations per neuron):
of the number of parallel physical neuron engines (N) and the 1
frame length (FL) or number of neurons multiplexed onto a single REMAP_RAM (Bytes) = log2 (PQ ) · FO · Q . (17)
8
physical neuron engine,
Two factors determine the maximum fanout supported by the
Q = N · FL. (10) system. The first is the size of this external RAM, which determines
the maximum number of destinations that can be stored. The
Thus, for a fixed number of total neurons Q , there are many
second is the communication bandwidth which limits the number
combinations of N and FL that will result in the desired total neuron
of destination events that can be replicated before saturating the
count. This tradeoff is shown in Fig. 26. In the neural architecture
system.
presented here, there are two types of internal RAM, the state cache
In the discussion above, we have assumed that both the synapse
and the input aligner cache. The state RAM size is:
RAM and the re-mapper RAM assume a uniform number of
STATE_RAM (Bytes) = S · FL (11) synapses or destinations per neuron. This restriction can be re-
moved with the addition of a second lookup stage. This additional
where S is the size of one neuron state in Bytes. The input lookup would return a pointer into synapse or re-mapper memory
alignment RAM size is: as well as a value specifying the number of locations to read beyond
IN_ALIGN_RAM (Bytes) = 2W · FL (12) the pointer. This would allow an arbitrary number of synapses or
destinations to be used on a per-neuron basis. The cost is the area
where W is the size of one synaptic weight in Bytes and the factor of a second bank of RAM (of size proportional to Q ) as well as the
of two accounts for the dual ping-pong banks of the input aligner. delay cost of the second table lookup.
The total internal RAM is:
INT_RAM (Bytes) = N · FL(S + 2W ). (13) 6.1.4. Resource allocation analysis
We account for the resources in the FPGA by using the effective
Thus we will have a constant total internal RAM if N · FL is held area of each resource. Modern integrated circuits, including FPGAs,
constant. For example, increasing the number of neuron engines are limited by circuits and wiring that exist in a two dimensional
N by two while decreasing the frame length FL by two, keeps surface, the area of the die. Hence it is sensible to use the area as
the total number of neurons Q and total internal RAM constant. the basic constraint for optimization process.
On the other hand, there is a distinct tradeoff between N and We begin our analysis by examining the area breakdown of the
FL while holding Q constant. Larger values of N (and smaller FL) system. The area of a neuron is:
results in more parallel computational engines, which increases
Anrn = ARAM + Alogic . (18)
performance (decreases computational delay). However, larger
values of FL (and smaller N) take up less area since the logic area The logic area per neuron is a constant Alogic , while the RAM area is
(not RAM) is proportional to N and independent of FL. These area ARAM = FL(S + 2W ), from (13). Using Amem = (S + 2W ), the area
relationships will be revisited in Section 6.2. of the neural array is:
Aarray = NAnrn = N (ARAM + Alogic )
6.1.2. Real-time speedup
Biological neural firing rates range from zero Hz to a few
= N (Amem · FL + Alogic ) (19)
hundred Hz. A computational rate of 1 kHz is sufficient to capture where N is the number of physical neuron engines.
the internal neural dynamics between firing events. In contrast, In the array, the input multiplexor scales proportional to N,
clock rates of standard digital FPGAs and standard cells ASICs are a while the output multiplexor is implemented as a binary tree.
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 19

There are log2 (N ) levels in a binary tree, and a total of N − 1 nodes. where N is the number of neuron engines, G0 is the average fraction
Thus, since both scale proportionally to N, they are incorporated of time the neuron does not spike, D0 is the time to compute one
into the Alogic term of the neuron engine. frame of the neural state for the whole system (Q neurons), G1 is
The total available area on the silicon die Atot is: the average fraction of time the neuron spikes and D1 is the time
to compute neural state plus the communication delay. Note that
Atot = Afix + Aarray
the ‘‘algorithm’’ is perfectly parallelizable. That is to say that the
= Afix + N (Amem · FL + Alogic ) (20) computation of neural state and firing can be distributed across the
parallel neural array without serialization. Thus there is only one
where Afix is the fixed area dedicated to support functions (PLLs,
level of parallelism (K = 1) and F0 = 1.
debug circuitry, JTAG, control registers, etc.).
The output of the neural array is multiplexed onto a shared AER
We need a common unit of area in order to perform operations
bus. If more than one event is generated in the array during the
on both Amem and Alogic . We use memory byte equivalent units. For
same clock cycle, one of the events will have to wait in a buffer for
logic, this is the size (in Bytes) of an SRAM that occupies the same
an open bus cycle. This contention for the shared communications
silicon area as the given block of logic. For memory this is simply
resource can be modeled using queueing theory. Here we use an
the size of the RAM. For ASICs, this conversion is accomplished by
M/M/1 queue model, where the server rate µ is a fixed constant,
determining the average mm2 area of both SRAM and logic gates
for a particular process. Then logic gates can be converted directly based on the clock frequency of the bus arbiter. The arrival rate
into SRAM Bytes, dropping the mm2 in the process. In FPGAs, we λ is a function of the average neuron firing (communication) rate
accomplish this conversion by determining the area ratio between and the number of parallel neuron engines: λ = G1 N. Using these
block RAMs and Configurable Logic Blocks (CLBs) or logic slices. For parameters, the expected communication contention delay for an
a Xilinx Spartan III FPGA, we estimate an 18 kB SRAM occupies the M/M/1 queue is:
same area as 32 logic slices. Thus we use a conversion of 64 B per 1 1
slice, subdivided as 32 B per slice LUT and 32 B per slice register. t̄ = = . (25)
µ−λ µ − G1 N
In our first case, we consider the case where we are constrained
not only by the fixed total silicon area, but also by requiring that the Given an average neural firing rate of 100 Hz = 1/100 s =
total number of neurons is held constant. The total area constraint 0.01 s, a system clock rate of 200 MHz, and real-time operation,
is given in (20) and the total neuron constraint is given in (10) the queueing parameters are: µ = 200 MHz and λ = N100 Hz.
when Q is equal to a constant. In this case, the two constraints only The computation and communication fractions are estimated as
intersect at a single point. Combining (20) and (10): we obtain: follows: g0 = 200 M, g1 = 100, gtot = g0 + g1 = 200,000,100,
g g
and thus: G0 = g 0 and G1 = g 1 . Note that if we operate at
Q tot tot
Atot = (Amem FL + Alogic ) + Afix (21) a timescale faster than real-time, then µ = 200 MHz/xRT and
FL g0 = 200 M/xRT .
QAlogic We solve our constrained optimization problem using the
FL = . (22) method of Lagrange multipliers. The Lagrangian is formed by
Atot − Afix − Amem Q
combining the cost function and the area constraint:
Given values of Q , Atot , Afix , Alogic , and Amem , we can directly
determine the value of FL that satisfies both constraints. And the 1
L(N , FL, Λ) = (G0 D0 + G1 D1 )
value of N at that point is given by (10). N
+ Λ N (Amem FL + Alogic ) + Afix − Atot .
 
(26)
6.2. Simplified cost function for symmetric multiprocessing
Substituting in, using D0 = Q and D1 = Q + α + t̄:
Here we apply this objective function to our symmetric array of 
α 1

parallel neural processing engines. The asymmetric cost function L = G0 FL + G1 FL + +
given by Eq. (1) discussed in Section 3.1 simplifies for symmetric N N (µ − G1 N )
architectures (Cassidy & Andreou, 2012) to: + Λ N (Amem FL + Alogic ) + Afix − Atot .
 
(27)
 
K −1 M −1
 Fj  Differentiating the Lagrangian with respect to the three variables
JED = Gij Dij (N , FL, Λ):
j =0
Nj i=0
∂L
 

K −1 M −1
γ −1 G1
Fj  = G1 +
∂N N 2 (µ − G1 N ) N (µ − G1 N )−2
 
× Njh Gijh Eijh (23)
j=0
Nj h∈{A,I } i=0 −G1 α
+ Λ Amem FL + Alogic = 0
 
+ 2
(28)
where Fj is the fraction of the algorithm that has parallelism of N
Nj . Each fraction Fj is divided into M cost components of the ∂L
= (G0 + G1 ) + Λ (NAmem ) = 0 (29)
architecture. Gij is the fraction of Fj that has the ijth cost component ∂ FL
of Dij or Eij . The ijth delay is Dij and the ijth energy cost is Eijh for ∂L
= N (Amem FL + Alogic ) + Afix − Atot .
 
the jth fraction of the algorithm and the active or idle processors, (30)
h ∈ {A, I }. The weighting parameter γ scales the relative influence ∂Λ
of delay and energy on the cost of the architecture. Since Fj is Given these three equations and three unknowns, we can solve
a fraction,
K −1 for the optimal architecture using standard numerical methods.
j=0 Fj must equal 1 and since Gij is also a fraction,
M −1 We can also view the constrained cost function graphically. The
i=0 Gij must equal 1. cost function JD , constrained by (20), is plotted in Fig. 27. (Note
Analysis of the symmetric parallel neural architecture proceeds the logarithmic scale on both axes.) Since the constrained cost
as follows. Optimizing for delay (and neglecting energy, γ = 0), function is monotonically decreasing, the optimum architecture
we have simply: (minimum JD ) occurs with a maximum number of parallel physical
1 neuron engines (maximum N given the area constraint) and no
JD = (G0 D0 + G1 D1 ) (24) multiplexing (frame length FL = 1).
N
20 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

Fig. 27. Cost function JD in units of cycles and total number of neurons Q versus Fig. 28. Communication limited cost function JD in units of cycles versus number
number of physical neuron engines. of physical neural engines. (NCR = neural computation rate.)

From this figure, we can see four different interesting regimes. Table 6
Area implementation results: Xilinx V5SX240.
The first one is when minimizing the cost of the system in terms
of delay JD . In this regime, we maximize parallelism, filling the Q (k) N FL REG (%) LUT (%) Used RAM (%) Used BRAMs (%)
area with physical neurons and not using any multiplexing, thus 512 8 64 k 1.3 0.8 49.6 49.6
FL = 1. This architecture is entirely composed of logic and no 512 16 32 k 2.6 1.5 49.6 49.6
state RAM. A second interesting regime occurs when maximizing 512 32 16 k 4.9 2.9 49.6 49.6
512 64 8k 9.4 5.5 49.6 49.6
the total number of neurons in the system Q , then we build a
512 128 4k 17.8 10.5 49.6 50.2
system with one physical neuron N = 1 and maximum FL. This
256 8 32 k 1.3 0.8 24.8 24.8
results in an architecture overwhelmingly composed of RAM. This 256 16 16 k 2.4 1.4 24.8 24.8
is intuitive since the marginal increase in area for a multiplexed 256 32 8k 4.7 2.7 24.8 24.8
neuron is 2 B, while the marginal increase in area for a physical 256 64 4k 8.8 5.2 24.8 26.0
neuron is 10 kB. A third case is to maximize performance, while 256 128 2k 16.6 9.9 24.8 49.2
still meeting a specified total number of neurons Q . In this case, we 128 8 16 k 1.2 0.7 12.4 12.4
128 16 8k 2.3 1.4 12.4 12.4
find the desired value of Q on the vertical axis, find the intersection
128 32 4k 4.4 2.6 12.4 12.6
with the Q curve (black dashed), and then find the corresponding 128 64 2k 8.3 4.9 12.4 25.0
value of N at that point. Finally, if we wish to find a ‘‘sweet spot’’, 128 128 1k 15.5 9.4 12.4 46.3
balancing the total neuron density and the performance, we could 128 256 512 32.2 49.2 12.4 34.3
choose the architecture at the knee of the curve. In Fig. 27, this 128 512 256 59.6 67.7 12.4 56.6
corresponds to approximately 256 physical neurons, a delay cost
of 8 thousand, and 2 million total neurons. compared to an ASIC. The next section presents our results using
In the cost function example curve in Fig. 27, the minimum oc- FPGAs, including discussion on limitations arising from the FPGA
curs with the maximum number of parallel neuron engines and organization.
FL = 1. The performance is bound by the limit that FL cannot
go below 1. Communication constraints imposes another possi- 6.3. Implementation results
ble limit to the number of parallel engines that could be imple-
mented. As the number of parallel engines increases, the commu- We implemented our neural array architecture, targeting Xil-
nication traffic on the shared AER bus increases. If the amount of inx FPGAs in order to build a working system as well as to verify
traffic exceeds the available bandwidth, the system breaks down. the analytical model. By using parameterized VHDL, we varied the
As the amount of traffic approaches this limit, the communica- architectural parameters and implemented different array config-
tion delay increases. By varying the neural computation rate (NCR) urations, thus exploring the design space. We simultaneously var-
while holding the average neural firing rate constant, we alter the ied the number of physical neurons N and the multiplexed frame
amount of traffic per simulation timestep, effectively varying the length FL by factors of 2, holding the total number of neurons Q
communication traffic generated in the system. Note that we in- constant. We repeated this process for four different values of Q :
crease NCR at the expense of decreasing xRT, according to Eq. (14). 128 thousand, 256 thousand, 512 thousand, 1 million. For the three
As we decrease the NCR, increasing the effective traffic rate, the smaller values of Q , we targeted a Xilinx Virtex 5 SX240 FPGA,
system performance will become communication limited. This is while for the one million neuron case, we targeted a Xilinx Virtex 6
shown in Fig. 28. For three values of NCR, we see the effect on the SX475 FPGA. The area implementation results for the V5SX240 are
system performance in terms of delay cost JD . For smaller values of shown in Table 6 while the results for the V6SX475 are shown in
NCR, the performance hits an asymptote, and the optimal architec- Table 7.
ture is prior to the asymptote. We can make several interesting observations based on the data
This architectural exploration assumes an ASIC or custom in these tables. First we see that the overall utilization of the FPGA
VLSI design substrate. However to test our architecture and our resources is rather low. Block RAM utilization is approximately
analysis, we used FPGAs as a surrogate design platform. FPGAs 50% for 512 thousand neurons and even lower in all of the other
are advantageous for rapid prototyping and reprogrammability, cases. Logic utilization ranges from less than a percent (N = 8) to
however, they have fixed resources and organization (memory approximately 10% for N = 128. This highlights the high overhead
sizes and ports, interconnect overhead, etc.). This places limits of using FPGAs with fixed resources. A blank substrate would allow
on the full range of architectures that can be implemented as a much higher utilization of the available silicon. Second, we see
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 21

Fig. 29. Area vs. neuron engines.

Table 7 Table 8
Area implementation results: Xilinx V6SX475. Summary of parameters for area model and MSE fit line.
Q (k) N FL (k) REG (%) LUT (%) Used RAM (%) Used BRAMs (%) Model MSE Model MSE Model MSE
512 k 512 k 256 k 256 k 128 k 128 k
1024 16 64 0.7 1.1 36.1 36.1
1024 32 32 1.3 2.1 36.1 36.1 Slope 10 10.1 10 9.53 10 8.93
1024 64 16 2.4 3.9 36.1 36.1 Offset 1024 1060 512 544 256 287
1024 128 8 4.5 7.4 36.1 36.1
1024 256 4 8.6 14.3 36.1 36.1
Using our model, we also plotted the dual constraints, constant
total neurons (10) and constant total area (20), shown in Fig. 30.
that the architecture is dominated by RAM, as compared to logic. As noted earlier, due to the rigid FPGA organization, we were
With fixed FPGA resources, the RAM is the limiting resource. Third, only able to use a fraction of the total FPGA area (generally less
note that in several cases, the RAM is actually limited by the than 50%). Thus, assuming a fraction of 35% of the total area, the
number of RAM ports, and hence number of BRAMs and not the maximum number of neural engines predicted by the analytical
actual amount of RAM actually used. With a constant Q , when N model is no more than 128 (if constrained to be a power of 2). This
is varied, FL is varied by an equivalent amount in the opposite analytical prediction matches the empirical results given in Table 6,
direction, thus the total RAM should remain constant, as given where architectures with N > 128 failed to map for Q = 512
by (13). This holds true for 512 and 1024 thousand neurons (see and 256 thousand. In the case of Q = 128 thousand, the FPGA
the ‘‘BRAM blks’’ column). However, for 128 and 256 thousand utilization greatly increases due to the jump from using block RAM
neurons, the number of used block RAMs begins to increase even to distributed RAM. Thus N = 256 and N = 512 are able to map.
though the used RAM does not. This is to accommodate the number
of ports needed by the architecture (# BRAMs ≈ RAM ports/2). 6.5. Cost model validation
Block RAMs come in a fixed size: 4 kB, and have two ports. Finally,
We also experimentally verified the frame time (hence
notice the cases of Q = 128 thousand and N = 256, 512. These
communication delay) of the neural array using a cycle–accurate
two design points deviate from the trends of linearly increasing
simulator (ModelSim). The results are shown in Fig. 31 together
LUTs and BRAMs increasing by the number of required ports. This
with the delay cost of the architecture, as predicted by the
deviation is due to a switch from storing internal state using block
analytical model (27). With a high neural computation rate (NCR),
RAMs to distributed RAM.
the communication overhead is negligible. Thus the frame time
is the dominant term in the delay cost of the architecture. Once
6.4. Area model validation
again the analytical model and the experimental results match very
Plotting the area data reveals a strong correlation between closely.
empirical results and our analytical model. According to (19) in our
7. Discussion
model, the array area is a linear function. Holding the total number
of neurons constant Q = N · FL, the area allocated to internal RAM In this paper we have revisited Marr’s levels of representation,
stays constant, while the area for the neural engines scales linearly emphasized the need for layered levels, the physical and the
with N. The empirical and analytical results are shown in Fig. 29. abstract, not in a hierarchy but rather in a recurrent relationship.
The solid circles are empirical data points, while the lines are the Computational theory sits on top and provides the foundations
first order analytical model, using Amem = 2 and Alogic = 10 kB. The for information processing architectures in brains and modern
analytical area model parameters and the minimum squared error computing systems. Furthermore we provide a concrete design
fit lines for the empirical data points are summarized in Table 8. methodology for architectural exploration and understanding
While the empirical data is very close to our first-order analytical through ‘‘constraints that deconstrain’’ (Kirschner & Gerhart, 1998)
model, we can see that the slope of the MSE lines change for the through the use of a cost function optimization and parametric
different values of Q , thus Alogic slightly varies as a function of FL. models at different levels of description that link software layers
In addition, the fit lines have a small offset (approximately 32 kB) of abstraction to hardware layers, as well as delay and energy
above the first-order model. Back annotating the model with the constrained by physical space.
more detailed model parameters would improve the analytical For example, we have demonstrated the architecture of
predictions even more. functional components, i.e. dynamical neurons from the simple
22 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

Fig. 30. Constant total neuron Q and constant total area Atot curves.

experiment in Section 5.5 involves multiple timescales and layers

of representation. At the base level is the neural computation
timescale (1 ms time ticks). The learning occurs at a longer/slower
timescale. Then we used simulated annealing to learn the
free parameters of the system (highest/longest timescale.) The
architecture was optimized for the continual processing at the
1 ms timescale. The learning events occur less frequently and are
less optimized in the data flow architecture (a read–modify–write
action on memory to update the synaptic weights.) At the
highest level, the simulated annealing algorithm was run on the
host PC (in software), updating parameters infrequently at the
longest timescale. The recent paper by Douglas and Martin linking
behavior to the architecture of the cortical sheet demonstrates
beautifully the links of information processing and computation
under physical constraints, adding the developmental dimension
in biological tissue (Douglas & Martin, 2012).
From an engineering perspective however, we may want to op-
Fig. 31. JD cost model validation: experiment vs. theory. timize an architecture for specific tasks such as natural language
processing or the functionality of a specific silicon nervous sys-
LIF to the more complex Izhikevich model. While the LIF neural tem for robotic applications. To do so, we must further develop the
model has been dominant in neuromorphic systems, we have ‘‘bottom up’’ models in such a way so that they can be parameter-
demonstrated that neurons with more complex behaviors can ized to be simple functions of an application domain, for instance
be added to neuromorphic systems with relative ease. This inference using graphical models. This optimization approach was
functionality has been enabled by shifting from a predominantly recently demonstrated for automatic speech recognition (Cassidy,
analog silicon neuron paradigm, to a digital silicon neuron Yu, Zhou, & Andreou, 2011) and can be generalized for asymmetric
paradigm abstracting functionality. The digital processing level spiking neural multiprocessors, defining an exciting direction for
of implementation is also advantageous for its stability and high future work. Rapid design time, low cost, flexibility, digital preci-
precision properties. Third, in terms of computational theory, the sion, and stability are characteristics that favor digital implemen-
architecture optimization work contributes a methodology for tation as a promising alternative to analog VLSI based approaches
analytical optimization of parallel architectures subject to the for designing neuromorphic systems. High computational power
physical constraints of delay, energy, and area. Our approach goes as well as low size, weight, and power (SWAP) are advantages
beyond what was recently discussed in a paper on architecture, that digital architectures offer over software based neuromorphic
constraints, and behavior by Doyle and Csete (2011), where the systems.
architecture of the brain is discussed in terms of the physical Some applications will favor minimizing power dissipation as
layers in clothing juxtaposed to layers of abstraction in biological opposed to performance. This is achieved by trading off faster
systems. The end result of our work at each of these layers of than real-time performance for lower clock frequency and a lower
abstraction is the advent of nano-CMOS digital neuromorphic supply voltage. In terms of the cost function in Eq. (1), this will be
systems which incorporate both computation and learning, that equivalent to setting the weighting parameter γ to a value greater
can scale into the range of millions of neurons per chip. than one. The latter will be applicable to a full custom CMOS design
Our goal was to explore the design of canonical structures, and with digital circuits operating in subthreshold (Martin, Pouliquen,
hence we have begun at the level of spiking neurons and proceeded Andreou, & Fraeman, 1996; Vittoz, 2005) at a few MHz (Imam et al.,
to design a neural architecture that is capable of supporting 2012; Merolla et al., 2011) or even in the hundreds of kHz at ultra
multiple functions. In the work presented here, we have focused low voltages (Lotze & Manoli, 2012). According to the formula P =
on ‘‘generic’’ design for a neural architecture without consideration CV 2 f , this will lead to significant power savings. Preliminary work
of the problem at hand. Partly our motivation is a belief that in this direction and analysis of communication architectures in
the brain is comprised of canonical computational structures this application regime, suggests that a switched mesh architecture
that have evolved to solve problems ranging from early sensory may be advantageous over the AER scheme used in this paper
processing to high-level cognition. The cortical column/STDP (Cassidy, Murray, Andreou, & Georgiou, 2011). It should be pointed
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 23

out that switched capacitor and charged based analog circuits minimizing costly off-chip accesses by keeping state local to each
(Bamford & Giulioni, 2010; Noack, Mayr, Partzsch, Schultz, & neural engine/processing unit.
Schuffny, 2012; Stanacevic & Cauwenberghs, 2005) could scale in Finally, the dawn of neuromorphic systems engineering at the
deep sub-micron CMOS and hence it is possible that one would see scale has arrived. We have presented a design methodology for
hybrid architectures involving analog state holding elements, in large-scale digital neuromorphic architectures for the nano-CMOS
synapses and even sophisticated neuron circuits (Folowosele et al., era. Our approach to the design of spiking neurons and STDP
2009). learning circuits relies on layered parallel computational struc-
Finally, our architecture does not appear to be a truly single- tures where neurons are abstracted as arithmetic logic units and
chip solution because of the two banks of SRAM external to communication processors. We demonstrated the validity of the
the FPGA. However, these external SRAM banks can be readily design methodology through the implementation of cortical de-
integrated into a single chip solution using a 3D VLSI process, velopment using spiking neurons and STDP learning and neural
such as provided by Tezzaron (Tezzaron, 2011). In this approach, architecture optimization in state-of-the-art Field Programmable
three VLSI tiers map to one tier for the parallel computation layer, Gate Arrays (FPGAs). The implementation of neural arrays, spiking
including state memory and logic, and two tiers of SRAM for the neurons as well as STDP learning rules have been prototyped in
synapse weights and re-mapper RAM. It is interesting to note that FPGAs. The latter are an ideal platform for investigating digital sil-
in the early days of neural networks, there were substantial efforts icon neuron architectures, due to the inherent flexibility and high-
to develop all digital bio-inspired systems (Hammerstrom, 1995, level design methodology and offer a stepping stone towards full
1998; Hammerstrom & Lulich, 1996; Wawrzynek et al., 1996; large-scale ASIC designs. With this work, we have taken a signif-
Wawrzynek, Asanovic, & Morgan, 1993). While these efforts were icant step towards realizing the goals of a new class of artificial,
partially successful and visionary at that time, they were limited high-performance, energy-efficient, parallel computational archi-
by technology and namely the inability to integrate substantial tectures inspired by the brain.
amounts of memory on the die. It would be interesting to see if
the emerging 3D-CMOS technology (Tezzaron, 2011) or the IBM
Acknowledgments
embedded DRAM technologies (Iyer et al., 2005) will help alleviate
this problem.
This work was supported by the EU ICT Grant (ICT-231168-
Also promising directions in the nano-CMOS era involve ex-
SCANDLE) ‘‘acoustic SCene ANalysis for Detecting Living Entities’’
ploiting more unconventional memory technologies such as nano
and by ONR MURI (N000141010278) ‘‘Figure-Ground Processing,
wires and switches (Avizienis et al., 2012), memristors (Strukov,
Saliency and Guided Attention for Analysis of Large Natural
Snider, Stewart, & Williams, 2008), nano and CMOL architectures
Scenes’’.
(Likharev, 2008) and system concepts that rely on these tech-
nologies (Gao & Hammerstrom, 2007; Gao, Zaveri, & Hammer-
References
strom, 2008; Hammerstrom & Zaveri, 2009). The early pulse cou-
pled mixed system designs that exploited timing to represent Amdahl, G. M. (1967). Validity of the single processor approach to achieving
analog state (Murray, Del Corso, & Tarassenko, 1991) may also large scale computing capabilities. In Proceedings AFIPS spring joint computer
see a re-emergence in the context of stochastic computation and conference (pp. 483–485).
Anderson, C. H., & Eliasmith, C. (2004). Neural engineering: computation, representa-
probabilistic event-based systems. It also seems increasingly likely tion, and dynamics in neurobiological systems. The MIT Press.
that process scalable charged-based (Anthony, Ma Kohler, Kurtze, Andreou, A. G., & Kalayjian, Z. K. (2002). Polarization imaging: principles and
Kushne, & Sollner, 2008) and switched-capacitor circuits (Bam- integrated polarimeters. IEEE Sensors Journal, 2(6), 566–575.
ford, Hogri et al., 2012; Folowosele et al., 2009; Georgiou, Andreou, Andreou, A. G., Meitzler, R. C., Strohben, K., & Boahen, K. A. (1995). Analog VLSI
neuromorphic image acquisition and pre-processing systems. Neural Networks,
& Pouliquen, 2006; Noack et al., 2012) may play a role yielding 8(7/8), 1323–1347.
true mixed-signal architectures that exploit the best of continuous Anthony, M., Kohler, E., Kurtze, J., Kushner, L., & Sollner, G. (2008). A process-
value and discrete value representations. scalable low-power charge-domain 13-bit pipeline ADC. In VLSI Circuits, 2008
IEEE Symposium on(pp. 222–223).
APT-Group. (2011a). SpiNNaker a chip multiprocessor for neural network
8. Conclusions simulation. Tech. rep., The University of Manchester. January.
APT-Group. (2011b). SpiNNaker application programming interface. Tech. rep.
Version 0.0, University of Manchester. December.
We conclude this paper with a summary of ideas and results APT-Group. (2011c). SpiNNaker software specification and design. Tech. rep.
that we believe are relevant to the design of future neuromorphic version 0.0, The University of Manchester. December.
systems aimed at commodity information processing at the scale. Arthur, J. V., & Boahen, K. A. (2010). Silicon–neuron design: a dynamical systems
approach. IEEE Transactions on Circuits and Systems I: Regular Papers, 58(5),
First, we demonstrated the advantages of the digital neuron ab- 1034–1043.
straction instead of the more biophysically realistic analog neu- Arthur, J. V., Merolla, P. A., Akopyan, F., Alvarez, R., Cassidy, A. S., & Chandra, S. et
al. (2012). Building block of a programmable neuromorphic substrate: a digital
rons. Second, to emulate the massive parallelism and high neuron neurosynaptic core. In Proceedings of the 2012 international joint conference on
density found in biology, we multiplex neural state computation in neural networks, IJCNN (pp. 1–8).
order to take advantage of high clock frequencies and dense SRAMs Avizienis, A. V., Sillin, H. O., Martin-Olmos, C., Shieh, H. H., Aono, M., Stieg, A. Z., et al.
(2012). Neuromorphic atomic switch networks. PLoS One, 7(8), e42772.
found in nanometer silicon technology. Third, we have presented Bamford, S. A., & Giulioni, M. (2010). Intimate mixing of analogue and digital signals
an energy–delay cost function that is the first step towards devel- in a field-programmable mixed-signal array with lopsided logic. In Proceedings
opmental and evolutionary design principles that we believe have of the 2010 IEEE biomedical circuits and systems conference, BioCAS 2010
shaped biological neural systems. A highly quantitative formula- (pp. 234–237).
Bamford, S. A., Hogri, R., Giovannucci, A., Taub, A. H., Herreros, I., Verschure, P. F.
tion, the cost function also presents a qualitative view of parallel M. J., et al. (2012). A VLSI field-programmable mixed-signal array to perform
processing subject to physical constraints in layered information neural signal processing and neural modeling in a prosthetic system. IEEE
processing architectures. Delay cost is minimized by maximizing Transactions on Neural Systems and Rehabilitation Engineering, 20(4), 455–467.
Bamford, S. A., Murray, A. F., & Willshaw, D. J. (2012). Spike-timing-dependent
parallelism in the system. Under a fixed physical size constraint— plasticity with weight dependence evoked from physical constraints. IEEE
we have employed area in this paper—the core area (processor and Transactions on Biomedical Circuits and Systems, 6(4), 385–398.
local memory) must be minimized in order to increase parallelism, Bartolozzi, C., & Indiveri, G. (2007). Synaptic dynamics in analog VLSI. Neural
Computation, 19(10), 2581–2603.
balanced by the simultaneous goal of maximizing processing den- Belhadj, B., Tomas, J., & Bornat, Y. (2009). Digital mapping of a realistic spike timing
sity through multiplexing. Both energy and delay costs are reduced plasticity model for real-time neural simulations. In Proceedings of the XXIV
by eliminating the serial fraction of the algorithm, as well as by conference on design of circuits and integrated systems, DCIS (pp. 1–6).
24 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

Bi, G. Q., & Poo, M. M. (1998). Synaptic modifications in cultured hippocampal Doyle, J. C., & Csete, M. (2011). Architecture, constraints, and behavior. Proceedings
neurons: dependence on spike timing, synaptic strength, and postsynaptic cell of the National Academy of Sciences of the United States of America, 108(Suppl. 3),
type. The Journal of Neuroscience, 18(24), 10464–10472. 15624–15630.
Boahen, K. A. (1996). Retinomorphic vision systems. In Proceedings of the 5th Eliasmith, C., & Stewart, T. C. (2011). Nengo and the neural engineering framework:
international conference on microelectronics for neural networks, MicroNeuro connecting cognitive theory to neuroscience. In Proceedings of the 33rd annual
(pp. 2–14). meeting of the cognitive science society (pp. 1–2).
Boahen, K. A. (2000). Point-to-point connectivity between neuromorphic chips Eliasmith, C., Stewart, T. C., Choo, X., Bekolay, T., DeWolf, T., Tang, Y., et al. (2012).
using address events. IEEE Transactions on Circuits and Systems II: Analog and A large-scale model of the functioning brain. Science, 338(6111), 1202–1205.
Digital Signal Processing, 47(5), 416–434. Etienne-Cummings, R., Kalayjian, Z. K., & Donghui, C. (2001). A programmable focal-
Boahen, K. A., & Andreou, A. G. (1992). A contrast sensitive silicon retina with plane MIMD image processor chip. IEEE Journal of Solid-State Circuits, 36(1),
reciprocal synapses. Advances in Neural Information Processing Systems, 3, 64–73.
Fasnacht, D. B., & Indiveri, G. (2011). A PCI based high-fanout AER mapper with 2 GB
764–772.
Boahen, K. A., Pouliquen, P. O., Andreou, A. G., & Jenkins, R. E. (1989). A RAM look-up table, 0.8 us latency and 66 MHz output event-rate. In Proceedings
heteroassociative memory using current-mode MOS analog VLSI circuits. IEEE of the 45th annual conference on information sciences and systems, CISS (pp. 1–6).
Transactions on Circuits and Systems, 36(5), 747–755. Federico, M. D., Mandolesi, P. S., Julian, P., & Andreou, A. G. (2008). Experimental
Bofill-i Petit, A., & Murray, A. F. (2004). Synchrony detection and amplification results of simplicial CNN digital pixel processor. IET Electronics Letters, 44(1),
by silicon neurons with STDP synapses. IEEE Transactions on Neural Networks, 27–29.
Folowosele, F. O., Harrison, A., Cassidy, A. S., Andreou, A. G., Etienne-Cummings, R., &
15(5), 1296–1304.
Mihalas, S. et al. (2009). A switched capacitor implementation of the generalized
Bruederle, D., Petrovici, M. A., Vogginger, B., Ehrlich, M., Pfeil, T., Millner, S., et al.
linear integrate-and-fire neuron. In Proceedings of the 2009 IEEE international
(2011). A comprehensive workflow for general-purpose neural modeling with
symposium on circuits and systems, ISCAS (pp. 2149–2152).
highly configurable neuromorphic hardware systems. Biological Cybernetics,
Galluppi, F., Davies, S., Furber, S. B., Stewart, T. C., & Eliasmith, C. (2012). Real
104(4–5), 263–296.
time on-chip implementation of dynamical systems with spiking neurons. In
Camuñas-Mesa, L. A., Zamarreño-Ramos, C., Linares-Barranco, A., Acosta-Jimenez,
Proceedings of the 2012 international joint conference on neural networks, IJCNN
A. J., Serrano-Gotarredona, T., & Linares-Barranco, B. (2012). An event-driven
(pp. 1–8).
multi-kernel convolution processor module for event-driven vision sensors.
Gao, C., & Hammerstrom, D. W. (2007). Cortical models onto CMOL and CMOS—
IEEE Journal of Solid-State Circuits, 47(2), 504–517.
architectures and performance/price. IEEE Transactions on Circuits and Systems
Cardoso, J., Diniz, P., & Weinhardt, M. (2010). Compiling for reconfigurable
I: Regular Papers, 54(11), 2502–2515.
computing: a survey. ACM Computing Surveys, 42(4), 13:1–13:64.
Gao, C., Zaveri, M. S., & Hammerstrom, D. W. (2008). CMOS/CMOL architectures for
Carmona, R., Espejo, S., Dominguez-Castro, R., Rodriguez-Vazquez, A., Roska, T., &
spiking cortical column. In Proceedings of the 2008 international joint conference
Kozek, T. et al. (1998). A 0.5 µm CMOS CNN analog random access memory
on neural networks, IJCNN (pp. 2441–2448).
chip for massive image processing. In Proceedings of the 5th IEEE international
Genov, R., & Cauwenberghs, G. (2001). Charge-mode parallel architecture for
workshop on cellular neural networks and their applications, CNAA (pp. 271–276).
vector–matrix multiplication. IEEE Transactions on Circuits and Systems II:
Cassidy, A. S., & Andreou, A. G. (2008). Dynamical digital silicon neurons. In
Analog and Digital Signal Processing, 48(10), 930–936.
Proceedings of the 2007 IEEE biomedical circuits and systems conference, BioCAS
Georgiou, J., & Andreou, A. G. (2006). High-speed, address-encoding arbiter
2007 (pp. 289–292).
architecture. IET Electronics Letters, 42(3), 170–171.
Cassidy, A. S., & Andreou, A. G. (2009). Analytical methods for the design and
Georgiou, J., & Andreou, A. G. (2007). Address-data event representation
optimization of chip-multiprocessor architectures. In Proceedings of the 43rd
for communication in multichip neuromorphic system architectures. IET
annual conference on information sciences and systems, CISS (pp. 482–487).
Electronics Letters, 43(14), 767.
Cassidy, A. S., & Andreou, A. G. (2012). Beyond Amdahl’s law: an objective Georgiou, J., Andreou, A. G., & Pouliquen, P. O. (2006). A mixed analog/digital
function that links multiprocessor performance gains to delay and energy. IEEE asynchronous processor for cortical computations in 3D SOI-CMOS. In
Transactions on Computers, 61(8), 1110–1126. Proceedings of the 2006 IEEE international symposium on circuits and systems,
Cassidy, A. S., Andreou, A. G., & Georgiou, J. (2011). A combinational digital logic
ISCAS (pp. 4955–4958).
approach to STDP. In Proceedings of the 2011 IEEE international symposium on
Giulioni, M., Camilleri, P., Mattia, M., Dante, V., Braun, J., & Del Giudice, P. (2011).
circuits and systems, ISCAS (pp. 673–676).
Robust working memory in an asynchronously spiking neural network realized
Cassidy, A. S., Denham, S. L., Kanold, P. O., & Andreou, A. G. (2007). FPGA based
with neuromorphic VLSI. Frontiers in Neuroscience, 5, 149.
silicon spiking neural array. In Proceedings of the 2007 IEEE biomedical circuits Goldberg, D. H., Cauwenberghs, G., & Andreou, A. G. (2001a). Analog VLSI spiking
and systems conference, BioCAS 2007 (pp. 75–78). neural network with address domain probabilistic synapses. In Proceedings
Cassidy, A. S., Murray, T., Andreou, A. G., & Georgiou, J. (2011). Evaluating on-chip of the 2001 IEEE international symposium on circuits and systems, ISCAS
interconnects for low operating frequency silicon neuron arrays. In Proceedings (pp. 241–244).
of the 2011 IEEE international symposium on circuits and systems, ISCAS Goldberg, D. H., Cauwenberghs, G., & Andreou, A. G. (2001b). Probabilistic synaptic
(pp. 2437–2440). weighting in a reconfigurable network of VLSI integrate-and-fire neurons.
Cassidy, A. S., Yu, K., Zhou, H., & Andreou, A. G. (2011). A high-level analytical Neural Networks, 14, 781–793.
model for application specific CMP design exploration. In Proceedings of the 2011 Goodman, D. F. M., & Brette, R. (2009). The Brian simulator. Frontiers in Neuroscience,
conference on design automation & test in Europe, DATE’11 (pp. 1–6). 3(2), 192–197.
Cembrano, G., Rodriguez-Vazquez, A., Galan, R., Jimenez-Garrido, F., Espejo, S., & GSI, (2011). GSI technology home. [Link]. February.
Dominguez-Castro, R. (2004). A 1000 FPS at 128 × 128 vision processor with Hall, T. S., Twigg, C. M., Gray, J. D., Hasler, P. E., & Anderson, D. V. (2005). Large-
8-bit digitized I/O. IEEE Journal of Solid-State Circuits, 39(7), 1044–1055. scale field-programmable analog arrays for analog signal processing. IEEE
Chicca, E., Badoni, D., Dante, V., D’Andreagiovanni, M., Salina, G., Carota, L., et al. Transactions on Circuits and Systems I: Regular Papers, 52(11), 2298–2307.
(2003). A VLSI recurrent network of integrate-and-fire neurons connected by Hammerstrom, D. W. (1995). A digital VLSI architecture for real-world applications.
plastic synapses with long-term memory. IEEE Transactions on Neural Networks, In An introduction to neural and electronic networks (2nd ed.) (pp. 335–358).
14(5), 1297–1307. Academic Press.
Chklovskii, D. B., Mel, B. W., & Svoboda, K. (2004). Cortical rewiring and information Hammerstrom, D. W. (1998). Digital VLSI for neural networks. In M. A. Arbib (Ed.),
storage. Nature, 431(7010), 782–788. The handbook of brain theory and neural networks (pp. 304–309). The MIT Press.
Choudhary, S., Sloan, S., Fok, S., Neckar, A., Trautmann, E., & Gao, P. et al. (2012). Hammerstrom, D. W., & Lulich, D. P. (1996). Image processing using one-
Silicon neurons that compute. In Proceedings of the 2012 international joint dimensional processor arrays. Proceedings of the IEEE, 84(7), 1005–1018.
conference on neural networks, IJCNN (pp. 1–8). Hammerstrom, D. W., & Zaveri, M. S. (2009). Prospects for building cortex-scale
Chua, L. O., & Yang, L. (1988). Cellular neural networks: theory. IEEE Transactions on CMOL/CMOS circuits: a design space exploration. In Proceedings of the 27th
Circuits and Systems, 35(10), 1257–1272. Norchip conference, NORCHIP-2009 (pp. 1–8).
Churchland, P. S., & Sejnowski, T. J. (1988). Perspectives on cognitive neuroscience. Harrison, A., Özgün, R., Lin, J., Andreou, A. G., & Etienne-Cummings, R. (2010). A spike
Science, 242(4879), 741–745. based 3D imager chip using a mixed mode encoding readout. In Proceedings
Culurciello, E., Etienne-Cummings, R., & Boahen, K. A. (2003). A biomorphic digital of the 2010 IEEE biomedical circuits and systems conference, BioCAS 2010
image sensor. IEEE Journal of Solid-State Circuits, 38(2), 281–294. (pp. 190–193).
Das, A. (2000). Optimizing coverage in the cortex. Nature Neuroscience, 3(8), Hodgkin, A. L., & Huxley, A. F. (1952). A quantitative description of membrane
750–752. current and its application to conduction and excitation in nerve. Journal of
de Garis, H., Shuo, C., Goertzel, B., & Ruiting, L. (2010). A world survey of artificial Physiology: London, 117(4), 500–544.
brain projects, Part I: large-scale brain simulations. Neurocomputing, 74(1–3). Horiuchi, T. K., & Koch, C. (1999). Analog VLSI-based modeling of the primate
Diorio, C., Hasler, P. E., Minch, B. A., & Mead, C. A. (1996). A single-transistor silicon oculomotor system. Neural Computation, 11(1), 243–265.
synapse. IEEE Transactions on Electron Devices, 43(11), 1972–1980. Hsin, C., Saighi, S., Buhry, L., & Renaud, S. (2010). Real-time simulation of biologically
Diorio, C., Hasler, P. E., Minch, B. A., & Mead, C. A. (1997). A floating-gate MOS realistic stochastic neurons in VLSI. IEEE Transactions on Neural Networks, 21(9),
learning array with locally computed weight updates. IEEE Transactions on 1511–1517.
Electron Devices, 44(12), 2281–2289. Hubel, D. H., & Wiesel, T. N. (1977). Ferrier lecture: functional architecture of
Dominguez-Castro, R., Espejo, S., Rodriguez-Vazquez, A., Carmona, R., Foldesy, P., macaque monkey visual cortex. Proceedings of the Royal Society B: Biological
Zarandy, A., et al. (1997). A 0.8 µm CMOS two-dimensional programmable Sciences, 198(1130), 1–59.
mixed-signal focal-plane array processor with on-chip binary imaging and Imam, N., Akopyan, F., Arthur, J. V., Merolla, P. A., Manohar, R., & Modha, D. S. (2012).
instructions storage. IEEE Journal of Solid-State Circuits, 32(7), 1013–1026. A digital neurosynaptic core using event-driven QDI circuits. In Proceedings
Douglas, R. J., & Martin, K. A. C. (2012). Behavioral architecture of the cortical sheet. of the 18th IEEE international symposium on asynchronous circuits and systems,
Current Biology, 22(24), R1033–R1038. ASYNC (pp. 25–32).
 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26 25

Indiveri, G., Chicca, E., & Douglas, R. J. (2004). A VLSI reconfigurable network of Mead, C. A., & Conway, L. (1979). Introduction to VLSI systems. Addison-Wesley
integrate-and-fire neurons with spike-based learning synapses. In Proceed- Publishers.
ings of the 2004 European symposium on artificial neural networks, ESANN Merolla, P. A., Arthur, J. V., Akopyan, F., Imam, N., Manohar, R., & Modha, D. S. (2011).
(pp. 405–410). A digital neurosynaptic core using embedded crossbar memory with 45 pJ per
Indiveri, G., Chicca, E., & Douglas, R. J. (2006). A VLSI array of low-power spiking spike in 45 nm. In 2011 IEEE custom integrated circuits conference, CICC 2011.
neurons and bistable synapses with spike-timing dependent plasticity. IEEE (pp. 1–4). IEEE.
Transactions on Neural Networks, 17(1), 211–221. Mihalas, S., & Niebur, E. (2009). A generalized linear integrate-and-fire neural model
Iyer, S. S., Barth, J. E., Jr., Parries, P. C., Norum, J. P., Rice, J. P., Logan, L. R., et al. (2005). produces diverse spiking behaviors. Neural Computation, 21(3), 704–718.
Embedded DRAM: technology platform for the Blue Gene/L chip. IBM Journal of Mill, R., Sheik, S., Indiveri, G., & Denham, S. L. (2011). A model for stimulus-specific
Research and Development, 49(2), 333–350. adaptation in neuromorphic analog VLSI. IEEE Transactions on Biomedical
Izhikevich, E. M. (2003). Simple model of spiking neurons. IEEE Transactions on Circuits and Systems, 1–8.
Misra, J., & Saha, I. (2010). Artificial neural networks in hardware a survey of two
Neural Networks, 14(6), 1569–1572.
decades of progress. Neurocomputing, 74(1–3), 239–255.
Izhikevich, E. M. (2004). Which model to use for cortical spiking neurons? IEEE
Modha, D. S., Ananthanarayanan, R., Esser, S. K., Ndirango, A., Sherbondy, A. J., &
Transactions on Neural Networks, 15(5), 1063–1070.
Singh, R. (2011). Cognitive computing. Communications of the ACM, 54(8), 62.
Kanold, P. O., & Shatz, C. (2006). Subplate neurons regulate maturation of cortical
Murray, A. F., Del Corso, D., & Tarassenko, L. (1991). Pulse-stream VLSI neural
inhibition and outcome of ocular dominance plasticity. Neuron, 51(5), 627–638.
networks mixing analog and digital techniques. IEEE Transactions on Neural
Karakiewicz, R., Genov, R., & Cauwenberghs, G. (2007). 480-GMACS/mW resonant Networks, 2(2), 193–204.
adiabatic mixed-signal processor array for charge-based pattern recognition. Nallatech, (2008). FPGA accelerated computing. [Link]. December.
IEEE Journal of Solid-State Circuits, 42(11), 2573–2584. Neumann, J. v. (1958). The computer and the brain. Yale University Press.
Opal-Kelly, (2012). FPGA USB modules. [Link]. Noack, M., Mayr, C., Partzsch, J., Schultz, M., & Schuffny, R. (2012). A switched-
Kestur, S., Park, M. S., Sabarad, J., Dantara, D., Narayanan, V., & Chen, Y. et al. capacitor implementation of short-term synaptic dynamics. In Proceedings of
(2012). Emulating mammalian vision on reconfigurable hardware. In 2012 the 19th international conference on mixed design of integrated circuits and
IEEE international symposium on field-programmable custom computing machines systems, MIXDES 2012. (pp. 214–218). IEEE.
(pp. 141–148). Pardo, F., Dierickx, B., & Scheffer, D. (1998). Space-variant nonorthogonal structure
Khan, M., Lester, D., Plana, L., Rast, A., Jin, X., & Painkras, E. et al. (2008). SpiNNaker: CMOS image sensor design. IEEE Journal of Solid-State Circuits, 33(6), 842–849.
mapping neural networks onto a massively-parallel chip multiprocessor. In Pearson, M., Pipe, A., Mitchinson, B., Melhush, G., Gilhespy, I., & Nibouche, M.
Proceedings of the 2008 international joint conference on neural networks, IJCNN (2007). Implementing spiking neural networks for real-time signal-processing
(pp. 2850–2857). and control applications: a model-validated FPGA approach. IEEE Transactions
Kirschner, M., & Gerhart, J. (1998). Evolvability. Proceedings of the National Academy on Neural Networks, 18(5), 1472–1487.
of Sciences of the United States of America, 95(15), 8420–8427. Pham, P.-H., Jelaca, D., Farabet, C., Martini, B., LeCun, Y., & Culurciello, E. (2012).
Koickal, T. J., Gouveia, L. C., & Hamilton, A. (2009). A programmable spike- NeuFlow: dataflow vision processing system-on-a-chip. In Proceedings of the
timing based circuit block for reconfigurable neuromorphic computing. 55th midwest symposium on circuits and systems, MWSCAS (pp. 1044–1047).
Neurocomputing, 72, 3609–3616. Posch, C., Matolin, D., & Wohlgenannt, R. (2011). A QVGA 143 dB dynamic range
Kumar, N., Himmelbauer, W., Cauwenberghs, G., & Andreou, A. G. (1998). An analog frame-free PWM image sensor with lossless pixel-level video compression and
VLSI chip with asynchronous interface for auditory feature extraction. IEEE time-domain CDS. IEEE Journal of Solid-State Circuits, 46(1), 259–275.
Transactions on Circuits and Systems II: Analog and Digital Signal Processing, 45(5), Pouliquen, P. O., Andreou, A. G., & Strohben, K. (1997). Winner-takes-all associative
600–606. memory: a Hamming distance vector quantizer. Analog Integrated Circuits and
La Rosa, M., Caruso, E., Fortuna, L., Frasca, M., Occhipinti, L., & Rivoli, F. (2005). Signal Processing, 13(1–2).
Neuronal dynamics on FPGA: Izhikevich’s model. In Proceedings of SPIE: Preissl, R., Wong, T. M., Datta, P., Flickner, M., Singh, R., Esser, S. K., et al. (2012).
bioengineered and bioinspired systems II (pp. 87–94). SPIE. Compass: a scalable simulator for an architecture for cognitive computing.
Laughlin, S. B., & Sejnowski, T. J. (2003). Communication in neuronal networks. In Proceedings of the 2012 International Conference for High Performance
Science, 301(5641), 1870–1874. Computing, Networking, Storage and Analysis, SC’12. (pp. 1–11). IEEE Computer
Lichtsteiner, P., Posch, C., & Delbruck, T. (2008). A 128 × 128 120 dB 15 µs Society.
latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid- Ramakrishnan, S., Hasler, P. E., & Gordon, C. (2012). Floating gate synapses with
State Circuits, 43(2), 566–576. spike-time-dependent plasticity. IEEE Transactions on Biomedical Circuits and
Likharev, K. K. (2008). CMOL: second life for silicon? Microelectronics Journal, 39(2), Systems, 5(3), 244–252.
177–183. Renaud, S., Tomas, J., Lewis, N., Bornat, Y., Daouzli, A., Rudolph, M., et al. (2010). PAX:
Lin, J. H., & Boahen, K. A. (2009). A delay-insensitive address-event link. In 15th IEEE a mixed hardware/software simulation platform for spiking neural networks.
symposium on asynchronous circuits and systems, ASYNC (pp. 55–62). Neural Networks, 23(7), 905–916.
Lin, J. H., Pouliquen, P. O., Andreou, A. G., Goldberg, A. C., & Rizk, C. G. (2012). Rice, K. L., Bhuiyan, M. A., Taha, T. M., Vutsinas, C. N., & Smith, M. C. (2009).
Flexible readout and integration sensors (FRIS): a bio-inspired, system-on-chip, FPGA implementation of Izhikevich spiking neurons for character recognition.
event based readout architecture. In Proceedings of SPIE: infrared technology and In Proceedings of the 2009 international conference on reconfigurable computing
applications XXXVIII conference (pp. 8353–1N). and FPGAs, ReConFig’09. (pp. 451–456). IEEE.
Lin, J. H., Pouliquen, P. O., Goldberg, A. C., Rizk, C. G., & Andreou, A. G. (2011). Roska, T., & Chua, L. O. (1993). The CNN universal machine: an analogic array
A bio-inspired event-driven architecture with pixel-level A/D conversion and computer. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal
non-uniformity correction. In Proceedings of the 45th annual conference on Processing, 40(3), 163–173.
information sciences and systems, CISS (pp. 1–6). Rumelhart, D. E., McClelland, J. L., & Group, P. R. (1987). Parallel distributed
Liu, S.-C., & Mockel, R. (2008). Temporally learning floating-gate VLSI synapses. In processing, foundations—Vol. 1. The MIT Press.
Proceedings of the 2008 IEEE international symposium on circuits and systems, Saighi, S., Bornat, Y., Tomas, J., Le Masson, G., & Renaud, S. (2010). A library of analog
ISCAS (pp. 2154–2157). operators based on the Hodgkin–Huxley formalism for the design of tunable,
Lotze, N., & Manoli, Y. (2012). A 62 mV 0.13 µm CMOS standard-cell-based design real-time, silicon neurons. IEEE Transactions on Biomedical Circuits and Systems,
technique using Schmitt-trigger logic. IEEE Journal of Solid-State Circuits, 47(1), 5(1), 3–19.
Schemmel, J., Grubl, A., Meier, K., & Mueller, E. (2006). Implementing synaptic
47–60.
Mahowald, M. (1992). VLSI analogs of neuronal visual processing: a synthesis plasticity in a VLSI spiking neural network model. In Proceedings of the 2006
of form and function. Ph.D. Thesis, Ph.D. Dissertation, California Institute of international joint conference on neural networks, IJCNN (pp. 1–6).
Technology. Schreiber, S., Machens, C. K., Herz, A. V. M., & Laughlin, S. B. (2002). Energy-efficient
Mandolesi, P. S., Julian, P., & Andreou, A. G. (2004). A scalable and programmable coding with discrete stochastic events. Neural Computation, 14(6), 1323–1346.
Serrano-Gotarredona, T., Andreou, A. G., & Linares-Barranco, B. (1999). AER
simplicial CNN digital pixel processor architecture. IEEE Transactions on Circuits
image filtering architecture for vision-processing systems. IEEE Transactions on
and Systems I: Regular Papers, 51(5), 988–996.
Mandolesi, P. S., Julian, P., & Andreou, A. G. (2006). A simplicial CNN visual processor Circuits and Systems I: Fundamental Theory and Applications, 46(9), 1064–1071.
Serrano-Gotarredona, T., Oster, M., Lichtsteiner, P., Linares-Barranco, A., Paz-
in 3D SOI-CMOS. In Proceedings of the 2006 IEEE international symposium on
Vicente, R., Gomez-Rodriguez, F., et al. (2009). CAVIAR: A 45 k neuron, 5 M
circuits and systems, ISCAS (pp. 1311–1314).
synapse, 12 G connects/s AER hardware sensory-processing-learning-actuating
Markram, H. (2011). Human brain project. [Link]
system for high-speed visual object recognition and tracking. IEEE Transactions
Markram, H., Gerstner, W., & Sjöström, P. J. (2011). A history of spike-timing-
on Neural Networks, 20(9), 1417–1438.
dependent plasticity. Frontiers in Synaptic Neuroscience, 3(4), 1–24.
Silver, R., Boahen, K. A., Grillner, S., Kopell, N., & Olsen, K. L. (2007). Neurotech for
Marr, D. (1982). Vision: a computational investigation into the human representation neuroscience: unifying concepts, organizing principles, and emerging tools. The
and processing of visual information. W.H. Freeman and Company. Journal of Neuroscience, 27(44), 11807–11819.
Martin, M. N., Pouliquen, P. O., Andreou, A. G., & Fraeman, M. E. (1996). Current- Sivilotti, M. A. (1991). Wiring considerations in analog VLSI systems, with
mode differential logic circuits for low power digital systems. In Proceedings of application to field programmable networks. Ph.D. Thesis, Ph.D. Dissertation,
the 39th midwest symposium on circuits and systems, MWSCAS (pp. 183–186). California Institute of Technology.
McClelland, J. L., Rumelhardt, D. R., & Group, P. R. (1987). Parallel distributed Song, S., & Abbott, L. F. (2000). Competitive Hebbian learning through spike-timing-
processing, psychological and biological models—Vol. 2. The MIT Press. dependent synaptic plasticity. Nature Neuroscience, 3(9), 919–926.
Mead, C. A. (1989). Analog VLSI and neural systems. Addison-Wesley Publishers. Stanacevic, M., & Cauwenberghs, G. (2005). Micropower gradient flow acoustic
Mead, C. A. (1990). Neuromorphic electronic systems. Proceedings of the IEEE, localizer. IEEE Transactions on Circuits and Systems I: Regular Papers, 52(10),
78(10), 1629–1636. 2148–2157.
26 A.S. Cassidy et al. / Neural Networks 45 (2013) 4–26

Stepanyants, A., Hof, P. R., & Chklovskii, D. B. (2002). Geometry and structural Wawrzynek, J., Asanovic, K., Kingsbury, B., Johnson, D., Beck, J., & Morgan, N. (1996).
plasticity of synaptic connectivity. Neuron, 34(2), 275–288. Spert-II: a vector microprocessor system. IEEE Computer, 29(3), 79–86.
Strukov, D. B., Snider, G. S., Stewart, D. R., & Williams, R. S. (2008). The missing Wawrzynek, J., Asanovic, K., & Morgan, N. (1993). The design of a neuro-
memristor found. Nature, 453(7191), 80–83. microprocessor. IEEE Transactions on Neural Networks, 4(3), 394–399.
Swindale, N. V., Shoham, D., Grinvald, A., Bonhoeffer, T., & Hübener, M. (2000). Wen, Q., & Chklovskii, D. B. (2008). A cost-benefit analysis of neuronal morphology.
Visual cortex maps are optimized for uniform coverage. Nature Neuroscience, Journal of Neurophysiology, 99(5), 2320–2328.
3(8), 822–826. Wen, Q., Stepanyants, A., Elston, G. N., Grosberg, A. Y., & Chklovskii, D. B. (2009).
Tezzaron, (2011). Tezzaron semiconductor. [Link]. February. Maximization of the connectivity repertoire as a statistical principle governing
Turing, A. M. (1952). The chemical basis of morphogenesis. Philosophical the shapes of dendritic arbors. Proceedings of the National Academy of Sciences
Transactions of the Royal Society of London: Series B, Biological Sciences, 237(641), of the United States of America, 106(30), 12536–12541.
37–72. Wijekoon, J. H. B., & Dudek, P. (2006). Simple analogue VLSI circuit of a cortical
Varshney, L. R., Sjöström, P. J., & Chklovskii, D. B. (2006). Optimal informa- neuron. In Proceedings of the 13th IEEE international conference on electronics,
tion storage in noisy synapses under resource constraints. Neuron, 52(3), circuits, and systems, ICECS (pp. 1344–1347).
409–423. Wolff, L. B., & Andreou, A. G. (1995). Polarization camera sensors. Image and Vision
Vittoz, E. A. (2005). Weak inversion for ultimate low-power logic. In Low-power Computing, 13, 497–510.
CMOS circuits: technology, logic design and CAD tools (pp. 1–18). CRC Press. Xilinx, (2011). [Link]. February.
Vogelstein, R. J., Mallik, U., Culurciello, E., Cauwenberghs, G., & Etienne-Cummings, Yu, T., Sejnowski, T. J., & Cauwenberghs, G. (2011). Biophysical neural spiking,
R. (2007). A multichip neuromorphic system for spike-based visual information bursting, and excitability dynamics in reconfigurable analog VLSI. IEEE
processing. Neural Computation, 19(9), 2281–2300. Transactions on Biomedical Circuits and Systems, 5(5), 420–429.