Download Overview of Neuromorphic Computing Chris Carothers, CCI Director

Overview of Neuromorphic Computing
Chris Carothers, CCI Director
Jim Hendler, IDEA Director
Rensselaer Polytechnic Institute
Office of Research
Outline
• Background
• Biological Neuron Structure
• From Bio Neuron to Silicon Neuron
• IBM TrueNorth Architecture
• AFRL: Hybrid Neuromorphic
Supercomputers
• Future Research Areas
Office of Research
2
Structure of a Real Neuron (from KSJ 4e, 2001)
Most neurons in the vertebrate nervous system have several
main features in common. The cell body contains the nucleus,
the storehouse of genetic information, and gives rise to two types
of cell processes, axons and dendrites. Axons, the transmitting
element of neurons, can vary greatly in length; some can extend
more than 3 m within the body. Most axons in the central
nervous system are very thin (between 0.2 and 20 µm in
diameter) compared with the diameter of the cell body (50 µm or
more). Many axons are insulated by a fatty sheath of myelin that
is interrupted at regular intervals by the nodes of Ranvier. The
action potential, the cell's conducting signal, is initiated either at
the axon hillock, the initial segment of the axon, or in some
cases slightly farther down the axon at the first node of Ranvier.
Branches of the axon of one neuron (the presynaptic neuron)
transmit signals to another neuron (the postsynaptic cell) at a site
called the synapse. The branches of a single axon may form
synapses with as many as 1000 other neurons. Whereas the
axon is the output element of the neuron, the dendrites
(apical and basal) are input elements of the neuron. Together
with the cell body, they receive synaptic contacts from other
neurons.
Office of Research
3
Neuron Classification (from KSJ 4e, 2000)
Neurons can be classified as unipolar, bipolar, or
multipolar according to the number of processes that
originate from the cell body.
A. Unipolar cells have a single process, with different
segments serving as receptive surfaces or releasing terminals.
Unipolar cells are characteristic of the invertebrate nervous
system.
B. Bipolar cells have two processes that are functionally
specialized: the dendrite carries information to the cell, and
the axon transmits information to other cells.
C. Certain neurons that carry sensory information, such as
information about touch or stretch, to the spinal cord belong to
a subclass of bipolar cells designated as pseudo-unipolar. As
such cells develop, the two processes of the embryonic
bipolar cell become fused and emerge from the cell body as a
single process. This outgrowth then splits into two processes,
both of which function as axons, one going to peripheral skin
or muscle, the other going to the central spinal cord.
D. Multipolar cells have an axon and many dendrites. They
are the most common type of neuron in the mammalian
nervous system. Three examples illustrate the large diversity
of these cells. Spinal motor neurons (left) innervate skeletal
muscle fibers. Pyramidal cells (middle) have a roughly
triangular cell body; dendrites emerge from both the apex (the
apical dendrite) and the base (the basal dendrites). Pyramidal
cells are found in the hippocampus and throughout the
cerebral cortex. Purkinje cells of the cerebellum (right) are
characterized by the rich and extensive dendritic tree in one
plane. Such a structure permits enormous synaptic input.
Office of Research
4
From Bio Neurons to Silicon Neurons (Hasler/Marr ‘13)
Office of Research
5
IBM TrueNorth Architecture (Cassidy et al)
Neuromorphic core: 256 axons, 256 neurons and 256x256 synapses
(a)
(b)
(c)
Fig. 3. TrueNorth architecture at core, chip, and multi-chip scale. (a) The building block is a neurosynaptic core, where horizontal lines are axons (inputs),
the “ square-end half-circle” symbol denotes axon delay buffers, cross points are individually programmable synapses, vertical lines are neuron inputs, and
triangles are neurons (outputs). Neurons behaviors are individually programmable with two examples shown. (b) Cores naturally tile using a 2D on-chip
mesh routing network. Long-range connections between neurons and axons are implemented by sending spike events (packets) over the mesh network. (c)
Individual chips also tile in 2D, with the routing network extending across chip boundaries through peripheral merge and split blocks.
represented by a bit Ai (t). Although each synaptic connection rons to destination axons. For additional efficiency, Compass
Architecture
richa enough
support both
network/compute
is binary,
it can mediate
multi-valuedtopost-synaptic
effect. neural
aggregates
spikes between pairs ofalgorithms
processes into aand
single MPI
neural biologically
relevant
Specifically,
each axon i is assigned
to behaviors
one of four types message; overlaps communication with computation; uses an
Gi , which corresponds to a weight specified individually for innovative synchronizati on scheme requiring just two commueach neuron. For example, axon–neuron connections can be nication steps regardless of the number of the processors; uses
set to be excitatory or inhibitory, and each with different meticulous load-balancing; and uses highly compressed data
synaptic strengths. Mathematically, at time t, neuron j Office
receives
structures for maintaining neuron and synapses states. These
of Research
Gi
Gi
6 all 6.3 million threads
input: Ai (t) ⇥ Wi, j ⇥ Sj (Listing 1, line 7), where Sj is a advances enabled Compass to exercise
IBM TrueNorth (alt. view Cassidy et al )
Office of Research
7
IBM TrueNorth Neuron Model (Cassidy et al)
Note: Vj(t) is a 20 bit signed integer
Office of Research
8
IBM TrueNorth Lib & Programming (Cassidy et al and Amir et al)
SC
ble
cka
Sta
C
LSM
Liquid State Machine
S
Scalable Classifier
C
C
p2 2
c
S
MCR
Corelet Library
ts
r
ifie
s
s
Cla
Averaging
Ad d New Corele
SC
Convolution
Filter
.
..
c3
L
C
L
Classifier
Module
Convolution
Network
C
p3
hine
Mac
e
t
a
Square-W ave
Filter
Stackable
Classifier
Summation
let
s
SC
Spectral Content
Estimator
Use Existing Co re
M
LS
Speaker
Identification
ier
ssif
Cla
LSM
MCR Corelet
LSM
SC
Liquid State Machine
SC
Scalable Classifier
C
S
L
C
C
Corelet
Laboratory
a2
2
M
ap
e
cor
pu
t
p4
c4
id
iqu
Transduction
To
Spikes
p1
e
cor
1
1n
c1
Sensors
Network Model
C
In
R
MC
et
rel
Co
COMPASS
Simulator
O
ut
pu
tM
ap
Transduction
From
Spikes
Visualization
Actuators
ocesses of composition and decomposition are illustrated in
Fig. 7. The Corelet Laboratory. We depict the complete development cycle
signment of the destination address a2 , cor e2 to neuron
with all the tools. The Corelet Library is a hierarchical library of all corelets
he TrueNorth program constructed by the Corelet MCR.
created by developers. The Corelet development process typically involves
es place during the recursive corelet construction. Corelet
creating and inheriting from the Corelet Library, composing sub-corelets
reating Corelet LSM, which in turn creates Corelets L and
together into new corelets, and submitting the new corelet into the library
or clarity). Corelet L, when created, connects output neuron
after verification. The new corelet then becomes available for use by other
n p1 , C 1 . Next, Corelet LSM connects connector C 1 to
applications, thus allowing composition. A concrete application is created by
necting p1 , C 1 and p2 , C 2 . Similarly, Corelet C, when Officeinstantiating
of Researchobjects from the created corelet class. The instantiated object
ects p4 , C 4 to input axon a2 , cor e2 . Next, Corelet SC
is then decomposed into a network model file
9 that represents a TrueNorth
TN’s 20 Biological Relevant Behaviors (Cassidy et al)
Office of Research
10
IBM TrueNorth Performance (Cassidy et al)
TN Peak
performance
is ~400G
SOPS/watt @
200
Hasler’s scale
indicates 1
ExaSOPS/watt to
reach brain
efficiencies
Analog SP is
at 1 TeraSOPS/watt
Office of Research
11
Other Related Projects
•
•
Human Brain Project
• SpiNNaker – Manchester UK, 18 ARM
chips, 250K neurons & 80M synapes in 36
watts.
• Blue Brain Project – EPFL, models all the
details of the human brain, simulation runs
on BG/Q at JSC in Germany
Qualcomm Zeroth NPU
•
•
•
Neurogrid – Stanford/Cornell
•
•
•
•
See Jeff Gehlhaar’s ASPLOS ‘14 keynote
Similar goals as TrueNorth, few specs available
Analog neuron
1M neurons & ~1B synaspe in 5 watts
Funded by NSF EMT program
HP’s Neuristor
•
Specialized transistor that emulates neuron
functionality more directly.
Office of Research
12
AFRL Proposal: Motivation
• Significant shift from parallelism across nodes to parallelism
within nodes (very “fat” nodes)
• Intra-node parallelism most exploit some sort of
“streaming/vector” GPU-like processing
• Potential to leave behind “asynchronous” data/compute
apps
• Fault tolerance & power (data movement) is a big challenge..
Office of Research
13
AFRL Project: Hybrid Neuromorphic Supercomputer
The question: how might a neuromorphic
“accelerator” type processor be used to
improve the application performance,
power consumption and overall system
reliability of future exascale systems?
Driven by the recent DOE SEAB report on high-performance
computing [22] which high-lights the neuromorphic architecture as
one that “is an emergent area for exploitation”.
Office of Research
14
Future Research Directions
•
•
Alt. Neuromorphic Architectures
• Increase degree of asynchronous spike event processing
• Increase core size, e.g., 1K, 4K or 8K neurons & axons
• Multi-cycle neurons and not single-cycle clock
• Mix of heterogeneous neuron cores
• Dataflow and not time-flow driven approach
• High “burst” clock rate? E.g., 1GHz neurons
Application areas:
• Neuromorphic cybersecurity
• Neuromorphic OS/run-time system
•
•
•
•
Fault tolerance is only one important function…
Neuromorphic Data Mining
Neuromorphic (Sensor) Network
Bio related apps (e.g., ultra fast MD sim  Anton)
Office of Research
15