Download Experiment 1

Information Processing
with Pulsed Neural Networks
Ulrich Ramacher
Corp. Research, Infineon Technologies AG, Munich
[email protected]
Prof. Ramacher, CPR ST
6-May-17 Page 1
Dilemma (1)
Prof. Ramacher, CPR ST
6-May-17 Page 2
lack of robustness
Dilemma (2)
Prof. Ramacher, CPR ST
6-May-17 Page 3
lack of architecture information
Dilemma (3)
• Non-invasive long-term recording of neural tissue.
• High-density sensor with 128128 Sensors in 11mm².
• Extended CMOS-process with biocompatible high-k
surface dielectric.
• Self-calibration circuitry and pre-amplification on chip.
• Applications in neurobiology and drug discovery
Sensor [mV]
Neuron [mV]
Infineon Neurochip
20µm
Time [ms]
Prof. Ramacher, CPR ST
6-May-17 Page 4
Gap between signal processing by few cells
and information processing by large arrays
Program
1. Use pulsed IAF neurons and adaptive synapses
(inhibitory, excitatory)
2. Use experimental evidence
3. Start from simple, complete vision systems
4. Find basic network structures for information processing
5. Find quantitative ansatz for information processing
6. Demonstrate usefulness for cell phones
Prof. Ramacher, CPR ST
6-May-17 Page 5
Neuron Model


a k ( t )  a k ( t0 )  
Wkl (t )  X l (t )  Wk 0  ik (t )dt 


t lS ( t )

t
k  R ( t0 ) :
 
0
or :
0
dak / dt  c  ak (t ) 
W
kl (t )  X l (t )  Wk 0  ik (t )
lS ( t0 )
Rule 1: Reaching the threshold, a neuron resets membrane potential to zero
and starts sending a pulse of 1 ms
Rule 2: A neuron either receives or sends
 deterministic dynamical system
Prof. Ramacher, CPR ST
6-May-17 Page 6
Experiment 1
Prof. Ramacher, CPR ST
6-May-17 Page 7
Frequency
Experiment 1
Prof. Ramacher, CPR ST
6-May-17 Page 8
Observations
• „Firing patterns don‘t come to an end“
• information processing is not a function of time

Does frequency of firing patterns characterize the net ?
Inet = - pn x ln pn
Is entropy a reproducible measure of information?
Prof. Ramacher, CPR ST
6-May-17 Page 9
Experiment 2:
Prof. Ramacher, CPR ST
6-May-17 Page 10
Experiment 2
Prof. Ramacher, CPR ST
6-May-17 Page 11
Findings
• distribution function of fp‘s is independent of initial conditions
 Inet = -  pn x ln pn
is a reproducibly measurable quantity
Prof. Ramacher, CPR ST
6-May-17 Page 12
Entropy as a function of mean synaptic weight
Each color
corresponds to
a different
realization of wij
Network size:
40 neurons
Prof. Ramacher, CPR ST
6-May-17 Page 13
Network size: 40 neurons
ISI
Prof. Ramacher, CPR ST
6-May-17 Page 14
Fire Rate
Synapse Model
(proposed by U. Ramacher, April 99)
k  E (t0 ), l  S (t0 ) :
else :



Wkl   Wkl     ak (t )     ( X l )
2

W    W
kl
kl
Adaptation rule : local, causal, simple
Wkl   Wkl   Wkl (t )  X l 



   Wkl ' (t )  X l '  Wk 0  ik (t )  ak (t 0)  
2
 l 'S (t0 )
Prof. Ramacher, CPR ST
6-May-17 Page 15
µ negative
• 20 x 24 pixels , each connected to a neuron by a constant synapse
• adaptive synapses connecting neighboured neurons
Prof. Ramacher, CPR ST
6-May-17 Page 16
synapses = coupled system of damped oscillators
µ positive
20x24 pixel, 10% noise
Prof. Ramacher, CPR ST
6-May-17 Page 17
synapses = coupled system of damped
exponentially rising and falling „elements“
Spot Detector = Illumination Encoder
Prof. Ramacher, CPR ST
6-May-17 Page 18
Pixels: 64 x 64
Prof. Ramacher, CPR ST
6-May-17 Page 19
Projection onto the receptive field
of a simple cell in the visual cortex
LGN
primary
visual
cortex (V1)
light
retina
Dr. Arne Heittmann
Prof. Ramacher, CPR ST
6-May-17 Page 20
stimulus
optic
nerve
convergence
Dr. Arne Heittmann
Prof. Ramacher, CPR ST
6-May-17 Page 21
Modeling the Experiment
Bright Stimulus
Dark Stimulus
y0
Dy
x0
Dx
Stimulus
Stimuli projected onto RF of a Simple Cell
 ( x0 , y0 )  ( H  B) 
Prof. Ramacher, CPR ST
6-May-17 Page 22
: measured pulse rate Gi: Gabor function
 G ( x, y)dxdy
i
( x0 , x0  Dx )( y0 , y0  Dy )
H: Spot Intensity B: Background Intensity
Pulse difference detector (1)
i1>i2


W KL   WKL     aK (t )     ( X L )
2

Prof. Ramacher, CPR ST
6-May-17 Page 23
i1<i2
Pulse difference detector (2)
Dynamics of the Synapse W41:
dWKL
   (WKL  W )    ( I pre  I ) WKL   ( X L )
dt
  0 , Ipre  Ig  0 , WKL  0
Prof. Ramacher, CPR ST
6-May-17 Page 24
Pulse difference detector (3)
Number of pulses
Characteristic
Neuron 2
Neuron 1
Neuron 4
i2
Prof. Ramacher, CPR ST
6-May-17 Page 25
i1 = 0.5
Q =1
WK0 = 0.08
td = 1ms
T = 0.5s
Architecture of feature detector
(proposed by A. Heittmann, 2003)
1
1
1
3
1
---
3
2
3
4
3‘ 3‘ 3‘
3
4
4
2
4‘ 4‘ 4‘
4
2
---
2
3‘
4‘
5‘
5
6
µ>0
µ<0
Prof. Ramacher, CPR ST
6-May-17 Page 26
constant
Shaping the response-profile of the detector
1

i
 ( x, y)dxdy
N i1 ( x0 , y0 ) 
( x0 , x0  Dx )( y0 , y0  Dy )
N i2 ( x0 , y0 ) 
2

i
 ( x, y )dxdy
( x0 , x0  Dx )( y0 , y0  Dy )
section of the retina
( x0 , y0 ) 
WK 0
1
2
 ( H  B)( Ni ( x0 , y0 )  Ni ( x0 , y0 ))
Q
 ( H  B) 
 G ( x, y)dxdy
i
 i1 ( x, y )  i2 ( x, y ) 
Prof. Ramacher, CPR ST
6-May-17 Page 27
H: Spot Intensity B: Background Intensity
Q
Gi ( x, y )
WK 0
Results of a detector implementation
Gabor-Wavelet
Prof. Ramacher, CPR ST
6-May-17 Page 28
measured profile
- 256 Gradient detectors
- size of receptive field: 17 x 17 Pixel
- T=750ms @ 1ms Pulse-duration
ideal
Prof. Ramacher, CPR ST
6-May-17 Page 29
Simulated Filter responses, T=750ms
Real
Imaginary
Real
Imaginary
90°
90°
0°
0°
The Head-Detector
Restriction
- single scale (keep eye-distance fixed)
Prof. Ramacher, CPR ST
6-May-17 Page 30
Check for Robustness in Eye-Brow Zone
Reference Image
Filter Response,
horizontal direction
region of
interest
20x20 Pixel
Prof. Ramacher, CPR ST
6-May-17 Page 31
new eye-brow
image
A simple memory (1) , 1 zone
Learning Phase

2

W KL  WKL     a K     ( X l )  exp   K (t )   L (t )  /  2 (t )
2


Detector Layer
K
Prof. Ramacher, CPR ST
6-May-17 Page 32
Input Layer

Memory Layer
WKK
K
L
 K (t )   K , M
K
1-1 connection between input and detector layer
1-1 connection between input and memory layer
full connection between detector and memory layer
Experiment 1: learned image in input and memory
Activity (number of events), 2ms window
Pulse-patterns of input-layer
Prof. Ramacher, CPR ST
6-May-17 Page 33
A simple memory (2), 1 Zone
Recognition Phase

2

W KL  WKL     a K     ( X l )  exp   K ,M   L (t )  /  2 (t )
2


Detector Layer
K
Prof. Ramacher, CPR ST
6-May-17 Page 34
Gabor Layer

Memory Layer
WKK
K
L
 K ,M
K
1-1 connection between input and detector layer
full connection between detector and memory layer
Experiment 3: recognition of non-learned eye-brow
Activity (number of events), 2ms window
Pulse-patterns of input-layer
Prof. Ramacher, CPR ST
6-May-17 Page 35
Experiment 2: Isolated neurons
Activity (number of events), 2ms window
Pulse-patterns of input-layer
Prof. Ramacher, CPR ST
6-May-17 Page 36
Zone-Architecture I
Detector Layer 1
Horizontal Orientations
F 1 ( x, y)
Detector Layer 2
Memory Layer
Vertical Orientations
F 2 ( x, y )
F j ( x0 , y0 )   I x, y   G j ( x  x0 , y  y0 )
x, y
G j ( x, y ) : Gabor-Kernel
I x, y 
Prof. Ramacher, CPR ST
6-May-17 Page 37
I x, y 
: Input Image
Zone-Architecture II
Zone 5
Zone 6
Zone 1
Zone 7
Zone 2
Zone 3
Zone 4
Zuordnung: Zonen zu Bildregionen
: Zone für Gabor-Wavlet mit horizontaler Orientierung
Prof. Ramacher, CPR ST
6-May-17 Page 38
: Zone für Gabor-Wavlet mit vertikaler Orientierung
Reference-Image and Test-Images
Reference-Image:
Test-Images:
Prof. Ramacher, CPR ST
6-May-17 Page 39
Face 0001
Face 0011
Face 0014
Face 0001
Normalized accumulated activity
Activity-Diagram
Memory
Zone 1
Detector
Memory
Zone 2
Detector
Memory
Detector
Zone 3
Memory
Detector
Zone 4
Memory
Detector
Time [ms]
Prof. Ramacher, CPR ST
6-May-17 Page 40
Zone 5
Face 0011
Normalized accumulated activity
Activity-Diagram
Memory
Zone 1
Detector
Memory
Zone 2
Detector
Memory
Detector
Zone 3
Memory
Detector
Zone 4
Memory
Detector
Time [ms]
Prof. Ramacher, CPR ST
6-May-17 Page 41
Zone 5
Face 0014
Normalized accumulated activity
Activity-Diagram
Memory
Zone 1
Detector
Memory
Zone 2
Detector
Memory
Detector
Zone 3
Memory
Detector
Zone 4
Memory
Detector
Time [ms]
Prof. Ramacher, CPR ST
6-May-17 Page 42
Zone 5
Binding of zones by synchrony
Binding Layer
…


W KL  WKL     aK      X L 
2

1
3
2
4
6
W zone i  1/  zone i K ,M
5
7
 zone4 K ,M
 zone3 K ,M
Zuordnung: Zonen zu Neuronen
des Spotdetektors
zur Bindung
Prof. Ramacher, CPR ST
6-May-17 Page 43
 zone2 K ,M
 zone1K ,M
 zone5 K ,M
Memory Layer
Results
~ 40ms
Spot_0001
~ 40ms
Spot_0011
Prof. Ramacher, CPR ST
6-May-17 Page 44
~ 40ms
Spot_0014
Column Architecture
Image plane
Detector Feature 1
Memory Feature 1
Zone 1
Detector Feature 2
Memory Feature 2
robust
recognition
Zone 2
.
.
.
Detector Feature n
Memory Feature n
Binding Object 1
Associative
Memory
Binding Object 2
Binding Object n
Prof. Ramacher, CPR ST
6-May-17 Page 45
Zone n
∙
∙
∙
∙
∙
∙
∙
The vision: a 3D-Vision-Cube
The Vision-Cube
• 3D-stacking-architecture
• Low-power
• Real time capabilities
• Integration of sensors and
information processing
• Distributes layers of
information processing
to layers of the stack
• Solves problem of
connectivity
Prof. Ramacher, CPR ST
6-May-17 Page 46
CMOS-sensor,
sensor array
Analogue-pulse
conversion
Feature-Detection
Gabor-Wavelets,
different orientations
…
Object-recognition
Object-detection
Design of a testchip for the „Synchrony detector“,
Base-Chip for the 3D-Stack
Infineon 130nm CMOS-Technology
pixel memory
(1 Pixel)
local
AER-subcircuit
testcircuits for 3Dinterconnects
Test circuit
Integrate-andFire-Neuron
3D-vias for supply and
digital control signals
adaptive
synapses
layout of 1
neuron
rows of 3D-vias for
layer-to-layer signal
distribution
array of
128 x 128 neurons,
64k synapses
control circuit
AER-encodercircuit
Prof. Ramacher, CPR ST
6-May-17 Page 47
Power consumption:
3mW @1.5V (analog) , 40-250mW @1.5V (digital)
Size: 7.6mm x 7.8mm
Gabor-Feature-Detector chip (layout) including a pulse
router for the 3D-integration
Infineon 130nm CMOS-Technology
Array of 128x128 (64k)
processing elements for
pulse processing and routing
Processing element:
- photo-Detector
- pulse-processing (gradient
detection)
- dynamic routing of pulses
3D- wiring channel for
vertical
signal distribution
SRAM for storing routing
information
Digital macro:
- routing circuit
- configuration
- AER-circuit (event based)
Prof. Ramacher, CPR ST
3D- wiring
6-May-17 Page 48
for power supply
and control signals
3D-Stacking
Chip 3
SOLID
Connection
Chip 2
15 µm
7 µm
Chip 1
14 µm
Layer 7
Layer 6
Layer 5
12 µm
12µm
12µm
Prof. Ramacher, CPR ST
6-May-17 Page 49
Bottom-Chip
Real 3D !!
Layer 7
Layer 6
Layer 5
Layer 4
Layer 3
Layer 2
Bottom Substrate, Layer 1
Prof. Ramacher, CPR ST
6-May-17 Page 50
13µm
7µm
Conclusion
• IAF neurons, adaptive synapses, only
• network built for
-- Gabor wavelet based feature cascade
-- memory
-- comparison of memory and detector plane
• synchrony of neurons indicative for
-- robust recognition of memory feature at detector plane
(elastic matching)
-- binding of features as object
• built in 130 nm CMOS
-- synchrony detector
-- Gabor wavelet detector
-- 3 D stack of 7 silicon chips
Prof. Ramacher, CPR ST
6-May-17 Page 51