Download Artificial Spiking Neural Networks

Functional Link Network
Support Vector Machines
2
1
2
2
( x , x , 2 x1 x2 )
Support Vector Machines
  x1   y1  
( x  y )        
  x2   y2  
2
2
  x12   y12  

 

   2 x1 x2    2 y1 y2  
 




2
2
x
y
2
2





 (( x)  ( y ))
where ( x)  ( x , 2 x1 x2 , x )
2
1
2
2
margin
separator
support vectors
Support Vector Machines
( x1 , y1 ),, ( xl , yl )    {1}
N
(w  x)  b  0, w   , b  
N
f ( x)  sign (( w  x)  b)
Support Vector Machines
yi (( w  xi )  b)  0
yi (( w  xi )  b)  1
minimize | w |
Support Vector Machines
Support Vector Machines
w  i vi xi for the support ve ctors xi
f ( x)  sign (i vi ( x  xi )  b)
Support Vector Machines
 :  F
N
k ( x, y ) : (( x)  ( y ))
Support Vector Machines
l
f ( x)  sign ( vi  k ( x, xi )  b)
i 1
l
f ( x)   vi  k ( x, xi )  b
i 1
Support Vector Machines
Support Vector Machines
k ( x, y)  ( x  y )
d
k ( x, y )  exp(  x  y /( 2 ))
2
k ( x, y )  tanh(  ( x  y )  )
2
Two Spiral Problem
SVM architecture
Application: text classification
•
•
•
•
Reuters “newswire” messages
Bag-of-words representation
Dimension reduction
Training SVM
Results
Break-even point = precision value at which precision and recall are nearly equal
Results
Application 2: face recognition
False detections
System architecture
Results
Results
Skin detection and real-time recognition
Neural Networks
Ccortex is a massive spiking neuron network
emulation and will mimic the human cortex,
the outer layer of gray matter at the
cerebral hemispheres, largely responsible
for higher brain functions. The emulation
covers up to 20 billion layered neurons and
2 trillion 8-bit connections.
Spiking Neural Networks
• From neurones to neurons
• Artificial Spiking Neural Networks
(ASNN)
– Dynamic Feature Binding
– Computing with spike-times
Neural Networks
• Artificial Neural Networks
– (neuro)biology -> Artificial Intelligence (AI)
– Model of how we think the brain processes
information
• New data on how the brain works!
– Artificial Spiking Neural Networks
Real Neurons
• Real cortical neurons communicate with
spikes or action potentials
Real Neurons
• The artificial sigmoidal neuron models the rate
at which spikes are generated
• artificial neuron computes function of weighted
input:
xj
xj = f(wij xi )
wijxi
Artificial Neural Networks
• Artificial Neural Networks can:
– approximate any function
• (Multi-Layer Perceptrons)
– act as associative memory
• (Hopfield networks, Sparse Distributed Memory)
– learn temporal sequences
• (Recurrent Neural Networks)
ANN’s
• BUT....
for understanding the brain the neuron model is
wrong
• individual spikes are important, not just rate
Binding Problem
• When humans view a scene containing a red circle
and a green square, some neurons
–
–
–
–
signal the presence of red,
signal the presence of green,
signal the circle shape,
Signal the square shape.
• The binding problem:
– how does the brain represent the pairing of color and
shape?
• Specifically, are the circles red or green?
Binding
• Synchronizing spikes?
New Data!
• neurons belonging to same percept tend to
synchronize (Gray & Singer, Nature 1987)
• timing of (single) spikes can be remarkably
reproducible
• Spikes are rare: average brain activity < 1Hz
– “rates” are not energy efficient
Computing with Spikes
• Computing with precisely timed spikes is
more powerful than with “rates”.
(VC dimension of spiking neuron models)
[W. Maass and M. Schmitt., 1999]
• Artificial Spiking Neural Networks??
[W. Maass Neural Networks, 10, 1997]
Artificial Spiking Neuron
• The “state” (= membrane potential) is a
weighted sum of impinging spikes
– spike generated when potential crosses threshold, reset
potential
Artificial Spiking Neuron
• Spike-Response Model:
where ε(t) is the kernel describing how a single
spike changes the potential:
(1
-t/)
P
S
P
: te


Artificial Spiking Neural Network
• Network of spiking neurons:
Error-backpropagation in ASNN
• Encode “X-OR” in (relative) spike-times
XOR in ASNN
• Change weights according to gradient
descent using error-backpropagation (Bohte et
al, Neurocomputing 2002)
• Also effective for unsupervised learning
(Bohte etal, IEEE Trans Neural Net. 2002)
Oil Application