Download NEURAL NETWORKS AND FUZZY SYSTEMS

NEURONAL DYNAMICS 2:
ACTIVATION MODELS
姓 名：王莹
学
号：0620110201
时
间：2006-10-09
3.1 NEURONAL DYNAMICAL SYSTEM
Neuronal activations change with time. The way
they change depends on the dynamical equations
as following:

x  g（FX ，FY ，
）
(3-1)

y  h（FX ，FY ，
）
(3-2)
2006.10.9
3.2 ADDITIVE NEURONAL DYNAMICS

first-order passive decay model
In the absence of external or neuronal stimuli,
the simplest activation dynamics model is:

x i   xi

y
j
 yj
(3-3)
(3-4)
2006.10.9
3.2 ADDITIVE NEURONAL DYNAMICS
since for any finite initial condition
xi (t )  xi (0)e
t
The membrane potential decays
exponentially quickly to its zero potential.
2006.10.9
3.2 ADDITIVE NEURONAL DYNAMICS

Passive Membrane Decay
Passive-decay rate
Ai  0 scales the rate to
the membrane’s resting potential.

xi   Ai xi
solution :
xi(t)  xi( 0 )e
-Ai t
Passive-decay rate measures: the cell membrane’s
resistance or “friction” to current flow.
2006.10.9
property
Pay attention to Ai property
The larger the passive-decay rate,the faster the
decay--the less the resistance to current flow.
3.2 ADDITIVE NEURONAL DYNAMICS

Membrane Time Constants
The membrane time constant C i scales the time
variable of the activation dynamical system.
The multiplicative constant model:

C i x i  -A i x i
（3－8）
2006.10.9
Solution and property
solution
x i (t )  xi (0)e

Ai
t
Ci
property
The smaller the capacitance ,the faster things change
As the membrane capacitance increases toward
positive infinity,membrane fluctuation slows to stop.
3.2 ADDITIVE NEURONAL DYNAMICS

Membrane Resting Potentials
Definition
Define resting Potential Pi as the activation value
to which the membrane potential equilibrates in the
absence of external or neuronal inputs:

C i x i  -A i x i  Pi
(3-11)
Solutions
x i (t)  x i (0)e
-
Ai
t
Ci
Pi

(1 - e
Ai
-
Ai
t
Ci
)
(3-12)
2006.10.9
Note
The time-scaling capacitance dose not affect
the asymptotic or steady-state solution.
The steady-state solution does not depend on the
finite initial condition.
In resting case,we can find the solution quickly.
3.2 ADDITIVE NEURONAL DYNAMICS

Additive External Input
Add input
Apply a relatively constant numeral input to a neuron.

x i  -x i  I i
(3-13)
solution
x i (t)  x i (0)e-t  I i (1- e -t )
(3-14)
Meaning of the input
Input can represent the magnitude of directly
experiment sensory information or directly apply
control information.
The input changes slowly,and can be assumed
constant value.
3.3 ADDITIVE NEURONAL FEEDBACK
Neurons do not compute alone. Neuron modify their
state activations with external input and with the feedback
from one another.

This feedback takes the form of path-weighted signals
from synaptically connected neurons.

3.3 ADDITIVE NEURONAL FEEDBACK

Synaptic Connection Matrices

n neurons in field FX
p neurons in field FY
The ith neuron axon in FX
jth neurons in FY
a synapse m ij
m ij is constant,can be positive,negative or zero.
Meaning of connection matrix
The synaptic matrix or connection matrix M is an
n-by-p matrix of real number whose entries are the
synaptic efficacies m ij.the ijth synapse is excitatory
if m ij  0 inhibitory if m ij  0

The matrix M describes the forward projections from
neuron field FX to neuron field FY

The matrix N describes the feedforward projections
from neuron field FY to neuron field F

X
The neural network can be specified by the 4-tuple
(M, N, FX , FY )

3.3 ADDITIVE NEURONAL FEEDBACK

Bidirectional and Unidirectional connection Topologies

Bidirectional networks
M and N have the same or approximately the same
structure. N  M T
M  NT


Unidirectional network
A neuron field synaptically intraconnects to itself.
BAM : Bidirectional associative memories
M is symmetric, M  M T the unidirectional
network is BAM
2006.10.9
Augmented field and augmented matrix
Augmented field
FX
FY
FZ  

FX

FY 
M connects FX to FY ,N connects FY to FX then the
augmented field FZ intraconnects to itself by the square
block matrix B
0
B  
N
M

0
2006.10.9
Augmented field and augmented matrix
In the BAM case,when N  M then B  BT hence
a BAM symmetries an arbitrary rectangular matrix M.
T
In the general case,
P
C  
N
M

Q
P is n-by-n matrix.
Q is p-by-p matrix.
T
P  PT Q  QT the neurons
If and only if, N  M
in FZ are symmetrically intraconnected
C  CT
2006.10.9
3.4 ADDITIVE ACTIVATION MODELS

Define additive activation model
n+p coupled first-order differential equations defines the
additive activation model

p

j 1
n
x i  -Ai x i   S j ( y j )n ji  I i （3－15）
y j  -Aj y j   Si ( xi )mij  J j
（3－16）
j 1
2006.10.9
additive activation model define
The additive autoassociative model correspond to a
system of n coupled first-order differential equations


n
x i  -Ai x i   S j ( x j )m ji  I i
（3－17）
j 1
2006.10.9
additive activation model define
A special case of the additive autoassociative
model


xi
Ci x i   
Ri

j
x j  xi
rij
 Ii
xi
  '   S j ( x j )mij  I i
Ri
j
where Ri'
is
n
1
1
1
 
'
Ri Ri
j rij
(3-18)
(3-19)
（3-20）
rij
measures the cytoplasmic resistance between
neurons i and j.
2006.10.9
Hopfield circuit and continuous additive BAM
Hopfield circuit arises from if each neuron has a strictly
increasing signal function and if the synaptic connection
matrix is symmetric


xi
Ci xi   '   S j ( x j )mij  I i
Ri
j

(3-21)
continuous additive bidirectional associative memories

p

j 1
n
x i  -Ai x i   S j ( y j )mij  I i
y j  -A j y j   Si ( xi )mij  J j
(3-22)
(3-23)
i 1
2006.10.9
3.5 ADDITIVE BIVALENT MODELS
Discrete additive activation models correspond to
neurons with threshold signal function

The neurons can assume only two value: ON and
OFF.
ON represents the signal value +1.
OFF represents 0 or –1.

Bivalent models can represent asynchronous and
stochastic behavior.

Bivalent Additive BAM

BAM-bidirectional associative memory
Define a discrete additive BAM with threshold signal
functions, arbitrary thresholds and inputs,an arbitrary but
constant synaptic connection matrix M,and discrete time
steps k.

p
xik 1   S j ( y kj )mij  I i
(3-24)
j 1
p
y kj 1   Si ( xik )mij  I j
(3-25)
i 1
2006.10.9
Bivalent Additive BAM

Threshold binary signal functions
 1

k
Si ( xi )  Si ( xik 1 )
 0

 1

k
S j ( y j )  S j ( y kj 1 )
 0

if
if
xik  U i
xik  U i
if
xik  U i
if
if
y kj  V j
y kj  V j
if
y kj  V j
(3-26)
(3-27)
For arbitrary real-value thresholds U  U1 , , U n 
for neurons FX V  V1 , , V p for neurons FY



2006.10.9
A example for BAM model
Example
A 4-by-3 matrix M represents the forward synaptic
projections from FX to FY .
A 3-by-4 matrix MT represents the backward synaptic
projections from FY to FX .

2
 3 0


 1 2 0 
M 
0
3
2


  2 1  1


  3 1 0  2


T
M  0 2 3 1 
 2

0
2

1


A example for BAM model


Suppose at initial time k all the neurons inFY are ON.
So the signal state vectorS (Yk ) at time k corresponds to
S (Yk )  (1 1 1)

Input
X k  ( x , x , x , x ,)  (5  2 3 1)
k
1

k
2
k
3
k
4
Suppose
Ui  V j  0
2006.10.9
A example for BAM model

is:
first:at time k+1 through synchronous operation,the result
S ( X k )  (1 0 1 1)
next:at time k+1 ,these FX signals pass “forward” through
the filter M to affect the activations of the FY neurons.

The three neurons compute three dot products,or
correlations.The signal state vector S ( X k ) multiplies each of
the three columns of M.

2006.10.9
A example for BAM model

the result is:
4
S ( X k ) M  (
i 1
Si ( xik )mi1 ,
 (5
 ( y1k 1
4
3)
y2k 1
4

i 1
Si ( xik )mi 2 ,
k
S
(
x
 i i )mi3 )
i 1
y3k 1 )
 Yk 1

4
synchronously compute the new signal state vector S (Yk 1 ):
S (Yk 1 )  (0 1 1)
A example for BAM model

the signal vector passes “backward” through the synaptic
filter S (Yk 1 ) at time k+2:
S (Yk 1 )M T  (2  2 5 0)
 ( x1k 2
 X k 2

x2k 2
x3k 2
x4k 2 )
synchronously compute the new signal state vector
S ( X k  2 )  (1 0 1 1)  S ( X k )
:
A example for BAM model
since S ( X k 2 )  S ( X k ) then

S (Yk 3 )  S (Yk 1 )
conclusion
These same two signal state vectors will pass back and
forth in bidirectional equilibrium forever-or until new
inputs perturb the system out of equilibrium.
A example for BAM model
asynchronous state changes may lead to different
bidirectional equilibrium

keep the first FY neurons ON,only update the second
and third FY neurons. At k,all neurons are ON.

Yk 1  S ( X k ) M  ( 5 4 3)

new signal state vector at time k+1 equals:
S (Yk 1 )  (1 1 1)
A example for BAM model

new FX
activation state vector equals:
X k 2  S (Yk 1 ) M T  ( 1  1 5  2)

synchronously thresholds
S ( X k 2 )  (0 0 1 0)

passing this vector forward to FY gives
Yk 3  S ( X k 2 ) M  (0 3 2)
S (Yk 3 )  (1 1 1)
 S (Yk 1 )
A example for BAM model

Similarly, S ( X k 4 )  S ( X k 2 )  (0
0 1 0)
for any asynchronous state change policy we apply to the
neurons FX
The system has reached a new equilibrium,the binary
pair (0 0 1 0), (1 1 1) represents a fixed point of the
system.

Conclusion
Different subset asynchronous state change policies
applied to the same data need not product the same fixedpoint equilibrium. They tend to produce the same equilibria.


All BAM state changes lead to fixed-point stability.
Bidirectional Stability

Definition
A BAM system ( Fx , F y , M ) is Bidirectional stable if all
inputs converge to fixed-point equilibria.

A denotes a binary n-vector in

B denotes a binary p-vector in
0,1n
0,1p
Bidirectional Stability

Represent a BAM system equilibrates to bidirectional fixed
point
Af
) as
 M
 MT
 M
 MT

 M
Af

( Af , B f
A
A'
A'
A ''
MT





B
B
B'
B'

Bf

Bf
Lyapunov Functions

Lyapunov Functions L maps system state variables to real
numbers and decreases with time. In BAM case,L maps the
Bivalent product space to real numbers.

Suppose L is sufficiently differentiable to apply the chain
rule:

L
n

i
L dxi

xi dt

i
L 
xi
xi
(3-28)
Lyapunov Functions

The quadratic choice of L
1
1
T
L  xIx 
2
2

x i2
（3-29）
i
Suppose the dynamical system describes the passive
decay system.


xi   xi

（3-30）
The solution
xi (t )  xi (0)e t
（3-31）
Lyapunov Functions

The partial derivative of the quadratic L:
L
 xi
xi

L

i
（3-32）

xi2 （3-33） or L  


i
2
xi
(3-34)
(3-35)
In either case L  0

(3-36)
At equilibrium
L0
This occurs if and only if all velocities equal zero

xi  0
Conclusion
A dynamical system is stable if some Lyapunov
Functions L decreases along trajectories.

A dynamical system is asymptotically stable if it
strictly decreases along trajectories

Monotonicity of a Lyapunov Function provides a
sufficient condition for stability and asymptotic stability.

Linear system stability
For symmetric matrix A and square matrix B,the quadratic
T
L

xAx
form
behaves as a strictly decreasing Lyapunov

function for any linear dynamical system x  xB if and
only if the matrix ABT  BA is negative definite.
T

L  xA x  x AxT
 xAB T x T  xBAx T
 x[ AB T  BA]x T
The relations between convergence rate
and eigenvalue sign
A general theorem in dynamical system theory relates
convergence rate and ergenvalue sign:

A nonlinear dynamical system converges
exponetially quickly if its system Jacobian
has eigenvalues with negative real parts.
Locally such nonlinear system behave as linearly.
Bivalent BAM theorem
The average signal energy L of the forward pass of theFX
Signal state vector S ( X ) through M,and the backward pass
Of the FY signal state vector S (Y ) through M T :
S ( X ) MS (Y ) T  S (Y ) MS ( X ) T
L
(3-46)
2
since S (Y ) M T S ( X ) T  [S (Y ) M T S ( X )T ]T
T
 S ( X )M S (Y )
L  S ( X )M S (Y )
T
n

p
 S ( x )S ( y )m
i
i
(3-47)
j
i
j
j
ij
(3-48)
Lower bound of Lyapunov function
The signal is Lyapunov function clearly bounded below.
For binary or bipolar,the matrix coefficients define the
attainable bound:
L
mij

i
j
The attainable upper bound is the negative of this expression.
Lyapunov function for the general BAM system
The signal-energy Lyapunov function for the general BAM
system takes the form
L  S ( X ) MS(Y )T  S ( X )[I  U ]T  S (Y )[J  V ]T
Inputs I  [ I1 , , I N ] and J  [ J 1 , , J P ] and
constant vectors of thresholds U  [U1 , , U N ] V  [V1 , , VN ]
the attainable bound of this function is.
L
 m  [ I
ij
i
j
i
i
 Ui ] 
[ J
j
j
V j ]
Bivalent BAM theorem
Bivalent BAM theorem.every matrix is bidrectionally stable
for synchronous or asynchronous state changes.
Proof consider the signal state changes that occur from
time k to time k+1,define the vectors of signal state
changes as:
S (Y )  S (Yk 1 )  S (Yk )
 S1 ( y1 ), , S p ( y p ) ,
S ( X )  S ( X k 1 )  S ( X k )
 S1 ( x1 ), , S n ( x n ) ,
Bivalent BAM theorem
define the individual state changes as:
S j ( y j )  S j ( y j k 1 )  S j ( y j k )
Si ( xi )  Si ( xi
k 1
)  S i ( xi )
k
We assume at least one neuron changes state from k
to time k+1.
Any subset of neurons in a field can change state,but in
only one field at a time.
For binary threshold signal functions if a state change
is nonzero,
Bivalent BAM theorem
Si ( xi )  1  0  1 Si ( xi )  0  1  1
For bipolar threshold signal functions
S i ( xi )  2
Si ( xi )  2
The “energy”change
L  Lk 1  Lk
L
Differs from zero because of changes in field FX or in
field FY
Bivalent BAM theorem
L  S ( X ) MS(Yk )T  S ( X )[I  U ]T
 S ( X )[S (Yk ) M T  [ I  U ]]T

 S ( x ) I   S ( x )U
  S ( x ) S ( y ) m   S ( x ) I   S ( x )U
   S ( x )[  S ( y ) m  I  U ]
  Si ( xi )[ xik 1  U i ]

S i ( xi ) S j ( y kj ) T mij 
i
j
i
i
i
i
i
i
i
k T
i
i
i
ij
j
j
i
i
0
j
i
k T
i
j
j
j
i
i
i
i
i
i
ij
i
i
i
i
i
Bivalent BAM theorem
Suppose S i ( xi )  0
Then Si ( xi )  Si ( xi
k 1
)  Si ( xi k )
 1 0
k 1
This implies xi  U i so the product is positive:
Si ( xi )[xik 1  U i ]  0
Another case suppose S i ( xi )  0
Si ( xi )  Si ( xi
k 1
 0 1
)  Si ( xi )
k
Bivalent BAM theorem
k 1
This implies xi
 Ui
so the product is positive:
Si ( xi )[xik 1  U i ]  0
So Lk 1  Lk  0 for every state change.
Since L is bounded,L behaves as a Lyapunov function for
the additive BAM dynamical system defined by before.
Since the matrix M was arbitrary,every matrix is
bidirectionally stable. The bivalent Bam theorem is proved.
Property of globally stable dynamical system
Two insights about the rate of convergence
First,the individual energies decrease nontrivially.the BAM
system does not creep arbitrary slowly down the toward the
nearest local minimum.the system takes definite hops into
the basin of attraction of the fixed point.
Second,a synchronous BAM tends to converge faster
than an asynchronous BAM.In another word, asynchronous
updating should take more iterations to converge.