Download Associative Memories Example

NN 5
Associative Memories
HOP
• We consider now NN models for unsupervised learning
problems, called auto-association problems.
• Association is the task of mapping patterns to patterns.
• In an associative memory the stimulus of an incomplete or
corrupted pattern leads to the response of a stored pattern
that corresponds in some manner to the input pattern.
• A neural network model most commonly used for (auto-)
association problems is the Hopfield network.
NN 5
1
Example
Corrupted image
• States
• Attractors
• Input
• Output
Intermediate state of
Hopfield net
Output
Bit maps.
Prototype patterns.
An arbitrary pattern
(e.g. picture with noise).
The best prototype for that pattern.
NN 5
Neural Networks
HOP
2
1
NN 5
HOP
HOPFIELD NETWORKS
• The Hopfield network implements a so-called content
addressable memory.
• A collection of patterns called fundamental memories is
stored in the NN by means of weights.
• Each neuron represents a component of the input.
• The weight of the link between two neurons measures
the correlation between the two corresponding
components over the fundamental memories. If the
weight is high then the corresponding components are
often equal in the fundamental memories.
NN 5
3
HOP
ARCHITECTURE: recurrent
z-1
Multiple-loop feedback system
with no self-feedback
z-1
z-1
unit-delay operator
NN 5
Neural Networks
4
2
NN 5
HOP
Hopfield discrete NN
• Input vectors values are in {-1,1} (or {0,1}).
• The number of neurons is equal to the input dimension.
• Every neuron has a link from every other neuron (recurrent
architecture) except itself (no self-feedback).
• The neuron state at time n is its output value.
• The network state at time n is the vector of neurons states.
• The activation function used to update a neuron state is the
sign function but if the input of the activation function is 0
then the new output (state) of the neuron is equal to the old
one.
• Weights are symmetric:
w = w
ij
ji
NN 5
5
HOP
Notation
• N: input dimension.
• M: number of fundamental memories.
f µ i i-th component of the µ fundamental
•
memory.
• xi (n ) State of neuron i at time n.
NN 5
Neural Networks
6
3
NN 5
HOP
Weights computation
1. Storage. Let f1, f2, … , fM denote a known set of Ndimensional fundamental memories. The synaptic weights of
the network are:
1

w ji =  M
0

M
fµ
∑
µ
=1
,i
j ≠ i, j = 1.......N
fµ, j
j =i
where wji is the weight from neuron i to neuron j. The elements
of the vectors fµ are in {-1,+1}. Once computed, the synaptic
weights are fixed.
NN 5
7
HOP
NN Execution
2. Initialisation. Let x probe denote an input vector (probe) presented to the
network. The algorithm is initialised by setting:
x j (0) = x probe , j
j = 1, ... , N
where xj(0) is the state of neuron j at time n = 0, and x probe,j is the j-th
element of the probe vector x probe.
3. Iteration Until Convergence. Update the elements of network state vector
x(n) asynchronously (i.e. randomly and one at the time) according to the rule
N

x j (n + 1) = sign ∑ w ji xi (n )
 i =1

j = 1, 2, ... , N
Repeat the iteration until the state vector x remains unchanged.
4.
Outputting. Let
x fixed
denote the fixed point (or stable state, that is such
that x(n+1)=x(n)) computed at the end of step 3. The resulting output y of the
network is:
y = x fixed
Neural Networks
NN 5
8
4
NN 5
HOP
Example 1
(-1, 1, 1)
weight
1
(1, 1, -1)
+
-
(1, 1, 1)
(-1, 1, -1)
+
-
-
2
3
(-1, -1, 1)
-
(1, -1, 1)
neuron
(-1, -1, -1)
stable states
(1, -1, -1)
-1 1 -1
-1 -1 -1
1 1 -1
-1 1 1
1 -1 -1
-1 -1 1
1 1 1
attraction basin 1
attraction basin 2
1 -1 1
NN 5
9
HOP
Example 2
• Separation of patterns using the two fundamental memories
(-1 -1 -1) ( 1 1 1):
• Find weights to obtain the following behavior:
-1 -1 -1
-1 -1 1
-1 1 -1
1 -1 -1
-1 1 1
1 -1 1
1 1 -1
1
2
1 1 1
3
wij = ?
NN 5
Neural Networks
10
5
NN 5
HOP
CONVERGENCE
• Every stable state x is at an “energy minimum”. (A
state x is stable if x(n+1)=x(n))
What is “energy”?
Energy is a function (Lyapunov function):
E: States → such that every firing (change of its output value) of a
neuron decreases the value of E:
x → x’ implies E(x’) < E(x)
E ( x ) = − 12 x T W x
= − 12 ∑ w ji x i x j
i, j
NN 5
11
HOP
CONVERGENCE
• A stored pattern establishes a correlation between pairs of
neurons: neurons tend to have the same state or opposite
state according to their values in the pattern.
• If wji is large, this expresses an expectation that neurons i
and j are positively correlated similarly. If it is small (very
negative) this indicates a negative correlation.
•
∑w x x
will thus be large for a state x, which is a stored
pattern (since wji will be positive if xi xj > 0 and negative if xi
xj < 0).
ij i
j
i, j
• The negative of the sum will thus be small.
NN 5
Neural Networks
12
6
NN 5
HOP
Energy Decreases
Claim:
Firing a neuron decreases the energy E.
Proof:
1.
Let l be the neuron that fires:
Either:
xl goes from -1 to 1 which implies
N
∑w
i =1
2.
x >0
(above threshold)
li i
xl goes from 1 to -1 which implies
N
∑w
i =1
x <0
(below threshold)
li i
N
Thus in both cases: ( xl '− xl ) ∑ wli xi > 0
i =1
value after
NN 5
13
HOP
Proof
Now we try to “factor” xl from the energy function E:
N N
E = − 12 ∑ ∑ w ji x i x j
i =1 j =1
N
N
pulled out the i = l case
= − 12 ∑ x i ∑ w ji x j
i =1
can be added
since wll = 0
j =1
N
N
N
j =1
j ≠l
i =1
i ≠l
j =1
N
N
N
N
j =1
j ≠l
i =1
i ≠l
i =1
i ≠l
j =1
j ≠l
= − 12 xl ∑ w jl x j − 12 ∑ x i ∑ w ji x j
= − 12 xl ∑ w jl x j − 12 ∑ x i wli xl − 12 ∑ x i ∑ w ji x j
N
= − xl ∑ w jl x j
j =1
j ≠l
+
NN 5
Neural Networks
Term independent of xl
14
7
NN 5
HOP
Proof
How does E change with xl changing?
E = value of E before change
E ' = value of E after change
N
N
j =1
j ≠l
j =1
j ≠l
E ' − E = − x l ' ∑ w j l x j + x l ∑ w jl x j
N
= − ( xl ' − x l ) ∑ w j l x j
j =1
j ≠l
we showed in previous slide that this quantity is > 0
(without ‘-’ sign)
So E’-E < 0 always.
NN 5
15
HOP
Convergence result
• We have shown that the energy E decreases with each
neuron firing.
• The overall number of states is finite ⊆ {+1, -1}N.
Then
• The energy cannot decrease forever.
• Firing cannot continue forever.
NN 5
Neural Networks
16
8
NN 5
HOP
Computer experiment
• We illustrate the behavior of the discrete Hopfield network as a content
addressable memory.
• n = 120 neurons (⇒n2 - n = 14280 weights).
• The network is trained to retrieve 8 black and white patterns. Each
pattern contains 120 pixels. The inputs of the net assume value +1 for
black pixels and -1 for white pixels.
• Retrieval 1: the fundamental memories are presented to the network to
test its ability to recover them correctly from the information stored in
the weights. In each case, the desired pattern was produced by the
network after 1 iteration.
NN 5
17
Computer experiment
These patterns are used as fundamental
memories to create the weight matrix.
NN 5
Neural Networks
18
9
NN 5
Computer experiment
• Retrieval 2: to test error-correcting capability of the
network. A pattern of interest is distorted by
randomly and independently reversing each pixel
of the pattern with a probability of 0.25. The
distorted pattern is used as a probe for the
network.
• The average number of iterations needed to recall,
averaged over the eight patterns, is about 31. The
net behaves as expected.
NN 5
19
Computer experiment
NN 5
Neural Networks
20
10
NN 5
Computer experiment
NN 5
21
Spurious attractor states
• Problem of incorrect retrieval. For example the network with
input corrupted pattern “2” converges to “6”.
• Spurious attractor states: the next slide shows 108 spurious
attractors found in 43097 tests of randomly selected digits
corrupted with the probability of flipping a bit at 0.25.
NN 5
Neural Networks
22
11
NN 5
Spurious attractor states
Local minima of E not
correlated with any of
the fundamental
memories
Combination of digit
1, digit 4 and digit 9
NN 5
23
Storage Capacity
• Storage capacity: the quantity of information that can be
stored in a network in such a way that it can be retrieved
correctly.
• Definition of storage capacity:
C=
number of fundamenta l patterns
number of neurons in the network
C=
number of fundamental patterns
number of weights in the network
NN 5
Neural Networks
24
12
NN 5
Storage Capacity: bounds
• Theorem: The maximum storage capacity of a discrete
Hopfield NN is bounded above by
C ≡
M
1
=
N
4 ln N
• That is, if β is the probability that the j-th bit of the m-th
fundamental memory is correctly retrieved for all j =1,..,N
and m = 1,…,M, then
lim N → ∞ β = 1 whenever M < N/ ( 4 ln N )
NN 5
25
Hopfield NN for Optimization
• Optimization problems (like Traveling Salesman)
can be encoded into Hopfield Networks
• Objective function corresponds to the energy of the
network
• Good solutions are stable states of the network
NN 5
Neural Networks
26
13
NN 5
TSP
• Travelling Salesman Problem (TSP):
Given N cities with distances dij .
What is the shortest tour?
NN 5
27
Encoding
• Construct a Hopfield network with N2 nodes.
• Semantics: nia = 1 iff town i is visited at step a
• Constraints:
∑n
ia
∑n
= 1, ∀a
ia
i
= 1, ∀i
a
The town distancies are encoded by weights, i.e.
w ijab = d ij
NN 5
Neural Networks
28
14
NN 5
Hopfield Network for TSP
place
city
1
2
3
4
A
0
1
0
0
B
1
0
0
0
C
D
0
0
0
0
0
1
1
0
Configuration not
allowed
Configuration allowed
NN 5
29
Energy and Weights
E=−
1
∑ wijab nia n jb
2 i , j , a ,b
• Nodes within each row ( i=j ) connected with
weights wijab = −γ
• Nodes within each column ( a=b ) connected
with weights wijab = −γ
• Each node is connected to nodes in columns
left and right with weight wijab = − d ij
NN 5
Neural Networks
30
15