Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Zero-configuration networking wikipedia , lookup
Computer network wikipedia , lookup
Piggybacking (Internet access) wikipedia , lookup
Cracking of wireless networks wikipedia , lookup
Network tap wikipedia , lookup
Airborne Networking wikipedia , lookup
List of wireless community networks by region wikipedia , lookup
NN 5 Associative Memories HOP • We consider now NN models for unsupervised learning problems, called auto-association problems. • Association is the task of mapping patterns to patterns. • In an associative memory the stimulus of an incomplete or corrupted pattern leads to the response of a stored pattern that corresponds in some manner to the input pattern. • A neural network model most commonly used for (auto-) association problems is the Hopfield network. NN 5 1 Example Corrupted image • States • Attractors • Input • Output Intermediate state of Hopfield net Output Bit maps. Prototype patterns. An arbitrary pattern (e.g. picture with noise). The best prototype for that pattern. NN 5 Neural Networks HOP 2 1 NN 5 HOP HOPFIELD NETWORKS • The Hopfield network implements a so-called content addressable memory. • A collection of patterns called fundamental memories is stored in the NN by means of weights. • Each neuron represents a component of the input. • The weight of the link between two neurons measures the correlation between the two corresponding components over the fundamental memories. If the weight is high then the corresponding components are often equal in the fundamental memories. NN 5 3 HOP ARCHITECTURE: recurrent z-1 Multiple-loop feedback system with no self-feedback z-1 z-1 unit-delay operator NN 5 Neural Networks 4 2 NN 5 HOP Hopfield discrete NN • Input vectors values are in {-1,1} (or {0,1}). • The number of neurons is equal to the input dimension. • Every neuron has a link from every other neuron (recurrent architecture) except itself (no self-feedback). • The neuron state at time n is its output value. • The network state at time n is the vector of neurons states. • The activation function used to update a neuron state is the sign function but if the input of the activation function is 0 then the new output (state) of the neuron is equal to the old one. • Weights are symmetric: w = w ij ji NN 5 5 HOP Notation • N: input dimension. • M: number of fundamental memories. f µ i i-th component of the µ fundamental • memory. • xi (n ) State of neuron i at time n. NN 5 Neural Networks 6 3 NN 5 HOP Weights computation 1. Storage. Let f1, f2, … , fM denote a known set of Ndimensional fundamental memories. The synaptic weights of the network are: 1 w ji = M 0 M fµ ∑ µ =1 ,i j ≠ i, j = 1.......N fµ, j j =i where wji is the weight from neuron i to neuron j. The elements of the vectors fµ are in {-1,+1}. Once computed, the synaptic weights are fixed. NN 5 7 HOP NN Execution 2. Initialisation. Let x probe denote an input vector (probe) presented to the network. The algorithm is initialised by setting: x j (0) = x probe , j j = 1, ... , N where xj(0) is the state of neuron j at time n = 0, and x probe,j is the j-th element of the probe vector x probe. 3. Iteration Until Convergence. Update the elements of network state vector x(n) asynchronously (i.e. randomly and one at the time) according to the rule N x j (n + 1) = sign ∑ w ji xi (n ) i =1 j = 1, 2, ... , N Repeat the iteration until the state vector x remains unchanged. 4. Outputting. Let x fixed denote the fixed point (or stable state, that is such that x(n+1)=x(n)) computed at the end of step 3. The resulting output y of the network is: y = x fixed Neural Networks NN 5 8 4 NN 5 HOP Example 1 (-1, 1, 1) weight 1 (1, 1, -1) + - (1, 1, 1) (-1, 1, -1) + - - 2 3 (-1, -1, 1) - (1, -1, 1) neuron (-1, -1, -1) stable states (1, -1, -1) -1 1 -1 -1 -1 -1 1 1 -1 -1 1 1 1 -1 -1 -1 -1 1 1 1 1 attraction basin 1 attraction basin 2 1 -1 1 NN 5 9 HOP Example 2 • Separation of patterns using the two fundamental memories (-1 -1 -1) ( 1 1 1): • Find weights to obtain the following behavior: -1 -1 -1 -1 -1 1 -1 1 -1 1 -1 -1 -1 1 1 1 -1 1 1 1 -1 1 2 1 1 1 3 wij = ? NN 5 Neural Networks 10 5 NN 5 HOP CONVERGENCE • Every stable state x is at an “energy minimum”. (A state x is stable if x(n+1)=x(n)) What is “energy”? Energy is a function (Lyapunov function): E: States → such that every firing (change of its output value) of a neuron decreases the value of E: x → x’ implies E(x’) < E(x) E ( x ) = − 12 x T W x = − 12 ∑ w ji x i x j i, j NN 5 11 HOP CONVERGENCE • A stored pattern establishes a correlation between pairs of neurons: neurons tend to have the same state or opposite state according to their values in the pattern. • If wji is large, this expresses an expectation that neurons i and j are positively correlated similarly. If it is small (very negative) this indicates a negative correlation. • ∑w x x will thus be large for a state x, which is a stored pattern (since wji will be positive if xi xj > 0 and negative if xi xj < 0). ij i j i, j • The negative of the sum will thus be small. NN 5 Neural Networks 12 6 NN 5 HOP Energy Decreases Claim: Firing a neuron decreases the energy E. Proof: 1. Let l be the neuron that fires: Either: xl goes from -1 to 1 which implies N ∑w i =1 2. x >0 (above threshold) li i xl goes from 1 to -1 which implies N ∑w i =1 x <0 (below threshold) li i N Thus in both cases: ( xl '− xl ) ∑ wli xi > 0 i =1 value after NN 5 13 HOP Proof Now we try to “factor” xl from the energy function E: N N E = − 12 ∑ ∑ w ji x i x j i =1 j =1 N N pulled out the i = l case = − 12 ∑ x i ∑ w ji x j i =1 can be added since wll = 0 j =1 N N N j =1 j ≠l i =1 i ≠l j =1 N N N N j =1 j ≠l i =1 i ≠l i =1 i ≠l j =1 j ≠l = − 12 xl ∑ w jl x j − 12 ∑ x i ∑ w ji x j = − 12 xl ∑ w jl x j − 12 ∑ x i wli xl − 12 ∑ x i ∑ w ji x j N = − xl ∑ w jl x j j =1 j ≠l + NN 5 Neural Networks Term independent of xl 14 7 NN 5 HOP Proof How does E change with xl changing? E = value of E before change E ' = value of E after change N N j =1 j ≠l j =1 j ≠l E ' − E = − x l ' ∑ w j l x j + x l ∑ w jl x j N = − ( xl ' − x l ) ∑ w j l x j j =1 j ≠l we showed in previous slide that this quantity is > 0 (without ‘-’ sign) So E’-E < 0 always. NN 5 15 HOP Convergence result • We have shown that the energy E decreases with each neuron firing. • The overall number of states is finite ⊆ {+1, -1}N. Then • The energy cannot decrease forever. • Firing cannot continue forever. NN 5 Neural Networks 16 8 NN 5 HOP Computer experiment • We illustrate the behavior of the discrete Hopfield network as a content addressable memory. • n = 120 neurons (⇒n2 - n = 14280 weights). • The network is trained to retrieve 8 black and white patterns. Each pattern contains 120 pixels. The inputs of the net assume value +1 for black pixels and -1 for white pixels. • Retrieval 1: the fundamental memories are presented to the network to test its ability to recover them correctly from the information stored in the weights. In each case, the desired pattern was produced by the network after 1 iteration. NN 5 17 Computer experiment These patterns are used as fundamental memories to create the weight matrix. NN 5 Neural Networks 18 9 NN 5 Computer experiment • Retrieval 2: to test error-correcting capability of the network. A pattern of interest is distorted by randomly and independently reversing each pixel of the pattern with a probability of 0.25. The distorted pattern is used as a probe for the network. • The average number of iterations needed to recall, averaged over the eight patterns, is about 31. The net behaves as expected. NN 5 19 Computer experiment NN 5 Neural Networks 20 10 NN 5 Computer experiment NN 5 21 Spurious attractor states • Problem of incorrect retrieval. For example the network with input corrupted pattern “2” converges to “6”. • Spurious attractor states: the next slide shows 108 spurious attractors found in 43097 tests of randomly selected digits corrupted with the probability of flipping a bit at 0.25. NN 5 Neural Networks 22 11 NN 5 Spurious attractor states Local minima of E not correlated with any of the fundamental memories Combination of digit 1, digit 4 and digit 9 NN 5 23 Storage Capacity • Storage capacity: the quantity of information that can be stored in a network in such a way that it can be retrieved correctly. • Definition of storage capacity: C= number of fundamenta l patterns number of neurons in the network C= number of fundamental patterns number of weights in the network NN 5 Neural Networks 24 12 NN 5 Storage Capacity: bounds • Theorem: The maximum storage capacity of a discrete Hopfield NN is bounded above by C ≡ M 1 = N 4 ln N • That is, if β is the probability that the j-th bit of the m-th fundamental memory is correctly retrieved for all j =1,..,N and m = 1,…,M, then lim N → ∞ β = 1 whenever M < N/ ( 4 ln N ) NN 5 25 Hopfield NN for Optimization • Optimization problems (like Traveling Salesman) can be encoded into Hopfield Networks • Objective function corresponds to the energy of the network • Good solutions are stable states of the network NN 5 Neural Networks 26 13 NN 5 TSP • Travelling Salesman Problem (TSP): Given N cities with distances dij . What is the shortest tour? NN 5 27 Encoding • Construct a Hopfield network with N2 nodes. • Semantics: nia = 1 iff town i is visited at step a • Constraints: ∑n ia ∑n = 1, ∀a ia i = 1, ∀i a The town distancies are encoded by weights, i.e. w ijab = d ij NN 5 Neural Networks 28 14 NN 5 Hopfield Network for TSP place city 1 2 3 4 A 0 1 0 0 B 1 0 0 0 C D 0 0 0 0 0 1 1 0 Configuration not allowed Configuration allowed NN 5 29 Energy and Weights E=− 1 ∑ wijab nia n jb 2 i , j , a ,b • Nodes within each row ( i=j ) connected with weights wijab = −γ • Nodes within each column ( a=b ) connected with weights wijab = −γ • Each node is connected to nodes in columns left and right with weight wijab = − d ij NN 5 Neural Networks 30 15