Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Topic 1 Neural Networks OUTLINES Neural Networks Cerebellar Model Articulation Controller (CMAC) Applications References C.L. Lin & H.W. Su, “Intelligent control theory in guidance and control system design: an overview,” Proc. Natl. Sci. Counc. ROC (A), pp. 15-30 Ming-Feng Yeh 1-2 1. Neural Networks As you read these words you are using a complex biological neural network. You have a highly interconnected set of 1011 neurons to facilitate your reading, breathing, motion and thinking. In the artificial neural network, the neurons are not biological. They are extremely simple abstractions of biological neurons, realized as elements in a program or perhaps as circuits made of silicon. Ming-Feng Yeh 1-3 Biological Inspiration Human brain consists of a large number (about 1011) of highly interconnected elements (about 104 connections per element) called neurons. Three principle components are the dendrites, the cell body and the axon. The point of contact is called a synapse. Ming-Feng Yeh 1-4 Biological Neurons Dendrites(樹突): carry electrical into the cell body Cell Body(細胞體): sums and thresholds these incoming signals Axon(軸突): carry the signal from the cell body out to other neurons Synapse(突觸): contact between an axon of one cell and a dendrites of another cell Ming-Feng Yeh 1-5 Neural Networks Neural Networks: a promising new generation of information processing systems, usually operate in parallel, that demonstrate the ability to learn, recall, and generalize from training patterns or data. Artificial neural networks are collections of mathematical models that emulate some of the observed properties of biological nervous systems and draw on the analogies of adaptive biological learning. Ming-Feng Yeh 1-6 Basic Model ~ 1 s1 s2 node … sn y = f(s1,s2,…,sn) y output input A neural network is composed of four pieces: nodes, connections between the nodes, nodal functions, and a learning rule for updating the information in the network. Ming-Feng Yeh 1-7 Basic Model ~ 2 Nodes: a number of nodes, each an elementary processor (EP) is required. Connectivity: This can be represented by a matrix that shows the connections between the nodes. The number of nodes plus the connectivity define the topology of the network. In the human brain, all neurons are connected to about 104 other neurons. Artificial nets can range from totally connected to a topology where each node is just connected to its nearest neighbors. Ming-Feng Yeh 1-8 Basic Model ~ 3 Elementary processor functions: A node has inputs s1,…, sn and an output y, and the node generates the output y as a function of the inputs. A learning rule: There are two types of learning: Supervised learning: you have to teach the networks the “answers.” Unsupervised learning: the network figures out the answers on its own. All the learning rules try to embed information by sampling the environment. Ming-Feng Yeh 1-9 Perceptron Model Suppose we have a two class problem. If we can separate these classes with a straight line (decision surface), then they are separable. The question is, how can we find the best line, and what do we mean by “best.” In n dimensions, we have a hyperplane separating the classes. These are all decision surfaces. Another problem is that you may need more than one line to separate the classes. Ming-Feng Yeh 1-10 Decision Surfaces x x x x o o o Linearly separable classes Ming-Feng Yeh x x x x x x x o o o x o o o o Multi-line decision surface 1-11 Single Layer Perceptron Model x1 w1 f(x) output inputs xn wn xi: inputs to the node; y: output; net w x i i i wi: weights; : threshold value. The output y can be expressed as: y f ( wi xi ) i The function f is called the nodal (transfer) function and is not the same in every application Ming-Feng Yeh 1-12 Nodal Function 1 1 1 -1 Hard-limiter Ming-Feng Yeh Threshold function Sigmoid function 1-13 Single Layer Perceptron Model Two-input case: w1x1 + w2x2 = 0 If we use the hard limiter, then we could say that if the output of the function is a 1, the input vector belongs to class A. If the output is a –1, the input vector belongs to class B. XOR: caused the field of neural networks to lose credibility in the 1940’s. The perceptron model could not draw a line to separate the two classes given by the exclusive-OR. Ming-Feng Yeh 1-14 Exclusive OR problem ? (0,1) x o (0,0) Ming-Feng Yeh o (1,1) ? x (1,0) 1-15 Two-layer Perceptron Model x1 y1 w11 w’1 w12 w21 x2 w22 z w’2 y2 The outputs from the two hidden nodes are y1 f ( w11 x1 w12 x2 ) y2 f ( w21 x1 w22 x2 ) The network output is z f (w1 ' y1 w2 ' y2 ) Ming-Feng Yeh 1-16 Exclusive-XOR problem f input patterns 00 01 output patterns 0 0.5 1 11 0 +1 1.5 1 10 +1 hidden unit +1 input units x Ming-Feng Yeh -2 g +1 output unit y 1-17 Exclusive-XOR problem g = sgn (1·x + 1·y 1.5) f = sgn (1·x + 1·y 2g 0.5) input (0,0) g=0 f=0 input (0,1) g=0 f=1 input (1,0) g=0 f=1 input (1,1) g=1 f=0 Ming-Feng Yeh 1-18 Multilayer Network output patterns ok f k ( wkj o j ) k j wkj o j f j ( w jioi ) i internal representation units j wji oi f i (ii ) i Ming-Feng Yeh input patterns 1-19 Weight Adjustment Adjust weights by: wji(l+1) = wji(l) + wji where wji(l) is the weight from unit i to unit j at time l (or the lth iteration) and wji is the weight adjustment. The weight change may be computed by the delta rule: wji = j ii where is a trial-independent learning rate and j is the error at unit j: j = tj oj where tj is the desired output and oj is the actual output at output unit j. Repeat iterations until convergence. Ming-Feng Yeh 1-20 Generalized Delta Rule p w ji (t pj o pj )i pi pji pi t pj : the target output for jth component of the output pattern for pattern p. o pj : the jth element of the actual output pattern produced by the presentation of input pattern p. i pi : the value of the ith element of the input pattern. pj t pj o pj p wij : is the change to be made to the weight from the ith to the jth unit following presentation of pattern p. Ming-Feng Yeh 1-21 Delta Rule and Gradient Descent 1 E p (t pj o pj ) 2 : the error on input/output pattern p 2 j E E p : the overall measure of the error. We wish to show that the delta rule implements a gradient descent in E when units are linear. We will proceed by simply showing that E p w ji pji pi which is proportional to p wij as prescribed by the delta rule. Ming-Feng Yeh 1-22 Delta Rule & Gradient Descent When there are no hidden units it is easy to compute the relevant derivative. For this purpose we use the chain rule to write the derivative as the product of two parts: the derivative of the error with respect to the output of the unit times the derivative of the output with respect to the weight. E p E p o pj w ji o pj w ji The first part tells how the error changes with the output of the jth unit and the second part tells how much changing wji changes that output. Ming-Feng Yeh 1-23 Delta Rule & Gradient Descent E p o pj (t pj o pj ) pj no hidden units The contribution of unit j to the error is simply proportional to pj. Since we have linear units, o pj w jii pi . i From which we conclude that Thus, we have Ming-Feng Yeh E p w ji o pj w ji i pi. pji pi 1-24 Delta Rule and Gradient Descent Combining this with the observation that E p E w ji p w ji should lead us to conclude that the net change in wji after one complete cycle of pattern presentations is proportional to this derivative and hence that the delta rule implements a gradient descent in E. In fact, this is strictly true only if the values of the weights are not changed during this cycle. Ming-Feng Yeh 1-25 Delta Rule for Activation Functions in Feedforward Networks The standard delta rule essentially implements gradient descent in sum-squared error for linear activation functions. Without hidden units, the error surface is shaped like a bowl with only one minimum, so gradient descent is guaranteed to find the best set of weights. With hidden units, however, it is not so obvious how to compute the derivatives, and the error surface is not concave upwards, so there is the danger of getting stuck in local minimum. Ming-Feng Yeh 1-26 Delta Rule for Semilinear Activation Functions in Feedforward Networks The main theoretical contribution is to show that there is an efficient way of computing the derivatives. The main empirical contribution is to show that the apparently fatal problem of local minima is irrelevant in a wide variety of learning tasks. A semilinear activation function is one in which the output of a unit is a non-decreasing and differentiable function of the net total input, net pj w ji o pi where o =i if unit i is an input unit. i i i Ming-Feng Yeh 1-27 Delta Rule for Semilinear Activation Functions in Feedforward Networks Thus, a semilinear activation function is one in which o pj f j (net pj ) and f is differentiable and non-decreasing. To get the correct generalization of the delta rule, we must set p w ji E p w ji where E is the same sum-squared error function defined earlier. Ming-Feng Yeh 1-28 Delta Rule for Semilinear Activation Functions in Feedforward Networks As in the standard delta rule, it is to see this derivative as resulting from the product of two parts: One part reflecting the change in error as a function of the change in the net input to the unit and one part representing the effect of changing a particular weight on the net input. E p E p net pj w ji The second factor is Ming-Feng Yeh net pj w ji net pj w ji w ji w jk o pk o pi k 1-29 Delta Rule for Semilinear Activation Functions in Feedforward Networks Define pj Thus, E p w ji E p net pj pj o pi This says that to implement gradient descent in E we should make our weight changes according to p w ji pj o pi just as in the standard delta rule. The trick is to figure out what pj should be for each unit uj in the network. Ming-Feng Yeh 1-30 Delta Rule for Semilinear Activation Functions in Feedforward Networks Compute pj E p net pj The second factor: o pj E p o pj o pj net pj net pj f j ' (net pj ) which is simply the derivative of the function fj for the jth unit, evaluated at the net input netpj to that unit. Note: o pj f j (net pj ) To compute the first factor, we consider two cases. Ming-Feng Yeh 1-31 Delta Rule for Semilinear Activation Functions in Feedforward Networks First, assume that unit uj is an output unit of the network. In this case, it follows from the definition of Ep that E p o pj (t pj o pj ) Thus, pj (t pj o pj ) f j ' (net pj ) for any output unit uj. Ming-Feng Yeh 1-32 Delta Rule for Semilinear Activation Functions in Feedforward Networks If uj is not an output unit we use the chain rule to write E p net pk E p net k k pk E p net pk o pj k net pk o pj wki o pi i wkj pk wkj k Thus, pj f j ' (net pj ) pk wkj k whenever uj is not an output unit. Ming-Feng Yeh 1-33 Delta Rule for Semilinear Activation Functions in Feedforward Networks If uj is an output unit: pj (t pj o pj ) f j ' (net pj ) If uj is not an output unit: pj f j ' (net pj ) pk wkj k The above two equations give a recursive procedure for computing the ’s for all units in the network, which are then used to compute the weight changes in the network. p w ji pj o pi Ming-Feng Yeh 1-34 Delta Rule for Semilinear Activation Functions in Feedforward Networks The application of the generalized delta rule, thus, involves two phases. During the first phase the input is presented and propagated forward through the network to compute the output value opj for each unit. This output is then compared with the targets, resulting in an error signal pj for each output unit. Ming-Feng Yeh 1-35 Delta Rule for Semilinear Activation Functions in Feedforward Networks The second phase involves a backward pass through the network (analogous to the initial forward pass) during which the error signal is passed to each unit in the network and the appropriate weight changes are made. Ming-Feng Yeh 1-36 Ex: Function Approximation g p = 1 + sin --- p 4 t p e + 1-2-1 Network Ming-Feng Yeh 1-37 Network Architecture p Ming-Feng Yeh 1-2-1 Network a 1-38 Initial Values W1 0 = – 0.27 – 0.41 b 1 0 = – 0.48 W2 0 = 0.09 – 0.17 – 0.13 b2 0 = 0.48 3 Initial Network Response: a2 Network Response Sine Wave 2 1 0 -1 -2 Ming-Feng Yeh -1 0 p 1 2 1-39 Forward Propagation Initial input: 0 a = p = 1 Output of the 1st layer: 1 1 1 0 1 a = f W a + b = l ogsig –0.27 1 + – 0.48 = logsi g – 0.75 –0.41 a1 – 0.13 – 0.54 1 -------------------0.75 1 + e = = 0.321 1 0.368 -------------0.54 ------1+ e Output of the 2nd layer: 2 2 2 1 2 a = f W a + b = purelin ( 0.09 – 0.17 0.321 + 0.48 ) = 0.446 0.368 error: 2 e = t – a = 1 + sin --- p – a = 1 + sin --- 1 – 0.446 = 1.261 4 4 Ming-Feng Yeh 1-40 Transfer Func. Derivatives n d 1 e f 1 (n) n dn 1 e (1 e n ) 2 1 1 1 1 1 ( 1 a )( a ) n n 1 e 1 e f 2 (n) d (n) 1 dn Ming-Feng Yeh 1-41 Backpropagation E p (t pj o pj ) 2 p w ji j E p w ji E p net pj net pj w ji net pj pj w ji The second layer sensitivity: pj (t pj o pj ) f j ' (net pj ) δ 2 2F 2 (n 2 )(t a) 2[ f 2 (n 2 )]e 2 1 1.261 2.522 The first layer sensitivity: pj f j ' (net pj ) pk wkj 2 2 w ( 1 a )( a ) 0 1,1 1 1 1 2 T 2 δ F (n )( W ) δ δ 1 1 2 0 (1 a2 )( a2 ) w1, 2 0 (1 0.321) 0.321 0.09 2.522 0 (1 0.368) 0.368 0.17 1 1 0.0495 0 . 0997 Ming-Feng Yeh 1 1 k 1-42 Weight Update p w ji pj o pi Learning rate 0.1 W 2 (1) W 2 (0) s 2 (a1 )T 0.09 0.17 0.1[2.522]0.321 0.368 0.171 0.0772 b 2 (1) b 2 (0) s 2 [0.48] 0.1[2.522] [0.732] W1 (1) W1 (0) s1 (a 0 )T 0.27 0.0495 0.265 0.1 [1] 0 . 41 0 . 0997 0 . 420 0.48 0.0495 0.475 1 1 1 b (1) b (0) s 0.1 0 . 13 0 . 0997 0 . 140 Ming-Feng Yeh 1-43 Choice of Network Structure Multilayer networks can be used to approximate almost any function, if we have enough neurons in the hidden layers. We cannot say, in general, how many layers or how many neurons are necessary for adequate performance. Ming-Feng Yeh 1-44 Illustrated Example 1 i g p = 1 + sin ----- p 4 1-3-1 Network 3 3 2 2 1 1 0 0 -1 -2 0 i 1 1 2 -1 -2 3 3 2 2 1 1 0 0 -1 -2 Ming-Feng Yeh -1 -1 0 i4 1 2 -1 -2 -1 0 i2 1 2 -1 0 1 2 i 8 1-45 Illustrated Example 2 3 6 g p = 1 + sin ------ p 4 2 p2 2 3 1-2-1 2 1 1 0 0 -1 -2 -1 0 1 2 3 2 -1 0 1 2 0 1 2 3 1-4-1 2 1 1 0 0 -1 -2 Ming-Feng Yeh -1 -2 1-3-1 -1 0 1 2 -1 -2 1-5-1 -1 1-46 Convergence g p = 1 + sinp 2 p 2 3 3 5 2 2 1 3 1 5 3 4 2 0 1 4 2 0 0 0 1 -1 -2 -1 0 1 2 -1 -2 -1 0 1 2 Convergence to Global Min. Convergence to Local Min. The numbers to each curve indicate the sequence of iterations. Ming-Feng Yeh 1-47