Survey

Survey

Document related concepts

Neural modeling fields wikipedia, lookup

Metastability in the brain wikipedia, lookup

Central pattern generator wikipedia, lookup

Nervous system network models wikipedia, lookup

Gene expression programming wikipedia, lookup

Pattern recognition wikipedia, lookup

Artificial neural network wikipedia, lookup

Hierarchical temporal memory wikipedia, lookup

Backpropagation wikipedia, lookup

Convolutional neural network wikipedia, lookup

Catastrophic interference wikipedia, lookup

Transcript

Neural Networks Kostas Kontogiannis E&CE General Concepts • Neurons: the cells that perform information processing in the brain. It is the fundamental functional unit of all nervous system tissue, including brain • Soma: The neuron’s cell body • Dendrites: collection of fibers branching out of the soma body cell • Axon: A single long fiber in the collection of dendrites. Eventually, the axon also branches into strands and substrands that connect to the dendrites and cell bodies of other neurons • Synapse: The point where stands from two neurons connect Neural Networks • A neural network is composed of a number of nodes, or units, connected by links. Each link has a numeric weight associated with it. • Weights are the primary means of long-term storage in neural networks, and learning usually takes place by updating the weights. • Each unit has a set of input links from other units, a set of output links to other units, a current activation level, and a means of computing the activation level at the next step in time, given its inputs and weights. Neural Networks • To build a neural network to perform some task, one must first decide how many units are to be used, what kind of units, and how the units are connected to form a network. • One then initializes the weights of the network, and “trains” the weights using a learning algorithm applied to a set of training examples for the task. • The use of examples also implies that one must decide how to encode the examples in terms of inputs and outputs of the network. Neural Networks • To build a neural network to perform some task, one must first decide how many units are to be used, what kind of units, and how the units are connected to form a network. • One then initializes the weights of the network, and “trains” the weights using a learning algorithm applied to a set of training examples for the task. • The use of examples also implies that one must decide how to encode the examples in terms of inputs and outputs of the network. Simple Computing Elements • Each unit performs a simple computation: It receives signals from its input links and computes a new activation level that it sends along each of its output links. • The computation of the activation level is based on the values of each input signal received from a neighboring node, and the weights of each input link. • The computation is split into two components. First is a linear function ini that computes weighted sum of the unit’s input values. Second is a nonlinear component called the activation function g, that transforms the weighted sum into the final value that serves as the unit’s activation value ai. Models for Activation Functions • Different models are obtained by using different mathematical functions for g. Three common choices are the step, sign, and sigmoid functions. +1 +1 +1 t ini ini -1 Step Sign Sigmoid Network Structures • There are a variety of kinds of network structure, each of which results in a very different computational properties. • The main distinction is between feed-forward and recurrent networks. • In a feed-forward network, the links can form arbitrary topologies. In essence these networks are DAGs. • Usually we deal with networks that are arranged in layers. In a layered feed-forward network, each unit is linked only to the units in the next layer; there are no links between units in the same layer, no links backward to a previous layer, and no links that skip a layer. Fundamental Network Types • Hopfield Networks: They use bi-directional connections with symmetric weights; all of the units are input and output units, the activation function g is the sign function; and the activation levels can only be +1 or -1. • Boltzmann Machines: also use symmetric weights, but include units that are neither input nor output units. They also use a stochastic activation function, such that the probability of the output being 1 is some function of the total weighted input. • Networks with no hidden units are called perceptrons. • Input units are directly connected to the external input sources. Output units are connected to the observed output. Hidden units are neither connected to input sources nor the observed output. • Networks with one or more layers of hidden units are called multi-layer networks. Perceptron Neural Network Learning function NEURAL-NETWORK-LEARNING(examples) returns network network = a network with randomly assigned weights; repeat for each e in examples do O = NEURAL-NETWORK-OUTPUT(network, e); T = the observed output values from e; update the weights in network based on e, O, T; end until all examples correctly predicted or stopping criterion is reached return network Essentially Err = T - O Wj = Wj + (a * Ij * Err) Multi-Layer Feed-Forward Networks • Initial work in the 1950’s. • Learning algorithms for multi-layer are neither efficient, nor can guarantee that they can converge to a global optimum • On the other hand, learning general functions from examples is an intractable problem in the worst case • • The most popular method for learning in multi-layer networks is called backpropagation. Back Propagation Learning • Learning in multi-layer feed-forward networks using back-propagation proceeds the same way as for perceptrons: example inputs are presented to the network, and if the network computes an output vector that matches the output, nothing is done. If there is an error, then the weights are adjusted to reduce the error. • The trick is to assess the blame for an error and divide it among the contributing weights. In perceptrons, this is easy because there is only one weight between each input and the output. But in multilayer networks, there are many weights connecting each input to an output, and each of these weights contributes to more than one output • The back-propagation algorithm is a sensible approach to dividing the contribution of each weight. Back Propagation Learning • As in the perceptron learning algorithm, we try to minimize the error between each target output and the output value computed by the network. • At the output layer, the weight update rule is very similar to the rule for the perceptrons. However, there are two differences: The activation of the hidden unit aj is used instead of the input value, and the rule contains a term for the gradient of the activation function. • If Erri is the error Ti - O at the output node, then the weight update rule for the link from unit j to unit i is W j,i = W j,i + (alpha * aj * Erri * g’(ini) • where g’ is the derivative of the activation function g, and the above can be rewritten as: Wj,i = Wj, i + alpha * aj * Deltai Back Propagation Learning • On the previous formula, for updating the connections between the input units and the hidden units, we need to define a quantity analogous to the error term for output nodes. • The idea is that hidden node j is “responsible” for some fraction of the error Deltai, in each of the output nodes to which it connects. Thus, the Deltai values are divided according to the strength of the connection between the hidden node and the output node, and propagated back to provide the Deltai values for the hidden layer. The propagation rule for the Delta values is the following: Deltai = g’(inj) * Sumi (Wj,i * Deltai) • Now the update rule for the weights between the inputs and the hidden layer is almost identical to the update rule for the output layer: W k,j = W k,j + (alpha * Ik * Deltaj) Back Propagation Learning • The learning algorithm can be summarized as follows: – Compute the Delta values for the output units using the observed behavior – Starting with the output layer, repeat the following for each layer in the network, until the earliest (closest to input) hidden layer is reached • Propagate the Delta values values back to the previous layer • Update the weights between the two layers Back Propagation Learning Algorithm Algorithm Back-Prop-Update(network, examples, alpha) : new network weights repeat for each e in examples do O = Run-Network(network, Ie) Erre = Te - O W j,i = W j,i + (alpha * aj * Erre i * g’(ini)) for each subsequent layer in network do Deltaj = g’(inj) * Sum i W j,i * Delta I W k,j = W k,j +(alpha * Ik * Deltaj) end end until network has converged Discussion • Expressiveness: Well suited for continuous input/output, but do not have the expressive power of general logical representations • Computational Efficiency: For m examples and |W| weights each epoch takes O(m|W|) time. The worst case number of epochs is exponential to the number of inputs • Generalization: Good on generalizing on continuous functions that vary smoothly with the input • Sensitivity to noise: Very sensitive to noise since they do non-linear regression • Transparency: Neural networks are essential black boxes • Prior knowledge: Difficult to chose good training examples, and the best network topology