* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download NEURAL NETWORKS
Neural coding wikipedia , lookup
Neural oscillation wikipedia , lookup
Optogenetics wikipedia , lookup
Biological neuron model wikipedia , lookup
Neural modeling fields wikipedia , lookup
Neuroeconomics wikipedia , lookup
Holonomic brain theory wikipedia , lookup
Neuropsychopharmacology wikipedia , lookup
Synaptic gating wikipedia , lookup
Pattern recognition wikipedia , lookup
Central pattern generator wikipedia , lookup
Neural binding wikipedia , lookup
Metastability in the brain wikipedia , lookup
Neural engineering wikipedia , lookup
Development of the nervous system wikipedia , lookup
Artificial neural network wikipedia , lookup
Nervous system network models wikipedia , lookup
Catastrophic interference wikipedia , lookup
Convolutional neural network wikipedia , lookup
NEURAL NETWORKS Vedat Tavşanoğlu What Is a Neural Network? Work on artificial neural networks, commonly referred to as "neural networks," has been motivated right from its inception by the recognition that the brain computes in an entirely different way from the conventional digital computer. What Is a Neural Network? The struggle to understand the brain owes much to the pioneering work of Ramon y Cajal (1911), who introduced the idea of neurons as structural constituents of the brain. Typically, neurons are five to six orders of magnitude slower than silicon logic gates; events in a silicon chip happen in the nanosecond (10-9 s) range, whereas neural events happen in the millisecond (10-3 s) range. What Is a Neural Network? However, the brain makes up for the relatively slow rate of operation of a neuron by having a truly staggering number of neurons (nerve cells) with massive interconnections between them. What Is a Neural Network? It is estimated that there must be on the order of 10 billion neurons in the human cortex, and 60 trillion synapses or connections (Shepherd and Koch, 1990). The net result is that the brain is an enormously efficient structure. Specifically, the energetic efficiency of the brain is approximately 10-16 joules (J) per operation per second. The corresponding value for the best computers in use today is about 10-6 joules per operation per second (Faggin, 1991). What Is a Neural Network? The brain is a highly complex, nonlinear, and parallel computer (informationprocessing system). It has the capability of organizing neurons so as to perform certain computations (e.g., pattern recognition, perception, and motor control) many times faster than the fastest digital computer in existence today. What Is a Neural Network? Consider, for example, human vision, which is an information-processing task (Churchland and Sejnowski, 1992; Levine, 1985; Marr, 1982). It is the function of the visual system to provide a representation of the environment around us and, more important, to supply the information we need to interact with the environment. What Is a Neural Network? The brain routinely accomplishes perceptual recognition tasks (e.g., recognizing a familiar face embedded in an unfamiliar scene) in something of the order of 100-200 ms, whereas tasks of much lesser complexity will take hours on conventional computers. What Is a Neural Network? For another example, consider the sonar of a bat. Sonar is an active echo-location system. In addition to providing information about how far away a target (e.g., a flying insect) is, a bat sonar conveys information about the relative velocity of the target, the size of the target, the size of various features of the target, and the azimuth and elevation of the target (Suga, 1990a, b). What Is a Neural Network? The complex neural computations needed to extract all this information from the target echo occur within a brain the size of a plum. Indeed, an echo-locating bat can pursue and capture its target with a facility and success rate that would be the envy of a radar or sonar engineer. What Is a Neural Network? How, then, does a human brain or the brain of a bat do it? At birth, a brain has great structure and the ability to build up its own rules through what we usually refer to as "experience." What Is a Neural Network? Indeed, experience is built up over the years, with the most dramatic development (i.e.,'· hard-wiring) of the human brain taking place in the first two years from birth; but the development continues well beyond that stage. During this early stage of development, about 1 million synapses are formed per second. What Is a Neural Network? Synapses are elementary structural and functional units that mediate the interactions between neurons. The most common kind of synapse is a chemical synapse, which operates as follows: What Is a Neural Network? A presynaptic process liberates a transmitter substance that diffuses across the synaptic junction between neurons and then acts on a postsynaptic process. Thus a synapse converts a presynaptic electrical signal into a chemical signal and then back into a postsynaptic electrical signal (Shepherd and Koch, 1990). What Is a Neural Network? In electrical terminology, such an element is said to be a nonreciprocal two-port device. In traditional descriptions of neural organization, it is assumed that a synapse is a simple connection that can impose excitation or inhibition, but not both on the receptive neuron. What Is a Neural Network? A developing neuron is synonymous with a plastic brain: Plasticity ([Latin plasticus, from Greek plastikos, from plastos, molded, from plassein, to mold; see pelə-2 in Indo-European roots.] )permits the developing nervous system to adapt to its surrounding environment (Churchland and Sejnowski, 1992; Eggermont, 1990). In an adult brain, plasticity may be accounted for by two mechanisms: the creation of new synaptic connections between neurons, and the modification of existing synapses. What Is a Neural Network? Axons, the transmission lines, and dendrites, the receptive zones, constitute two types of cell filaments that are distinguished on morphological grounds; an axon has a smoother surface, fewer branches, and greater length, whereas a dendrite (so called because of its resemblance to a tree) has an irregular surface and more branches (Freeman, 1975). What Is a Neural Network? Neurons come in a wide variety of shapes and sizes in different parts of the brain. The figure illustrates the shape of a pyramidal cell, which is one of the most common types of cortical neurons. Like many other types of neurons, it receives most of its inputs through dendritic spines. The pyramidal cell can receive 10,000 or more synaptic contacts and it can project onto thousands of target cells. What Is a Neural Network? Just as plasticity appears to be essential to the functioning of neurons as informationprocessing units in the human brain, so it is with neural networks made up of artificial neurons. What Is a Neural Network? In its most general form, a neural network is a machine that is designed to model the way in which the brain performs a particular task or function of interest; the network is usually implemented using electronic components or simulated in software on a digital computer. What Is a Neural Network? In most cases the interest is confined largely to an important class of neural networks that perform useful computations through a process of learning. What Is a Neural Network? To achieve good performance, neural networks employ a massive interconnection of simple computing cells referred to as "neurons" or "processing units." We may thus offer the following definition of a neural network viewed as an adaptive machine: What Is a Neural Network? A neural network is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: 1 Knowledge is acquired by the network through a learning process . 2 Interneuron connection strengths known as synaptic weights are used to store the knowledge. What Is a Neural Network? The procedure used to perform the learning process is called a learning algorithm, the function of which is to modify the synaptic weights of the network in an orderly fashion so as to attain a desired design objective. What Is a Neural Network? The modification of synaptic weights provides the traditional method for the design of neural networks. Such an approach is the closest to linear adaptive filter theory, which is already well established and successfully applied in such diverse fields as communications, control, radar, sonar, seismology, and biomedical engineering (Haykin, 1991; Widrow and Stearns, 1985). What Is a Neural Network? However, it is also possible for a neural network to modify its own topology, which is motivated by the fact that neurons in the human brain can die and that new synaptic connections can grow. What Is a Neural Network? Neural networks are also referred to in the literature as neurocomputers, connectionist networks, parallel distributed processors, etc. Benefits of Neural Networks From the above discussion, it is apparent that a neural network derives its computing power through: 1. its massively parallel distributed structure, 1. its ability to learn and therefore generalize; generalization refers to the neural network producing reasonable outputs for inputs not encountered during training (learning). Benefits of Neural Networks How does the following example help you to generalize ? confer: L. conferre- con-, together, ferre, to bring v.t. to give, to bestow (to place or put by), to talk or consult together defer: L. differre- dis-, asunder (adv. apart, into parts, separately), ferre, to bear , to carry v.t. to put off to another time, to delay defer: L. deferre- de-, down, ferre, to bear v.i. to yield (to the wishes or opinions of another, or to authority), v.t. to submit or to or to lay before somebody differ: L. differre- dif.( for dis-), apart, ferre, to bear v.i. to be unlike, distinct or various infer: L. inferre- in-, into, ferre, to bring v.t.. to bring on, to drive as a conclusion prefer: L. preaferre- prea-,in front of, ferre, to bear v.t. to set in front, to put forward, offer, submit, present, for acceptance or consideration, to promote Benefits of Neural Networks convene L. convenire, con- together, and venire, to come v.i. to come together, v.i. to call together convent v.t. to convene convention the act of convening, :an assembly, esp. of special delegates for some common object, an agreement (Geneva Convention) invent L. invenire, inventum, in-, upon, venire, to come v.t. to find, to device or contrive prevent L. preavenire, prea- in front of, venire, to come v.t. to precede, to be, go, act, earlier than, to preclude, to stop, keep, or hinder effectually, to keep from coming to pass Benefits of Neural Networks Synonym:1432 (but rare before 18c.), from L. synonymum, from Gk. synonymon "word having the same sense as another," noun use of neut. of synonymos "having the same name as, synonymous," from syn- "together, same" + onyma, Aeolic dialectal form of onoma "name" (see name). Synonymous is attested from 1610. Benefits of Neural Networks Antonym:1870, created to serve as opposite of synonym, from Gk. anti- "equal to, instead of, opposite" (see anti-) + -onym "name" (see name). Anonymous:1601, from Gk. anonymos "without a name," from an- "without" + onyma, Æolic dialectal form of onoma "name" (see name). Benefits of Neural Networks These two information-processing capabilities,i.e., (1) massively parallel distributed structure (2) the ability to generalize make it possible for neural networks to solve complex (large-scale) problems that are currently intractable. In practice, however, neural networks cannot provide the solution working by themselves alone. Rather, they need to be integrated into a consistent system engineering approach. Benefits of Neural Networks 1. 2. 3. Specifically, a complex problem of interest is decomposed into a number of relatively simple tasks, and neural networks are assigned a subset of the tasks e.g., pattern recognition, associative memory, control,etc. that match their inherent capabilities. It is important to recognize, however, that we have a long way to go (if ever) before we can build a computer architecture that mimics the human brain. Properties and Capabilities of Neural Networks 1. Nonlinearity A neuron is basically a nonlinear device. Consequently, a neural network, made up of an interconnection of neurons, is itself nonlinear. Moreover, the nonlinearity is of a special kind in the sense that it is distributed throughout the network. Nonlinearity is a highly important property, particularly if the underlying physical mechanism responsible for the generation of an input signal (e.g., speech signal) is inherently nonlinear. Properties and Capabilities of Neural Networks 2. Input-Output Mapping A popular paradigm of learning called supervised learning involves the modification of the synaptic weights of a neural network by applying a set of labeled training samples or task examples. Each example consists of a unique input signal and the corresponding desired response. Properties and Capabilities of Neural Networks The network is presented an example picked at random from the set, and the synaptic weights(free parameters) of the network are modified so as to minimize the difference between the desired response and the actual response of the network Properties and Capabilities of Neural Networks The training of the network is repeated for many examples in the set until the network reaches a steady state, where there are no further significant changes in the synaptic weights; The previously applied training examples may be reapplied during the training session but in a different order. Properties and Capabilities of Neural Networks Thus the network learns from the examples by constructing an input-output mapping for the problem at hand. Properties and Capabilities of Neural Networks Such an approach brings to mind the study of nonparametric statistical inference which is a branch of statistics dealing with model-free estimation, or, from a biological viewpoint, tabula rasa learning (Geman et al., 1992). (tabula rasa: a smoothed or blank tablet, a mind not yet influenced by outside impressions and experiences) ([Medieval Latin tabula rāsa : Latin tabula, tablet + Latin rāsa, feminine of rāsus, erased.] Properties and Capabilities of Neural Networks Consider, for example, a pattern classification task, where the requirement is to assign an input signal representing a physical object or event to one of several prespecified categories (classes). In a nonparametric approach to this problem, the requirement is to "estimate" arbitrary decision boundaries in the input signal space for the patternclassification task using a set of examples, and to do so without invoking a probabilistic distribution model. Properties and Capabilities of Neural Networks A similar point of view is implicit in the supervised learning paradigm, which suggests a close analogy between the input-output mapping performed by a neural network and nonparametric statistical inference. paradigm:1.Grammar. a.a set of forms all of which contain a particular element, esp. the set of all inflected forms based on a single stem or theme. b.a display in fixed arrangement of such a set, as boy, boy's, boys, boys'. 2.an example serving as a model; pattern. [Origin: 1475–85; < LL paradīgma < Gk parádeigma pattern (verbid of paradeiknýnai to show side by side), equiv. to para- para-1 + deik, base of deiknýnai to show (see deictic) + -ma n. suffix ] Properties and Capabilities of Neural Networks analogy: 1550, from L. analogia, from Gk. analogia "proportion," from ana- "upon, according to" + logos "ratio," also "word, speech, reckoning." A mathematical term used in a wider sense by Plato. Properties and Capabilities of Neural Networks 3. Adaptivity. Neural networks have a built-in capability to adapt their synaptic weights to changes in the surrounding environment. In particular, a neural network trained to operate in a specific environment can be easily retrained to deal with minor changes in the operating environmental conditions. Properties and Capabilities of Neural Networks Moreover, when it is operating in a nonstationary environment (i.e., one whose statistics change with time), a neural network can be designed to change its synaptic weights in real time. The natural architecture of a neural network for pattern classification, signal processing, and control applications, coupled with the adaptive capability of the network, make it an ideal tool for use in adaptive pattern classification, adaptive signal processing, and adaptive control. Properties and Capabilities of Neural Networks As a general rule, it may be said that the more adaptive we make a system in a properly designed fashion, assuming the adaptive system is stable, the more robust its performance will likely be when the system is required to operate in a nonstationary environment. Properties and Capabilities of Neural Networks It should be emphasized, however, that adaptivity does not always lead to robustness; indeed, it may do the very opposite. For example, an adaptive system with short time constants may change rapidly and therefore tend to respond to spurious disturbances, causing a drastic degradation in system performance. Properties and Capabilities of Neural Networks To realize the full benefits of adaptivity, the principal time constants of the system should be long enough for the system to ignore spurious (L. spurius, false) disturbances and yet short enough to respond to meaningful changes in the environment; the problem described here is referred to as the stability-plasticity dilema (Grossberg, 1988). Adaptivity (or “in situ,(L.)in the original situation” training as it is sometimes referred to) is an open research topic. Properties and Capabilities of Neural Networks 4. Evidential Response In the context of pattern classification, a neural network can be designed to provide information not only about which particular pattern to select, but also about the confidence in the decision made. This latter information may be used to reject ambiguous patterns, should they arise, and thereby improve the classification performance of the network. Properties and Capabilities of Neural Networks 5. Contextual Information (L. contextus,contexere-con-, texere, textum, to weave) Knowledge is represented by the very structure and activation state of a neural network. Every neuron in the network is potentially affected by the global activity of all other neurons in the network. Consequently, contextual information is dealt with naturally by a neural network. Properties and Capabilities of Neural Networks 6. Fault Tolerance A neural network, implemented in hardware form, has the potential to be inherently fault tolerant in the sense that its performance is degraded gracefully under adverse operating conditions (Bolt, 1992). For example, if a neuron or its connecting links are damaged, recall of a stored pattern is impaired in quality. However, owing to the distributed nature of information in the network, the damage has to be extensive before the overall response of the network is degraded seriously. Thus, in principle, a neural network exhibits a graceful degradation in performance rather than catastrophic failure. Properties and Capabilities of Neural Networks 7. VLSI Implementability The massively parallel nature of a neural network makes it potentially fast for the computation of certain tasks. This same feature makes a neural network ideally suited for implementation using very-Iarge-scale-integrated (VLSI) technology. The particular virtue of VLSI is that it provides a means of capturing truly complex behavior in a highly hierarchical fashion (Mead and Conway, 1980), which makes it possible to use a neural network as a tool for real-time applications involving pattern recognition, signal processing, and control. Properties and Capabilities of Neural Networks 8. Uniformity of Analysis and Design. Basically, neural networks enjoy universality as information processors. We say this in the sense that the same notation is used in all the domains involving the application of neural networks. This feature manifests itself in different ways: Neurons, in one form or another, represent an ingredient common to all neural networks. This commonality makes it possible to share theories and learning algorithms in different applications of neural networks. Modular networks can be built through a seamless integration of modules. Properties and Capabilities of Neural Networks analysis: [Medieval Latin, from Greek analusis, a dissolving, from analūein, to undo : ana-, throughout; see ana- + lūein, to loosen; see leu- in Indo-European roots.] (Download Now or Buy the Book) The American Heritage® Dictionary of the English Language, Fourth Edition Copyright © 2006 by Houghton Mifflin Company. Published by Houghton Mifflin Company. All rights reserved.Online Etymology Dictionary - Cite This Source - Share This analysis 1581, "resolution of anything complex into simple elements" (opposite of synthesis), from M.L. analysis, from Gk. analysis "a breaking up," from analyein "unloose," from ana- "up, throughout" + lysis "a loosening" (see lose). Psychological sense is from 1890. Phrase in the final (or last) analysis (1844), translates Fr. en dernière analyse. Properties and Capabilities of Neural Networks Design: 1548, from L. designare "mark out, devise," from de- "out" + signare "to mark," from signum "a mark, sign." Originally in Eng. with the meaning now attached to designate (1646, from L. designatus, pp. of designare); many modern uses of design are metaphoric extensions. Designer (adj.) in the fashion sense of "prestigious" is first recorded 1966; designer drug is from 1983. Designing "scheming" is from 1671. Designated hitter introduced in American League baseball in 1973, soon giving wide figurative extension to designated. Properties and Capabilities of Neural Networks 9. Neurobiological Analogy The design of a neural network is motivated by analogy with the brain, which is a living proof that fault-tolerant parallel processing is not only physically possible but also fast and powerful. Neurobiologists look to (artificial) neural networks as a research tool for the interpretation of neurobiological phenomena. Properties and Capabilities of Neural Networks For example, neural networks have been used to provide insight on the development of premotor (relating to, or being the area of the cortex of the frontal lobe lying immediately in front of the motor area of the precentral gyrus(Any of the prominent, rounded, elevated convolutions on the surfaces of the cerebral hemispheres. [Latin gȳrus, circle; see gyre.] ) )circuits in the oculomotor (1.Of or relating to movements of the eyeball: an oculomotor muscle. 2.Of or relating to the oculomotor nerve. [Latin oculus, eye; see okw- in Indo-European roots + motor.] system (responsible for eye movements) and the manner in which they process signals (Robinson, 1992). On the other hand, engineers look to neurobiology for new ideas to solve problems more complex than those based on conventional hard-wired design techniques. Properties and Capabilities of Neural Networks Here, for example, we may mention the development of a model sonar receiver based on the bat (Simmons et aI., 1992). The batinspired model consists of three stages: (1) a front end that mimics the inner ear of the bat in order to encode waveforms; (2) a subsystem of delay lines that computes echo delays; (3) a subsystem that computes the spectrum of echoes, which is in turn used to estimate the time separation of echoes from multiple target glints. Properties and Capabilities of Neural Networks The motivation is to develop a new sonar receiver that is superior to one designed by conventional methods. The neurobiological analogy is also useful in another important way: It provides a hope and belief (and, to a certain extent, an existence proof) that physical understanding of neurobiological structures could indeed influence the art of electronics and thus VLSI (Andreou, 1992). Properties and Capabilities of Neural Networks With inspiration from neurobiological analogy in mind, it seems appropriate that we take a brief look at the structural levels of organization in the brain. Properties and Capabilities of Neural Networks 1.2 Structural Levels of Organization in the Brain The human nervous system may be viewed as a three-stage system,(Arbib, 1987). Block-diagram representation of nervous system Properties and Capabilities of Neural Networks Central to the system is the brain, represented by the neural (nerve) net in this figure, which continually receives information, perceives it, and makes appropriate decisions. Two sets of arrows are shown in this figure: 1 Those pointing from left to right indicate the forward transmission of information-bearing signals through the system. 2 The arrows pointing from right to left signify the presence of feedback in the system. Properties and Capabilities of Neural Networks The receptors in the figure convert stimuli from the human body or the external environment into electrical impulses that convey information to the neural net (brain). The effectors, on the other hand, convert electrical impulses generated by the neural net into discernible responses as system outputs. . In the brain there are both small-scale and large-scale anatomical organizations, and different functions take place at lower and higher levels. Properties and Capabilities of Neural Networks This figure shows a hierarchy of interwoven levels of organization that has emerged from the extensive work done on the analysis of local regions in the brain (Churchland and Sejnowski, 1992; Shepherd and Koch, 1990). Properties and Capabilities of Neural Networks Proceeding upward from synapses that represent the most fundamental level and that depend on molecules and ions for their action, we have neural microcircuits dendritic trees, and then neurons. Properties and Capabilities of Neural Networks A neural microcircuit refers to an assembly of synapses organized into patterns of connectivity so as to produce a functional operation of interest. A neural microcircuit may be likened to a silicon chip made up of an assembly of transistors. Properties and Capabilities of Neural Networks The smallest size of microcircuits is measured in micrometers (m), and their fastest speed of operation is measured in milliseconds. The neural microcircuits are grouped to form dendritic subunits within the dendritic trees of individual neurons. The whole neuron, about 100 m in size, contains several dendritic subunits. Properties and Capabilities of Neural Networks At the next level of complexity, we have local circuits (about 1 mm in size) made up of neurons with similar or different properties; these neural assemblies perform operations characteristic of a localized region in the brain. Properties and Capabilities of Neural Networks This is followed by interregional circuits made up of pathways, columns, and topographic maps, which involve multiple regions located in different parts of the brain. Properties and Capabilities of Neural Networks Topographic maps are organized to respond to incoming sensory information. These maps are often arranged in sheets, as in the superior colliculus, where the visual, auditory, and somatosensory maps are stacked in adjacent layers in such a way that stimuli from corresponding points in space lie above each other. Finally, the topographic maps, and other interregional circuits mediate specific types of behavior in the central nervous system . Properties and Capabilities of Neural Networks It is important to recognize that the structural levels of organization described herein are a unique characteristic of the brain. They are nowhere to be found in a digital computer, and we are nowhere close to realizing them with artificial neural networks. Nevertheless, we are inching our way toward a hierarchy of computational levels similar to that described in the last figure. Properties and Capabilities of Neural Networks The artificial neurons we use to build our neural networks are truly primitive in comparison to those found in the brain. The neural networks we are presently able to design are just as primitive compared to the local circuits and the interregional circuits in the brain. Properties and Capabilities of Neural Networks What is really satisfying, however, is the remarkable progress that we have made on so many fronts during the past 20 years. With the neurobiological analogy as the source of inspiration, and the wealth of theoretical and technological tools that we are bringing together, it is for certain that in another 10 years our understanding of artificial neural networks will be much more sophisticated than it is today. Properties and Capabilities of Neural Networks Our primary interest here is confined to the study of artificial neural networks from an engineering perspective, to which we refer simply as neural networks. We begin the study by describing the models of (artificial) neurons that form the basis of the neural networks considered in these lectures. Models of a Neuron Models of a Neuron A neuron is an information-processing unit that is fundamental to the operation of a neural network. The figure on the next slide shows the model for a neuron. Models of a Neuron Nonlinear model of a neuron Models of a Neuron 1. A set of synapses or connecting links, each of which is characterized by a weight or strength of its own. Specifically, a signal xj at the input of synapse j connected to neuron k is multiplied by the synaptic weight wkj. It is important to make a note of the manner in which the subscripts of the synaptic weight wkj are written. Models of a Neuron The first subscript refers to the neuron in question and the second subscript refers to the input end of the synapse to which the weight refers; the reverse of this notation is also used in the literature. The weight wkj is positive if the associated synapse is excitatory; it is negative if the synapse is inhibitory (Middle English inhibiten, to forbid, from Latin inhibēre, inhibit, to restrain, forbid : in-, in; see in-2 + habēre, to hold; see ghabh- in Indo-European roots.] ) Models of a Neuron 2.An adder for summing the input signals, weighted by the respective synapses of the neuron; the operations described here constitute a linear combiner. Models of a Neuron 3.An activation function for limiting the amplitude of the output of a neuron. The activation function, is also referred to in the literature as a squashing function in that it squashes (limits) the permissible amplitude range of the output signal to some finite value. Typically, the normalized amplitude range of the output of a neuron is written as the closed unit interval [0,1] or alternatively [-1,1]. Models of a Neuron 4. The model of a neuron also includes an externally applied threshold k that has the effect of lowering the net input of the activation function. On the other hand, the net input of the activation function may be increased by employing a bias term rather than a threshold; the bias is the negative of the threshold. Models of a Neuron In mathematical terms, we may describe neuron k by writing the following pair of equations: p Mathematical Model of a Neuron uk wkj x j j 1 yk ( uk k ) where xj’s are the input signals; wkj’s are the synaptic weights of neuron k; uk is the linear combiner output; k is the threshold; is the activation function; and yk is the output signal of the neuron. Models of a Neuron Block-Diagram Representation of a Neuron p uk wkj x j j 1 yk ( u k k ) Models of a Neuron The use of threshold k has the effect of applying an affine transformation to the output uk of the linear combiner in the model of the figure, as shown by vk u k k Models of a Neuron In particular, depending on whether the threshold k is positive or negative, the relationship between the effective internal activity level or activation potential vk of neuron k and the linear combiner output uk is modified in the manner illustrated in the figure. Note that as a result of this affine transformation, the graph of vk versus uk no longer passes through the origin. Models of a Neuron The threshold k is an external parameter of artificial neuron k. We may account for its presence as in the above equation. Equivalently, we may formulate the combination of the two equations as follows: p vk wkj x j j 0 yk (vk ) Models of a Neuron Here we have added a new synapse, whose input is x0 1 and whose weight is wk 0 k Models of a Neuron We may therefore reformulate the model of neuron k as in the figure, where the effect of the threshold is represented by doing two things: Models of a Neuron (1) adding a new input signal fixed at -1, and (2) adding a new synaptic weight equal to the threshold k . Alternatively, we may model the neuron as in the following slide: Y1 Models of a Neuron where the combination of fixed input xo = + 1 and weight wkO = bk accounts for the bias bk• Although the models of the two figures are different in appearance, they are mathematically equivalent. Slayt 92 Y1 YTU; 15.03.2005 Models of a Neuron Signal-Flow Graph Representation of a Neuron uk p w j 1 kj xj yk ( uk k ) Models of a Neuron Signal-Flow Graph Representation of a Neuron Two different types of links may be distinguished: (a) Synaptic links, defined by a linear input-output relation. Specifically, the node signal xj is multiplied by the synaptic weight wkj to produce the node signal vk . (b) Activation links, defined in general by a nonlinear input-output relation. This form of relationship is the nonlinear activation function given as (.) Models of a Neuron The Activation Function The activation function, denoted by y (v ) defines the output y of a neuron in terms of the activity level at its input v. Models of a Neuron We may identify three basic types of activation functions: Threshold Function 2. Piecewise-linear Function 3. Sigmoid Function 1. Models of a Neuron Threshold (hard limiter or binary activation ) Function (leading to discrete perceptron) 1. (v ) 1 1 (v) sgn(v) 2 2 1 0 (a) Unipolar v Models of a Neuron (v ) (v) sgn(v) 1 v 00 0 -1 (b) Bipolar Models of a Neuron 2. Piecewise-linear Function (v ) 1 1 1 1 (v ) v v 2 2 2 2 1 0.5 -0.5 0 0 0 (a) Unipolar v y ij (t ) f ( xij ) 1 xij (t ) 1 x ij (t ) 1 2 Models of a Neuron (v ) 1 (v ) v 1 v 1 2 1 -1 0 1 -1 (b) Bipolar v Models of a Neuron 3. Sigmoid Function (v ) 1 0.5 (a) Unipolar 1 (v ) ;a 0 av 1 e v Models of a Neuron (v ) 1 v -1 (b) Bipolar 1 e av 2 = -1 ; a 0 (v ) av av 1 e 1 e Models of Artificial Neural Networks DEFINITION OF Neural Network (Jacek M. Zurada, ARTIFICIAL NEURAL SYSTEMS, 1992, West Publishing Company) A Neural Network is an interconnection of neurons such that neuron outputs are connected, through weights, to all other neurons including themselves; both lagfree and delay connections are allowed. Models of Artificial Neural Networks Neural Networks Viewed as Directed Graphs 1. 2. Block-Diagram Representation (BDR) Signal-Flow Graph Representation (SFGR) These are obtained when BDR and SFGR for the neurons are used. Models of Artificial Neural Networks An alternative definition of Neural Network (Simon Haykin, NEURAL NETWORKS, 1994, Macmillan College Publishing Company) A neural network is a directed graph (SFG) consisting of nodes with interconnecting synaptic and activation links, and which is characterized by four properties: Each neuron is represented by a set of linear synaptic links, an externally applied threshold, and a nonlinear activation link. The threshold is represented by a synaptic link with an input signal fixed at a value of -1. Models of Artificial Neural Networks 2. The synaptic links of a neuron weight their respective input signals. 3. The weighted sum of the input signals defines the total internal activity level of the neuron in question. 4. The activation link squashes the internal activity level of the neuron to produce an output that represents the output of the neuron. Network Architectures In general, we may identify four different classes of network architectures: 1. Single-Layer Feedforward Networks 2. Multilayer Feedforward Networks 3. Recurrent Networks 4. Lattice Structures Network Architectures 1. Single-Layer Feedforward Networks A layered neural network is a network of neurons organized in the form of layers. In the simplest form of a layered network, we just have an input layer of source nodes that projects onto an output layer of neurons (computation nodes), but not vice versa. Network Architectures In other words, this network is strictly of a feedforward type. It is illustrated on the following slide for the case of four nodes in both the input and output layers. Such a network is called a single-layer network, with the designation "single layer" referring to the output layer of computation nodes (neurons). In other words, we do not count the input layer of source nodes, because no computation is performed there. Network Architectures Network Architectures 2. Multilayer Feedforward Networks The second class of a feedforward neural network distinguishes itself by the presence of one or more hidden layers, whose computation nodes are correspondingly called hidden neurons or hidden units. The function of the hidden neurons is to intervene between the external input and the network output. Network Architectures Network Architectures By adding one or more hidden layers, the network is enabled to extract higher-order statistics, for (in a rather loose sense) the network acquires a global perspective despite its local connectivity by virtue of: the extra set of synaptic connections the extra dimension of neural interactions. The ability of hidden neurons to extract higher-order statistics is particularly valuable when the size of the input layer is large. Network Architectures The source nodes in the input layer of the network supply respective elements of the activation pattern (input vector), which constitute the input signals applied to the neurons (computation nodes) in the second layer (i.e., the first hidden layer). The output signals of the second layer are used as inputs to the third layer, and so on for the rest of the network. Network Architectures Typically, the neurons in each layer of the network have as their inputs the output signals of the preceding layer only. The set of output signals of the neurons in the output (final) layer of the network constitutes the overall response of the network to the activation pattern supplied by the source nodes in the input (first) layer. Network Architectures This graph illustrates the layout of a multilayer feedforward neural network for the case of a single hidden layer. For brevity this network is referred to as a 10-4-2 network in that it has 10 source nodes, 4 hidden neurons, and 2 output neurons. Network Architectures As another example, a feedforward network with p source nodes, h1 neurons in the first hidden layer, h2 neurons in the second layer, and q neurons in the output layer, say, is referred to as a p-h1-h2-q network. Network Architectures The neural network of this figure is said to be fully connected in the sense that every node in each layer of the network is connected to every other node in the adjacent forward layer. Network Architectures If, however, some of the communication links (synaptic connections) are missing from the network, we say that the network is partially connected. A form of partially connected multilayer feedforward network of particular interest is a locally connected network. An example of such a network with a single hidden layer is presented on the next slide. Each neuron in the hidden layer is connected to a local (partial) set of source nodes that lies in the immediate neighborhood. Network Architectures Such a set of localized nodes feeding a neuron is said to constitute the receptive field of the neuron. Likewise, each neuron in the output layer is connected to a local set Partially connected of hidden neurons. feedforward neural network Network Architectures 3. Recurrent (Feedback or Dynamical) Networks A recurrent neural network distinguishes itself from a feedforward neural network in that it has at least one feedback loop. For example, a recurrent network may consist of a single layer of neurons with each neuron feeding its output signal back to the inputs of all the other neurons, as illustrated in the architectural graph of the figure on the right. In the structure depicted in this figure there are no self-feedback loops in the Recurrent network with no network; self-feedback refers to a situation where the output of a neuron self-feedback loops and no is fedback to its own input. hidden neurons Network Architectures The recurrent network illustrated on the previous slide also has no hidden neurons. Here we illustrate another class of recurrent networks with hidden neurons. The feedback connections shown originate from the hidden neurons as well as the output neurons. Recurrent network with hidden neurons Network Architectures The presence of feedback loops, be it as in the recurrent structure with or without hidden neurons, has a profound impact on the learning capability of the network, and on its performance. Moreover, the feedback loops involve the use of particular branches composed of unit-delay elements (denoted by z-1), which result in a nonlinear dynamical behavior by virtue of the nonlinear nature of the neurons. Network Architectures 4. Lattice (Multicategory Perceptron) Structures A lattice consists of a one-dimensional, two-dimensional, or higher-dimensional array of neurons with a corresponding set One dimensional lattice of 3 neurons of source nodes that supply the input signals to the array; the dimension of the lattice refers to the number of the dimensions of the space in which the graph lies. A lattice network is really a feedforward network with the output neurons arranged in rows and columns. Two dimensional lattice of 3-by-3 neurons The Perceptron The perceptron is the simplest form of a neural network used for the classification of a special type of patterns said to be linearly separable (i.e., patterns that lie on opposite sides of a hyperplane). Basically, it consists of a single neuron with adjustable synaptic weights and threshold, as shown in the figures. The Perceptron The algorithm used to adjust the free parameters of this neural network first appeared in a learning procedure developed by Rosenblatt (1958, 1962) for his perceptron brain model. Indeed, Rosenblatt proved that if the patterns (vectors) used to train the perceptron are drawn from two linearly separable classes, then the perceptron algorithm converges and positions the decision surface in the form of a hyperplane between the two classes. The proof of convergence of the algorithm is known as the perceptron convergence theorem. The Perceptron The single-layer perceptron depicted has a single neuron. Such a perceptron is limited to performing pattern classification with only two classes. By expanding the output (computation) layer of the perceptron to include more than one neuron, we may correspondingly form classification with more than two classes. However, the classes would have to be linearly separable for the perceptron to work properly. The Perceptron From this model we find that the linear combiner output (i.e., hard limiter input) is p v wkj x j j 1 The purpose of the perceptron is to classify the set of externally applied stimuli x1 ,x2,…., xp into one of two classes, C1 or C2, say. The decision rule for the classification is to assign the point represented by the inputs x1 ,x2,…., xp to class C1, if the perceptron output y is + 1 and to class C2 if it is -1. The Perceptron To develop insight into the behavior of a pattern classifier, it is customary to plot a map of the decision regions in the p-dimensional signal space spanned by the p input variables x1 ,x2,…., xp. In the case of an elementary perceptron, there are two decision regions separated by a hyperplane defined by p w j 1 kj xj 0 The Perceptron This is illustrated here for the case of two input variables xl and x2, for which the decision boundary takes the form of a straight line called the decision line. A point (x1,x2) that lies above the decision line is assigned to class C1, and a point (x1,x2) that lies below the decision line is assigned to class C2. Note also that the effect of the threshold is merely to shift the decision line away from the origin. The synaptic weights w1 w2, ..,wp of the perceptron can be fixed or adapted on an iteration-by-iteration basis. For the adaptation, we may use an error-correction rule known as the perceptron convergence algorithm. The Perceptron We find it more convenient to work with the modified signal flow graph given here. In this second model, which is equivalent to that of the previous figure, the threshold is treated as a synaptic weight is connected to a fixed input equal to -1. We may thus define the (p + 1)-by-1 (augmented) input vector and the corresponding (augmented) weight vector as: x [ x1 x2 ... ... x p 1]t w [ w1 w2 ... ... wp ]t The Perceptron Pattern Space Any pattern can be represented by a point in n-dimensinal Euclidean space En called the pattern space. Points in that space corresponding to members of the pattern set are n-tuple vectors x. The Perceptron Example 1: Consider the six patterns in two dimensional pattern space shown in the following figure. x2 (0, 0) (-0.5, -1) (-1, -2) (2,0) x1 (1.5,-1) (1, -2) The Perceptron Design a perceptron such that these are classified according to their membership in sets as follows : 2 1.5 1 , , : class 1 1 2 0 0 0.5 1 , : class 2 , 1 2 0 The Perceptron One possible decision line is given by x2= 2x1-2 which is drawn in the following figure. x2 x = 2x -2 2 1 (0, 0) (-0.5, -1) (-1, -2) (2,0) x1 (1.5,-1) (1, -2) The Perceptron One decision surface for this line is obtained as: x3 2 x1 x2 2 x3 0 2 x1 x2 2 0 gives the points on the decision line x3 0 2 x1 x2 2 0 gives the part of the surface above the decision line x3 0 2 x1 x2 2 0 gives the part of the surface below the decision line Such a pattern classification can be performed by the following (discrete) perceptron (dichotomizer): dichotomize: to divide or separate into two parts dicha: in two; tomia: to cut The Perceptron x1 x2 -2 1 -2 -1 + v sgn(v) y y sgn(2 x1 x2 2) Single-Layer Feedforward Neural Network Example 2: Assume that a set of eight points, P0, P1... , P7 , in three-dimensional space is available. The set consists of all vertices of a three-dimensional cube as follows: {P0(-l, -1, -l), P1(-l, -1, l), P2(-1, 1, -1), P3(-1, 1, 1), P4(1, -1, -l), P5(1, -1, 1), P6(1,1, -1), P7(1, 1, 1)} Elements of this set need to be classified into two categories The first category is defined as containing points with two or more positive ones; the second category contains all the remaining points that do not belong to the first category. Single-Layer Feedforward Neural Network Classification of points P3, P5, P6, and P7 can be based on the summation of coordinate values for each point evaluated for category membership. Notice that for each point Pi (x1, x2, x3) ,where i = 0, ... , 7, the membership in the category can be established by the following calculation: 1, then category 1 If sgn( x1 x2 x3 ) 1, then category 2 Single-Layer Feedforward Neural Network The neural network given below implements the above expression: Single-Layer Feedforward Neural Network The network above performs the threedimensional Cartesian space partitioning as illustrated below : Single-Layer Feedforward Neural Network Discriminant Functions In Example 1 x3 2 x1 x2 2 can be viewed as a Discriminant Function. We may also write g ( x1 , x2 ) 2 x1 x2 2 or x1 g ( x ) 2 x1 x2 2 where x = x2 Single-Layer Feedforward Neural Network g ( x1 , x2 ) 2 x1 x2 2 can also be viewed as the equation of a plane in 3-D Euclidean space. On the other hand g ( x1 , x2 ) 0 2 x1 x2 2 0 is the intersection line of the above plane with the xy-plane. Single-Layer Feedforward Neural Network Obviously: g ( x ) 0 2 x1 x2 2 0 gives the points on the decision line g ( x ) 0 2 x1 x2 2 0 gives the points on the plane above the decision line g ( x ) 0 2 x1 x2 2 0 gives the points on the plane below the decision line Single-Layer Feedforward Neural Network Since on the decision line we have g ( x1 , x2 ) 0 we can write g ( x1 , x2 ) g ( x1 , x2 ) dg ( x1 , x2 ) dx1 dx2 0 x1 x2 where dx1 and dx2 are the increments given to x1 and x2 on the decision line. Single-Layer Feedforward Neural Network Now defining where g ( x1 , x2 ) x dx1 1 and dr g ( x1 , x2 ) g ( x1 , x2 ) dx2 x 2 g and dr are known to be the gradient vector (or normal vector) and the tangent vector, respectively, Single-Layer Feedforward Neural Network The gradient vector points toward the positive side of the decision line. However, there are two normal vectors, one pointing toward the positive side, 1, and the other toward the negative side, 2 =-1. For the above example the gradient and normal vectors are given by: g ( x1 , x2 ) x 2 2 2 1 , 1 g ( x1 , x2 ) ,2 g ( x1 , x2 ) g ( x1 , x2 ) 1 1 1 x 2 Single-Layer Feedforward Neural Network In fact 2 is obtained from g ( x1 , x2 ) 0 Note that 1 and 2 are the projections of the normal vectors on the x-y plane of two intersecting planes whose intersection line is given by g ( x1 , x2 ) 0 Single-Layer Feedforward Neural Network Although 1 and 2 are unique, there are infinetely many plane pairs whose intersection line is given by g ( x1 , x2 ) 0 Plane pairs can be built by appropriately augementing the 2-D normal vectors 1 and 2 to 3-D normal vectors which will be the normal vectors of the two intersecting planes. Single-Layer Feedforward Neural Network The 2-D normal vectors are plane vectors given in the x-y plane. 2 2 1 ,2 1 1 These can be augmented to 3-D by adding a third component, say 2, yielding 2 2 n1 1 , n2 1 2 2 Single-Layer Feedforward Neural Network The details of building the augmented vectors are shown below: g n1 -2 -1 -2 x2 Decision line n 1 2 0 1 2 x1 Single-Layer Feedforward Neural Network Note that 1 and 2 are the normal vectors of the plane that is perpendicular to the x-y plane and intersects the x-y plane at the decision line. On the other hand the vectors n1 and n2 are the normal vectors of the planes obtained by rotating the above perpendicular plane around the decision line by and , respectively. Single-Layer Feedforward Neural Network We can now determine the equations for the these planes by using the normal vector-point form of plane equation given as: t n ( x x0 ) 0 where: •n is the normal vector of the plane, •x is the vector connecting any point on the plane to the origin, •x0 is the vector connecting a fixed point on the plane to the origin. Single-Layer Feedforward Neural Network This means that x-x0 represents the vector connecting all possible points x on the plane to fixed point x0 on the same plane. That is x-x0 is a vector that lies on the plane. Now let us find the plane equations for the two normal vectors found above. Single-Layer Feedforward Neural Network Let x0 be the point (1,0,0) on the decision line. We can write: x1 1 2 1 For n1 1 2 1 2 x2 0 0 g1 x1 x2 1 2 2 g1 0 x1 1 2 1 For n2 1 2 1 2 x2 0 0 g 2 x1 x2 1 2 2 g 2 0 Single-Layer Feedforward Neural Network Because of the way g1(x) and g2(x) are built we can state the following: g1 ( x) g 2 ( x) 0 on the positive side of the decision line g 2 ( x) g1 ( x) 0 on the negative side of the decision line 2 2 1 1 For n1 1 g1 x1 x2 1 For n2 1 g 2 x1 x2 1 2 2 2 2 Single-Layer Feedforward Neural Network g n2 n1 g2 Decision Decisio line g1 x1 Decision line x2 Single-Layer Feedforward Neural Network Now we can compute g1(x) and g2(x) for the selected patterns in Example 1. Class 1 Class 2 (2,0) (1.5,-1) (1,-2) (0,0) (-0.5,1) (-1,-2) g1-g2>0 g1-g2>0 g1-g2>0 g2-g1>0 g2-g1>0 g2-g1>0 Single-Layer Feedforward Neural Network Henceforth, such gi(x) functions will be called Discriminant Functions. We can conclude that: g1 ( x) g 2 ( x) for the patterns in Class 1 g 2 ( x) g1 ( x) for the patterns in Class 2 Single-Layer Feedforward Neural Network Minimum Distance Classification The classification of two clusters is carried out in such a way that the boundary of these two clusters is drawn as a line perpendicular to and passing through the midpoint of the line connecting the center points of two clusters . Therefore the boundary line is the perpendicular bisector of a connecting line. Single-Layer Feedforward Neural Pi Network Positive side xi-xj xi Negative side P0=(xi+xj)/2 Pj xj 0 Single-Layer Feedforward Neural Network Now we will derive the equation of the boundary line. Let the vector x and x0 represent any point on this and the point P0, respectively. Then the following must hold: ( xi x j ) ( x x 0 ) 0 t which can be written in the form 1 ( xi x j ) ( x ( xi x j )) 0 2 t Single-Layer Feedforward Neural Network and 1 t ( xi x j ) x ( xi x j ) ( xi x j ) 0 2 t or 2 1 2 ( xi x j ) x ( xi x j ) 0 2 t Single-Layer Feedforward Neural Network Now defining 1 gij ( x ) ( xi x j ) x ( xi 2 t 2 2 xj ) We have already seen that the boundary (decision) line can be taken as the intersection of two planes gi and gj . Single-Layer Feedforward Neural Network Therefore gij ( x ) gi ( x ) g j ( x ) where we have called gi (x) discriminant functions and shown that they are associated with plane equations. Single-Layer Feedforward Neural Network Now using the two equations above we obtain 2 1 2 ( xi x j ) x ( xi x j ) gi ( x ) g j ( x ) 2 t which can be used to make the following identification: 1 gi ( x ) xi x xi 2 t 2 1 g j ( x) x j x x j 2 t 2 Single-Layer Feedforward Neural Network gi (x) can also be expressed as: gi ( x ) wi x wi ,n1 t Therefore we can make the identification: wi = x i 1 wi ,n1 xi 2 2 Single-Layer Feedforward Neural Network An alternative approach towards the construction of discriminant functions may be taken as follows: Let us assume that a minimum–distance classification is requried to classify patterns into R categories. Each of the classes is represented by its center point Pi , i=1,2,…..,R. The Euclidean distance between an input pattern x and the point Pi is given by the norm of the vector x-xi as: x xi ( x xi ) ( x xi ) t Single-Layer Feedforward Neural Network A minimum–distance classifier computes the distance from a pattern of unknown classification to each of the center points Pi . Then the category number of the point that yields the minimum distance is assigned to the unknown pattern. Squaring the above equation yields x xi 2 1 t x x - 2x x + x xi = x x - 2(x x - xi xi ) > 0 2 t t i t i t t i Single-Layer Feedforward Neural Network Since xxt is independent of i, this term is constant with respect to the categories. Therefore, in order to minimize the distance x xi we need to maximize 1 t gi (x) = x x - xi xi 2 t i which is called a discriminant function. Single-Layer Feedforward Neural Network Example 3: A linear minimum-distance classifier will be designed for the three points given as: 10 2 5 x1 , x2 , x3 2 5 5 It is also assumed that the index of each point (pattern)corresponds to its class number. The three points and the connecting lines constitute a triangle which is shown on the next slide: Single-Layer Feedforward Neural Network x2 P3(-5,5) P1(10,2) 0 P2(2,-5) x1 Single-Layer Feedforward Neural Network Now let us draw the circle passing through all three vertices of the triangle, the circumcircle. We can conclude that each boundary is a perpendicular bisector of the triangle. A perpendicular bisector of a triangle is a straight line passing through the midpoint of a side and being perpendicular to it, i.e. forming a right angle with it. The three perpendicular bisectors meet at a single point, the triangle's circumcenter; this point is the center of the circumcircle. Single-Layer Feedforward Neural Network x2 P3(-5,5) 0 P2(2,-5) P1(10,2) x1 Single-Layer Feedforward Neural Network Now using 10 2 5 x1 , x2 , x3 2 5 5 and 1 t gij ( x ) ( xi x j ) x ( xi 2 we obtain 2 2 xj ) Single-Layer Feedforward Neural Network 1 2 2 g12 ( x ) ( x1 x2 ) x ( x1 x2 ) 2 t t 10 2 x1 1 [(100 4) (4 25)] 2 5 x2 2 8 x1 7 x2 37.5 Single-Layer Feedforward Neural Network 1 2 2 g13 ( x ) ( x1 x3 ) x ( x1 x3 ) 2 t t 10 5 2 5 15 x1 3 x2 27 x1 1 x 2 [(100 4) (25 25)] 2 Single-Layer Feedforward Neural Network 1 2 2 g 23 ( x ) ( x2 x3 ) x ( x2 x3 ) 2 t t 2 5 x1 1 [(4 25) (25 25)] 5 5 x2 2 7 x1 10 x2 10.5 Single-Layer Feedforward Neural Network Now using wi = x i 1 wi ,n1 xi 2 2 we obtain 10 2 5 w1 2 ; w2 5 ; w3 5 ; -52 -14.5 -25 Single-Layer Feedforward Neural Network and using gi ( x ) wi x wi ,n1 t we obtain g1 ( x ) 10 x1 2 x2 52 g 2 ( x ) 2 x1 5 x2 14.5 g3 ( x ) 5 x1 5 x2 25 Single-Layer Feedforward Neural Network A block diagram producing the three discriminant functions is shown below: x1 x2 -1 10 2 52 2 -5 14.5 5 -5 25 10 x1 2 x2 52 2 x1 5 x2 14.5 5 x1 5 x2 25 Single-Layer Feedforward Neural Network The discriminant values for the three patterns P1(10,2), P2(2,-5) and P3(-5,5) are shown in the table below: Input g1(x)=10x1+2x2-52 Class 1 [10 2]t 52 Class 2 [2 -5]t -42 Class 3 [-5 5]t -92 g2(x)= 2x1-5x2-14.5 -4.5 14.5 -49.5 g3(x)=-5x1+5x2-25 -65 -60 25 Discriminant Single-Layer Feedforward Neural Network As required by the definition of the discriminant function, the responses on the diagonal are the largest in each column. It will be shown later that the same is true for any three points P1,P2 ,P3 taken from the three decision regions H1,, H2, H3 provided that the decision regions are determined as shown above. Therefore using a maximum selector at the output will provide the required function from the network. Single-Layer Feedforward Neural Network Using the same network with TLUs (bipolar activation functions) will result in the outputs given in the table below: Input Class 1 [10 2]t sgn(g1(x)=5x1+3x2-5) 1 Class 2 [2 -5]t -1 Class 3 [-5 5]t -1 sgn(g2(x)= -x2-2) -1 1 -1 sgn(g3(x)=-9x1+x2) -1 -1 1 Single-Layer Feedforward Neural Network The diagonal entries=1 The offdiagonal entries=-1 However, as the next example will demonstrate this is not true for any three points P1,P2 ,P3 taken from the three decision regions H1, H2, H3. Single-Layer Feedforward Neural Network The response of the same network to the patterns Q1(5,0), Q2(0,1) and Q3(-4,0) are shown in the table below: Input Discriminant g1(x)=10x1+2x2-52 Class 1 [5 0]t -2 Class 2 [0 1]t -50 Class 3 [-4 0]t -92 g2(x)= 2x1-5x2-14.5 -4.5 -19.5 -22.5 g3(x)=-5x1+5x2-25 -50 -20 -5 Single-Layer Feedforward Neural Network The responses on the diagonal are still the largest in each column. However, using the same network with TLUs (bipolar activation functions) will result in the outputs given in the table on the next slide: Single-Layer Feedforward Neural Network Input sgn(g1(x)=10x1+2x2-52) Class 1 [5 0]t -1 Class 2 [0 1]t -1 Class 3 [-4 0]t -1 sgn(g2(x)= 2x1-5x2-14.5) -1 -1 -1 sgn(g3(x)=-5x1+5x2-25) -1 -1 -1 Discriminant Single-Layer Feedforward Neural Network It is therefore impossible to use TLUs once the decision lines are calculated using the minimum-distance calssification procedure. The only way out is using a maximum selector. The explanation of the responses on the diagonal being the largest in each column will now be made in detail. Single-Layer Feedforward Neural Network The discriminant functions determine the plane equations g1 10 x1 2 x2 52 0 g 2 2 x1 5 x2 14.5 0 g3 5 x1 5 x2 25 0 Single-Layer Feedforward Neural Network These planes are shown on the next slide. It is easily seen that: For any point in H1 : g1(x)>g2(x) and g1(x)>g3(x) For any point in H2 : g2(x)>g1(x) and g2(x)>g3(x) For any point in H3 : g3(x)>g1(x) and g3(x)>g2(x) Single-Layer Feedforward Neural Network 100 50 0 gi -50 -100 -150 -200 10 5 0 -5 x2 -10 -10 0 -5 x1 5 10 Single-Layer Feedforward Neural Network The decision regions H1,H2, H3 are projections of the planes g1,g2 and g3, respectively, on the x1-x2 plane and the decision lines are the projections of the intersection lines of the planes gi on the x1-x2 plane which are shown on the next slide. Single-Layer Feedforward Neural Network x 2 H3 g13 (x)=0 H1,, g1 ( x) g 2 ( x) P3(-5,5) g1 ( x) g3 ( x) g3 ( x) g1 ( x) P123(2.337,2.686) g3 ( x) g 2 ( x) 0 g 2 ( x) g1 ( x) g23 (x)=0 H2 g 2 ( x) g3 ( x) P2(2,-5) g12 (x)=0 P1(10,2) x1 Single-Layer Feedforward Neural Network A MATLAB plot of the projections of the intersection lines of the planes gi are shown on the next slide Single-Layer Feedforward Neural Network 30 20 10 0 -10 -20 -30 -30 -20 -10 0 10 20 30 Single-Layer Feedforward Neural Network The projections of the intersection lines of the planes gi on the x1-x2 plane are shown to be given by the following line equations: g12 ( x ) 8 x1 7 x2 37.5 0 g13 ( x ) 15 x1 3 x2 27 0 g 23 ( x ) 7 x1 10 x2 10.5 0 The previous slide shows the segments that can be seen from the top. Single-Layer Feedforward Neural Network The continuation of the line g12=0 remains underneath the plane g3. The continuation of the line g23=0 remains underneath the plane g1. The continuation of the line g13=0 remains underneath the plane g2. Single-Layer Feedforward Neural Network A classifier using a maximum selector is shown on the next slide. The maximum selector selects the maximum discriminant and responds with the number of the discriminant having the largest value. Single-Layer Feedforward Neural Network x1 x2 -1 10 g1(x) 2 52 2 -5 14.5 5 -5 25 1 g2(x) Maximum i=1,2, or 3 2 selector g3(x) 3 Classifier using the maximum selector Single-Layer Feedforward Neural Network The classifier can be redrawn as follows: 10 x1 g1(x) 2 x2 1 52 -1 2 x1 g2(x) Maximum i=1,2, or 3 5 2 selector x2 14.5 -1 -5 x1 g3(x) x2 5 3 25 -1 Classifier using the maximum selector Single-Layer Feedforward Neural Network x1 x2 -1 x1 x2 -1 10 x1 x2 -1 -5 2 g1(x) 52 2 -5 g2(x) Maximum i=1,2, or 3 2 selector g3(x) 3 14.5 5 25 1 Classifier using the maximum selector Single-Layer Feedforward Neural Network In the above we have designed a classifier which was based on the minimumdistance classification for known clusters and derived the network with three perceptrons from the discriminant functions which were interpreted as plane equations. Instead, now let us consider the network on the next slide which is obtained as a result of training a network with three perceptrons using the same input patterns P1(10,2), P2(2,-5) and P3(-5,5) as in the previous network . Single-Layer Feedforward Neural Network x1 x2 -1 5 x1 3 x2 5 5 TLU#1 3 5 0 -1 2 1 -9 0 x2 2 9x1 x2 TLU#2 TLU#3 Single-Layer Feedforward Neural Network In fact gi(x)=0 define the intersection of gi planes with x1-x2 plane. Therefore the TLU divides the gi planes into two regions: (1)the upper-half plane which is above x1-x2 plane and (1)the lower-half plane which is below x1-x2 plane. Single-Layer Feedforward Neural Network The decision lines are obtained by setting gi(x)=0 5 x1 3x2 5 0 x2 2 0 9 x1 x2 0 which are given on the next slide. The shaded areas are indecision regions which will become clear in the following discussion. Single-Layer Feedforward Neural Network 5 x1 3 x2 5 0 g1 (5,5) 15 1 g (5,5) 7 1 2 g 2 (5,5) 50 1 x2 Q1(0,9) g1 (0,9) 22 1 g (0,9) 29 1 2 g 2 (0,9) 9 1 9 x1 x2 0 P3(-5,5) P1(10,2) x1 0 x2 2 0 g1 (1, 3) 19 1 g (1, 3) 1 1 2 g 2 (1, 3) 6 1 g1 (10, 2) 51 1 g (10, 2) 4 1 2 g 2 (10, 2) 88 1 Q2(4,-4) Q3(-1,-3) P2(2,-5) g1 (2, 5) 10 1 g (2, 5) 3 1 2 g 2 (2, 5) 23 1 g1 (4, 4) 3 1 g (4, 4) 2 1 2 g 2 (4, 4) 40 1 Single-Layer Feedforward Neural Network The discriminant values g1(x), g2(x), g3(x) for the same three patterns P1(10,2), P2(2,-5) and P3(-5,5) are shown in the table below: Input Class 1 [10 2]t 51 Class 2 [2 -5]t -10 Class 3 [-5 5]t -15 g2(x)= -x2-2 -4 3 -7 g3(x)=-9x1+x2 -88 -23 50 Discriminant g1(x)=5x1+3x2-5 Single-Layer Feedforward Neural Network The outputs of the network with three discrete perceptrons are shown in the table below: Input Class 1 [10 2]t 1 Class 2 [2 -5]t -1 Class 3 [-5 5]t -1 sgn(g2(x)= -x2-2) -1 1 -1 sgn(g3(x)=-9x1+x2) -1 -1 1 sgn(g1(x)=5x1+3x2-5) Single-Layer Feedforward Neural Network The table on the previous slide shows that the new discriminant functions g1 ( x ) 5 x1 3 x2 5 g 2 ( x ) x2 2 g3 ( x ) 9 x1 x2 classify the paterns P1(10,2), P2(2,-5) and P3(-5,5) in the same way as the discriminant functions g1 ( x ) 10 x1 2 x2 52 g 2 ( x ) 2 x1 5 x2 14.5 g3 ( x ) 5 x1 5 x2 25 Single-Layer Feedforward Neural Network Conclusion: The network, which is obtained through the perceptron learning algorithm, and the network obtained using the maximum-distance classification procedure have classified the three points P1(10,2), P2(2,-5) and P3(-5,5) in exactly the same way, i.e., P1 (10,2) class 1 P2 (2,-5) class 2 P3 (-5,5) class 3 Single-Layer Feedforward Neural Network Now consider the patterns Q1(0,9), Q2(4,-4) and Q3(-1,-3) which fall into shaded areas. The discriminant values for these patterns are shown in the table on the next slide: Single-Layer Feedforward Neural Network Input [0 9]t [4 -4]t [-1 -3]t g1(x)=5x1+3x2-5 22 3 -19 g2(x)= -x2-2 -29 2 1 9 -40 6 Discriminant g3(x)=-9x1+x2 Single-Layer Feedforward Neural Network Since g1 (0,9) g3 (0,9), g 2 (0,9) g3 ( 1, 3)>g 2 (1, 3), g1 (1, 3) g1 (4,-4)>g 2 (4,-4),g3 (4,-4) if we use a maximum selector instead of the three TLUs the network can decide that Q1 (0,9) and Q 2 (4,-4) class 1 Q3 (1, 3) class 3 Single-Layer Feedforward Neural Network On the other hand, if we use TLUs we would obtain the outputs in the following table: Input [0 9]t [4 -4]t [-1 -3]t g1(x)=5x1+3x2-5 1 1 -1 g2(x)= -x2-2 -1 1 1 g3(x)=-9x1+x2 1 -1 1 Discriminant Single-Layer Feedforward Neural Network In order to make a classification we should have a column with one 1 and two -1s. Therefore according to the table obtained non of the three patterns Q1(0,9), Q2(4,-4) and Q3(-1,-3) could be classified into any class. Therefore according to the network with TLUs the shaded areas will be called indecision regions. Single-Layer Feedforward Neural Network Now let us consider the planes defined by g1 5 x1 3 x2 5 g 2 x2 2 g3 9 x1 x2 which are plotted on the next slide: Single-Layer Feedforward Neural Network 100 50 gi 0 -50 -100 10 0 x2 -10 -10 -8 -6 -4 -2 x1 0 2 4 6 8 10 Single-Layer Feedforward Neural Network The projections of the intersection lines of the planes gi(x) on the x1-x2 plane are given by g12 : 5 x1 4 x2 3 0 g 23 : 9 x1 2 x2 2 0 g13 : 14 x1 2 x2 5 0 The segments that can be seen from the top are plotted on the next slide. Single-Layer Feedforward Neural Network 15 10 5 0 -5 -10 -15 -10 -5 0 5 10 15 20 Single-Layer Feedforward Neural Network The continuation of the line g12=0 remains underneath the plane g3. The continuation of the line g23=0 remains underneath the plane g1. The continuation of the line g13=0 remains underneath the plane g2. Single-Layer Feedforward Neural Network x1 x2 w11 v1 wk1 v2 wK1 wK2 wkj vk xj xn Input nodes wmn vm y1 1 Neurons 2 y2 k yk K ym Output nodes Single-Layer Feedforward Neural Network v1 w11 x1 w12 x2 ......... w1 j x j ...... w1J xJ y1 f ( v1 ) v2 w21 x1 w22 x2 ......... w2 j x j ...... w2 J xJ y2 f ( v2 ) vk wk1 x1 wk 2 x2 ......... wkj x j ...... wkJ xJ yk f ( vk ) vK wK 1 x1 wK 2 x2 ......... wKj x j ...... wKJ xJ y K f ( vK ) Single-Layer Feedforward Neural Network v1 w11 v w 2 21 . . . . vK wK 1 w12 w22 . . wK 2 . . w1J x1 . . w1J x2 . . . . . . . . . . wKJ xJ v Wx v1 y1 f ( v1 ) y f ( v ) v2 2 2 . . Γ . . . . yK f ( vJ ) v J y Γ(v) Single-Layer Feedforward Neural Network y1 f (.) y 0 2 . . . . yK 0 0 . . f (.) . . . . . . 0 . . . . v1 v 2 . . . f (.) vJ 0 0 . y Γ[Wx] Single-Layer Feedforward Neural Network Example 1: x1 x x2 1 v1 5 3 5 x1 5 x1 3 x2 5 x3 v 0 1 2 x x 2 x 2 3 2 2 v3 9 1 0 1 9 x1 x2 y1 sgn(5 x1 3x2 5 x3 ) y sgn( x 2 x ) 2 3 2 y3 sgn(9 x1 x2 ) Two-Layer Feedforward Neural Network Example 1: Design a neural network such that the network maps the shaded region of plane x1, x2 into y = 1, and it maps its complement into y = -1, where y is the output of the neural network. In summary, the network will provide the mapping of the entire x1, x2 plane into one of the two points ±1 on the real number axis. Two-Layer Feedforward Neural Network Solution: The inputs to the neural network will be x1, x2 and the threshold value -1. Thus the input vector is given as: x1 x x2 1 Two-Layer Feedforward Neural Network The boundaries of the shaded region are given by the equations: x1 1 0 x1 2 0 x2 0 x2 3 0 Two-Layer Feedforward Neural Network The shaded region satisfies the inequalities: x1 1 x1 1 0 x1 2 x1 2 0 x2 0 x2 3 or x2 0 x2 3 0 Two-Layer Feedforward Neural Network These inequalities may be implemented using four neurons: Two-Layer Feedforward Neural Network The equations for the first layer are obtained as: v1 1 0 1 v 1 0 2 x1 2 x2 v3 0 1 0 1 v4 0 1 3 y sgn( x1 1) sgn( x1 2) sgn( x2 ) sgn( x2 3) where binary (threshold or hard limiter )activation function, i.e., discrete perceptron is used. t Two-Layer Feedforward Neural Network Let us discuss the mapping performed by the first layer. Note that each of the neurons 1 through 4 divides the plane xl ,x2 into two half-planes. The half-planes where the neurons' responses are positive (+ 1) have been marked with arrows pointing toward the positive response half plane. The response of the. second layer can be easily obtained as y sgn( y1 y2 y3 y4 3.5) Two-Layer Feedforward Neural Network The resultant neural network The Perceptron Training Algorithm For the development of the perceptron learning algorithm for a single-layer perceptron, we find it more convenient to work with the modified signal-flow graph model given here. In this second model, which is equivalent to that of the previous figure, the threshold is treated as a synaptic weight connected to a fixed input equal to -1. We may thus define the (p + 1)-by-1 input vector and the corresponding weight vector as: x [ x1 x2 ... ... xn 1]t w [ w1 w2 ... ... wn ]t The Perceptron Training Algorithm These vectors are respectively called the augmented input vector and the augmented weight vector. For fixed n, the equation wtx = 0, plotted in p-dimensional space with coordinates xl; xz, ... , xp, defines a hyperplane as the decision surface between two different classes of inputs. Suppose then the input variables of the single-layer perceptron originate from two linearly separable classes that fall on the opposite sides of some hyperplane. Let Xl be the subset of training vectors xl(1), xl(2), ... that belong to class C1, and let X2 be the subset of training vectors x2(1), x2(2), ... that belong to class C2. The union of Xl and X2 is the complete training set X. The Perceptron Training Algorithm Given the sets of vectors X and X to train the classifier, l 2 the training process involves the adjustment of the weight vector w in such a way that the two classes Cl and C2 are separable. These two classes are said to be linearly separable if a realizable setting of the weight vector w exists. Conversely, if the two classes Cl and C2 are known to be linearly separable, then there exists a weight vector w such that we may state: The Perceptron Training Algorithm w x - 0 for every input vector x belonging to class C1 w x0 for every input vector x belonging to class C2 t t The Perceptron Training Algorithm Given the subsets of training vectors X1 and X2, the training problem for the elementary ·perceptron is then to find a weight vector w such that the two inequalities above are satisfied. However, until this is achieved in the itermediate steps we will have wx - 0 for some input vectors x belonging to class C2 w x0 for some input vector x belonging to class C1 t t The Perceptron Training Algorithm In the former case therefore wt x will be reduced until wt x 0 is achieved, and in the latter case wt x will be increased until wt x 0 is reached. Here we will begin to examine neural network classifiers that derive their weights during the learning cycle. The Perceptron Training Algorithm • The sample pattern vectors x1 , x2, ... , xp, called the training sequence, are presented to the machine along with the correct response. • The response is provided by the teacher and specifies the classification information 'for each mput vector. The classifier modifies its parameters by means of iterative, supervised learning. The Perceptron Training Algorithm The network learns from 'experience by comparing the targeted correct response with the actual response. The classifier structure is usually adjusted after each incorrect response based on the error value generated. The Perceptron Training Algorithm Let us now look again at the dichotomizer introduced and defined earlier. We will develop a supervised training procedure for this two-class linear classifier. Assuming that the desired response is provided, the error signal is computed. The error information can be used to adapt the weights of the discrete perceptron. The Perceptron Training Algorithm First we examine the geometrical conditions in the augmented weight space. This will make it possible to devise a meaningful training procedure for the dichotomizer under consideration. The decision surface equation in n+1 dimensional augmented pattern space is w x0 t The Perceptron Training Algorithm When the above equation is considered in the pattern space then it is written for fixed weights w(1), w(2),……….,w(k). Therefore the variables of the function f(wt(i )x) are x1, x2,……….,xn+1 , the components of the pattern vector. The Perceptron Training Algorithm x2 f(wt(i )x)=0 w (i ) x1 The normal vector w(i ) (weight wector) points toward the side of the pattern space for which wt(i )x > 0, called the positive side. The Perceptron Training Algorithm When the above equation is considered in the weight space then it is written for fixed patterns x(1), x(2),……….,x(p). Therefore the variables of the function f(wtx(i)) are w1, w2,……….,wn+1 , the components of the weight vector. The Perceptron Training Algorithm w2 f(wtx(i))=0 w1 x ( i) The normal vector x(i) (pattern vector) points toward the side of the weight space for which wtx(i) > 0, called the positive side. The Perceptron Training Algorithm ∇ f ( w t x (i )) f ( w1 , w2 ,....., wn1 ) w1 x1 (i ) f ( w1 , w2 ,....., wn1 ) x2 (i ) w2 x( i ) f ( w1 , w2 ,....., wn1 ) xn1 (i ) f ( w1 , w2 ,....., wn1 ) wn1 In further discussion it will be understood that the normal vector will always point toward the side of the space for which wtx> 0, called the positive side, or semispace, of the hyperplane. The Perceptron Training Algorithm Decision hyperplane in augmented weight space for a five pattern set from two classes The Perceptron Training Algorithm that the vectors x(i) points toward the positive side of the decision hyperplanes wtx(i)= 0. Note By labeling each decision boundary in the augmented weight space with an arrow pointing into the positive half-plane, we can easily find a region in the weight space that satisfies the linearly separable classification. The Perceptron Training Algorithm To find the solution for weights, we will look for the intersection of the positive decision regions due to the prototypes of class 1·and of the negative decision regions due to the prototypes of class 2. The Perceptron Training Algorithm Inspection of the figure reveals that the intersection of the sets of weights yielding all five correct classifications of depicted patterns is in the shaded region of the second quadrant as shown in the figure above. The Perceptron Training Algorithm Let us now attempt to arrive iteratively at the weight vector w located in the shaded weight solution area. To accomplish this, the weights need to be adjusted from the initial value located anywhere in the weight space. The Perceptron Training Algorithm This assumption is due to our ignorance of the weight solution region as well as weight initialization. The adjustment discussed, or network training, is based on an error-correction scheme. The Perceptron Training Algorithm At this point we will introduce the Perceptron Learning (Traning) Rule (Algorithm). The perceptron learning rule is of central importance for supervised learning of neural networks. The weights are initialized at any values in this method The Perceptron Training Algorithm A neuron is considered to be an adaptive element. Its weights are modifiable depending on the input signal it receives, its output value, and the associated teacher (supervisor) response. The Perceptron Training Algorithm The weight vector is changed according to the following: w(i 1) w(i ) w(i ) where w(i ) cr w(i ), x(i ), d (i ) x(i ) and • d(i) is the teacher’s (supervisor’s) signal • r is the learning signal •c is a positive number called the learning constant depending on the sign of r. The Perceptron Training Algorithm Here we have used ∇ f ( w t x( i )) w 1 x (i ) t ( w x( i )) 1 x2 (i ) t (w x( i )) = x( i ) w2 x ( i ) n1 ( w t x( i )) w n 1 This reveals that the change in the weight vector is in the direction of steepest ascent (or descent)of wtx(i). The Perceptron Training Algorithm Perceptron Learning (Traning) Rule (Algorithm). In this case the learning signal is defined as: r (i ) d (i ) y (i ) where d(i) is the desired output signal and y(i) is the actual output signal for the input pattern x(i) given by: y (i ) sgn( w (i) x(i) ) t The weight adjustment is given by: w(i ) c[d (i ) sgn( w (i ) x (i ))] x (i ) t The Perceptron Training Algorithm d = 1, i.e.,class 1 is input : 1) y =sgn( wt x) 1,i.e., the input is misclassified r d y 1 (1) 2; the correction is in the direction of steepest ascent and given as w(i) 2c x(i) 2) y =sgn( wt x) 1,i.e., the input is correctly classified r d y 1 1 0; no correction d = -1, i.e.,class 2 is input : 1) y sgn( wt x) 1,i.e., the input is correctly classified r d y 1 (1) 0; no correction 2) y sgn(wt x) 1,i.e., the input is misclassified r d y 1 (1) 2; the correction is in the direction of steepest decent and given as w(i) 2c x(i) The Perceptron Training Algorithm EXAMPLE: The trained classifier should provide the following classification of four patterns x with known class membership d: x(1)= 1, x(3) =3, x(2)= -0.5, x(4)=-2, d1= d3 = 1: class C1 d2 = d4 = -1: class C2 The Perceptron Training Algorithm The augmented input vectors are given as: 1 0.5 3 2 x(1) , x(2) , x(3) , x(4) 1 1 1 1 x( x x(1) x The Perceptron Training Algorithm Let us choose an arbitrary augmented weight vector of 2.5 w(1) 1.75 With x (1) being the input , we obtain 1 wt (1) x(1) 2.5 1.75 0.75 0 1 and binary activation function (discrete perceptron) sgn( wt (1) x(1)) 1 Hence x(1) is classified as being in class C2. However this is not true. Therefore a correction has to be made. The Perceptron Training Algorithm The question to be asked at this point is: How do we make this correction? The answer depends on which training algorithm used. Since sgn{wt(1)x(1)}=-1, one thing is certain,however, is that the correction should me made in such a way that t wx increases. In order that we achieve this we must first find out if there is a direction in which the decrease or, for that matter, increase takes place. To show this, let us consider the surface given by : The Perceptron Training Algorithm z f ( w1 , w2 ,....., wn1 ) We can write: df ( w1 , w2 ,....., wn1 ) f ( w1 , w2 ,....., wn1 ) f ( w1 , w2 ,....., wn1 ) f ( w1 , w2 ,....., wn1 ) dw1 dw2 ..... dwn1 w1 w2 wn1 Let us now restrict ourselves to the case of 3 dimensions, namely, z , w1 , w2 or more succinctly z , x, y The Perceptron Training Algorithm Now consider the surface z f ( x, y ) If the level curves are interpreted as contour lines of the landscape, i.e., of the surface, then along these curves z f ( x, y ) constant The Perceptron Training Algorithm consequently, we obtain dz df ( x, y ) 0 hence f ( x , y ) f ( x , y ) dx dy 0 df ( x , y ) x y where dx and dy are the increments given to x and y on the level curve. The Perceptron Training Algorithm Now defining f ( x , y ) x dx f ( x , y ) f ( x , y ) and dr dy y where f and dr are known to be the gradient vector and the tangent vector,respectively, The Perceptron Training Algorithm we can write df ( x , y ) f dr 0 t This means that the gradient vector and the tangent vector are orthogonal vectors. Moreover it can be shown that the gradient vector points in the direction of steepest ascent of the function f(x,y). Furthermore, the gradient is the rate of climb in the direction of steepest ascent. The Perceptron Training Algorithm Now consider the surface z f ( x, y ) ( x 50) ( y 50) 32 2 2 The following MATLAB program plots this surface: 2 The Perceptron Training Algorithm close all clear all for x=1:1:100; for y=1:1:100; f(x,y)=(x-50).^2+(y-50).^2-1024; end end mesh(f);title('f(x,y)=x^2+y^2-1024'); figure,imshow(f,[ ],'notruesize');colormap(jet); colorbar;title('f(x,y)=x^2+y^2-1024'); The Perceptron Training Algorithm z=f(x,y)=(x-50)2+(y-50)2-322 z=f(x,y)=(x-50)2+(y-50)2-322 4000 3000 4000 3000 2000 2000 1000 1000 0 0 -1000 -1000 -2000 100 100 80 50 -2000 100 60 50 40 0 20 0 0 20 0 z=f(x,y)=(x-50)2+(y-50)2-322 3500 3000 2500 2000 1500 1000 500 0 -500 -1000 40 60 80 100 The Perceptron Training Algorithm The level curves are obtained from z f ( x, y ) ( x 50) ( y 50) 32 Ci 2 2 2 where Ci are constants. y C3 16384 C2 9216 C1 4096 x The Perceptron Training Algorithm f ( x, y ) x 2( x 50) dx f ( x, y ) and dr dy ( , ) f x y 2( 50) y y Considering the four quadrants of the circle: Q.1 2( x 50) 0 Q.2 2( x 50) 0 2( y 50) 0 Q.3 2( x 50) 0 2( y 50) 0 Q.4 2( x 50) 0 2( y 50) 0 2( y 50) 0 The Perceptron Training Algorithm the gradient vector points in directions as given below: Q.2 Q.1 Q.3 Q.4 The Perceptron Training Algorithm The fact that the gradient vector is orthogonal to the tangent vector proves that it is in the direction of steepest ascent or steepest descent. The directions found for the example show that the gradient vector points in the direction of ascent of the function f(x,y). Combining the two facts we can conclude that it points in the direction of steepest ascent. The Perceptron Training Algorithm x1( 4 ) 2 x1( 3 ) 3 x ( 2 ) 0.5 x ( 1 ) 1 , x ( ) x( 1 ) 1 , x( 2 ) 1 , x ( ) 3 4 x2 ( 4 ) 1 x2 ( 3 ) 1 x2 ( 2 ) 1 x2 ( 1 ) 1 In the weight space the following straight lines represent the decision lines: w1 w1 1 w2 w1 w2 0 w2 w1 1 0.5 w2 w1 0.5w2 0 w2 0.5w1 1 w1 3 w2 w1 3w2 0 w2 3w1 1 w1 2 w2 w1 2 w2 0 w2 2 w1 1 The Perceptron Training Algorithm Decision lines in weight space w2 Initial weight vector x(2) x(4) x(1) x(3) w1 2 4 3 1 The Perceptron Training Algorithm The corresponding gradient vectors are computed as follows: The Perceptron Training Algorithm for x( 1 ) ( wt x( 1 )) x ( 1 ) 1 w 1 t t 1 x ( ) w x( 1 ) w1 w2 ( w x( 1 )) 1 t 1 ( w x( 1 )) x1( 1 ) w 2 for x( 2 ) ( wt x( 2 )) x ( 2 ) 0.5 w 1 t t 1 x ( ) w x( 1 ) 0.5w1 w2 ( w x( 2 )) 2 t 1 ( w x( 2 )) x1( 2 ) w 2 for x( 3 ) ( wt x( 3 )) x ( 3 ) 3 w 1 t t 1 w x( 3 ) 0.5w1 w2 ( w x( 3 )) t x( 3 ) 1 ( w x( 3 )) x1( 3 ) w 2 for x( 4 ) ( wt x( 4 )) x ( 4 ) 2 w 1 t t 1 x ( ) w x( 4 ) 2 w1 w2 ( w x( 4 )) 4 t 1 ( w x( 4 )) x1( 4 ) w 2 The Perceptron Training Algorithm wtx(3)<0 wtx(3)>0 w2 Decision lines and gradient vectors in weight space Initial weight vector 2.5 w(1) 1.75 2 wtx(2)>0 wtx(2)<0 x(2) x(4) x(1) x(3) w1 wtx(4)>0 4 wtx(1)>0 wtx(4)<0 wtx(1)<0 3 1 The Perceptron Training Algorithm Now we can concentrate on the particular training (or learning) algorithm (or rule). The Perceptron Training Algorithm This is a supervised learning algorithm. This means that at each step the correction is made according to the directive given by the supervisor as shown in the following figure. The Perceptron Training Algorithm yi x di Weight learning rule: di is provided only in the case of supervised learning The Perceptron Training Algorithm Now consider r d i sgn( w x ) t Since d i 1 , sgn( w x ) 1 t r can take on one of the three values: 2, 2, 0 The Perceptron Training Algorithm forIn fact d i 1, sgn( w x ) 1 r 2 t for d i 1, sgn( w x ) 1 r 2 t for Since d i 1, sgn( w x ) 1 r 0 t and d i 1, sgn( w x ) 1 r 0 t Therefore we can define the correction rule in terms of the correction amount at the nth step as follows: Δwi ( n ) ( n )( d i ( n ) sgn( w ( n )x( n )))( w ( n )x( n )) t t The Perceptron Training Algorithm Δwi ( n ) ( n )( d i ( n ) sgn( w ( n )x( n )))x( n ) t yi di The Perceptron Training Algorithm In order for the correct cllasification of the entire x( 1 ), x( 2 ), x( 3 ), and x( 4 ) training set with respective class memberships d ( 1 ) 1, d ( 2 ) 1, d ( 3 ) 1; and d ( 4 ) 1 the following four inequalities must hold: t w ( N )x( 1 ) 0 w ( N )x( 2 ) 0 t wt ( N )x( 3 ) 0 wt ( N )x( 4 ) 0 where w(N) is the final weight vector that provides correct classification for the entire training set. The Perceptron Training Algorithm This means that after N-1 training steps the weight vector w(N) ends up in the solution area, which is the shaded area in the following figure. The Perceptron Training Algorithm Weight Space wtx(3)<0 wtx(3)>0 w2 Initial weight vector 2.5 w(1) 1.75 2 x(2) x(4) x(1) wtx(2)>0 wtx(2)<0 x(3) w1 wtx(1)>0 wtx(4)>0 wtx(4)<0 4 wtx(1)<0 3 1 The Perceptron Training Algorithm The training has so far been shown in the weight space. This is achieved using the decision lines defined by x(1), x(2), x(3) nd x(4). However, the original decision lines determined by the perceptron at each step are defined in the pattern space as this enables the classification to be easily seen. These decision lines are defined by w(1), w(2) w(3) and w(4). In the following we show the correction steps of the weight vector as well as the corresponding decision surfaces in the pattern space. The Perceptron Training Algorithm In the pattern space w( 1 )x 0 determines the the decision line defined by the initial weight vector 2.5 w(1) 1.75 as x1 w (1) x 2.5 1.75 2.5 x1 1.75 x2 0 x2 1.429 x1 x2 t The Perceptron Training Algorithm The corresponding gradient vector is computed as follows: ( wt ( 1 )x ) w ( 1 ) 2.5 x 1 t t 1 w ( 1 )x 2.5 x1 x2 ( w ( 1 )x w ( 1 ) t 1.75 ( w ( 1 )x ) w2 ( 1 ) x2 which is the initial weight vector. As the gradient vector lies on the side of w (1) x 2.5 x1 1.75 x2 0 t where w (1) x 2.5 x1 1.75 x2 0 t The Perceptron Training Algorithm However, x(1) and x(3) have class 1 ,i.e., d1= d3=1 and x(2) and x(4) have class 2 ,i.e., d2= d4=-1. This means that x(1), x(2), x(3), x(4) all are wrongly classified. The Perceptron Training Algorithm Weight Space Initial weight vector 2.5 w( 1 ) 1.75 Pattern Space x wt ( 1 )x 2.5 1.75 1 2.5 x1 1.75 x2 0 x2 1.429 x1 x2 Initial decision line Weight vector is orthogonal to corresponding decision line The Perceptron Training Algorithm Weight Space w2 Pattern Space x2 wt ( 1 ) x 0 w( 1 ) (is orthogonal todecion line) x(2) Initial weight vector w1 x(4) wt ( 1 ) x 0 x(1) 2.5 w(1) 1.75 x wt ( 1 )x 2.5 1.75 1 2.5 x1 1.75 x2 0 x2 1.429 x1 x2 Decision line for initial weight vector x(3) x1 The Perceptron Training Algorithm Pattern x(1) is input Weight Space Pattern Space Initial weight vector 2.5 w( 1 ) 1.75 x wt ( 1 )x 2.5 1.75 1 2.5 x1 1.75 x2 0 x2 1.429 x1 x2 Initial decision line Weight vector is orthogonal to corresponding decision line Initial decision line wt x( 1 ) w1 1 w2 w1 w2 0 w2 w1 1 1 x( 1) 1 First input vector Decision line is orthogonal to corresponding input vector The Perceptron Training Algorithm Step 1:Pattern x(1) is input Weight Space Pattern Space w2 1 x2 wt ( 1 ) x 0 w( 1 ) wt x( 1 ) 0 x(1) Initial weight vector 2.5 w(1) 1.75 wt x( 1 ) w1 x(2) w1 wtx(1)>0 wtx(1)<0 1 w2 w1 w2 0 w2 w1 1 x(4) wt ( 1 ) x 0 x(1) x(3) x1 Line 1 is decision line x wt ( 1 )x 2.5 1.75 1 2.5 x1 1.75 x2 0 x2 1.429 x1 x2 The Perceptron Training Algorithm Step 1 (Update 1): Pattern x(1) is input Weight Space Pattern Space Initial weight First input Initial decision line vector vector 2.5 w( 1 ) 1.75 1 x( 1) 1 x wt ( 1 )x 2.5 1.75 1 2.5 x1 1.75 x2 0 x2 1.429 x1 x2 1 y( 1 ) sgn( wt ( 1 )x( 1 )) sgn( 2.5 1.75 ) 1 1 d ( 1 ) y( 1 ) 1 ( 1 ) 2 Updated weight vector 2.5 1 1.5 w(2) w(1) x(1) 1.75 1 2.75 Updated decision line x wt ( 2 )x 1.5 2.75 1 1.5 x1 2.75 x2 0 x2 0.545 x1 x2 The Perceptron Training Algorithm Step 1 (Update 1): :Pattern x(1) is input Weight Space Pattern Space 1.5 w( 2 ) w2 x2 2.75 w( 2 ) wt ( 2 ) x 0 wt x( 1 ) 0 x(1) Initial weight vector 2.5 w(1) 1.75 wt x( 1 ) w1 x(2) x(4) w1 wt ( 2 ) x 0 wtx(1)>0 wtx(1)<0 1 w2 w1 w2 0 w2 w1 1 x(1) 2 x(3) x1 Line 2 is decision line x wt ( 2 )x 1.5 2.75 1 1.5 x1 2.75 x2 0 x2 0.545 x1 x2 The Perceptron Training Algorithm Pattern Space Weight Space Step 2 (Update 2): :Pattern x(2) is input Weight vector to be updated 1.5 w( 2 ) 2 . 75 Second input vector 0.5 x( 2 ) 1 Decision line to be updated x wt ( 2 )x 1.5 2.75 1 1.5 x1 2.75 x2 0 x2 0.545 x1 x2 0. 5 y( 2 ) sgn( wt ( 2 )x( 2 )) sgn( 1.5 2.75 ) 1 1 d ( 2 ) y( 2 ) 1 1 ) 2 Second update 1.5 0.5 1 w(3) w(2) x(2) 1 1.75 2.75 Updated decision line x wt ( 3 )x 1 1.75 1 x1 1.75 x2 0 x2 0.57 x1 x2 The Perceptron Training Algorithm Step 2 (Update 2): :Pattern x(2) is input Weight Space Pattern Space w( 3 ) w2 x2 w( 3 ) wt ( 3 ) x 0 wt x( 2 ) 0 wtx(2)>0 x(2) Initial weight vector wtx(2)<0 w1 w1 x(2) x(4) 0 .5 w2 w1 0.5w2 0 w2 0.5w1 1 3 x(1) x(3) w t ( 3 ) x 0 x1 Line 3 is decision line The Perceptron Training Algorithm Pattern Space Weight Space Step 3 (Update 3): :Pattern x(3) is input Weight vector to be updated 1 w( 3 ) 1.75 Third input vector Decision line to be updated 3 x( 3 ) 1 x wt ( 3 )x 1 1.75 1 x1 1.75 x2 0 x2 0.57 x1 x2 Step 3:Pattern x(3) is input 3 y( 3 ) sgn( w ( 3 )x( 3 )) sgn( 1 1.75 ) 1 1 t d ( 3 ) y( 3 ) 1 ( 1 ) 2 1 3 2 w(4) w(3) x(3) 1.75 1 2.75 Updated decision line x wt ( 4 )x 2 2.75 1 2 x1 2.75 x2 0 x2 0.73 x1 x2 The Perceptron Training Algorithm Step 3 (Update 3): Pattern x(3) is input Weight Space Pattern Space w2 x2 w(4) wt x( 3 ) 0 wtx(3)>0 Initial weight vector x(3) x(4) w1 wtx(3)<0 w1 wt ( 4 ) x 0 3 w2 3w1 w2 0 w2 3w1 1 x(2) wt ( 4 ) x 0 x(1) x(3) x1 4 Line 4 is decision line The Perceptron Training Algorithm Pattern Space Weight Space Step 4 (Update 4): :Pattern x(4) is input Weight vector to be updated 2 w( 4 ) 2.75 Fourth input vector Decision line to be updated 2 x( 4 ) 1 x wt ( 4 )x 2 2.75 1 2 x1 2.75 x2 0 x2 0.73 x1 x2 Step 4:Pattern x(4) is input 2 y( 4 ) sgn( w ( 4 ) x( 4 )) sgn( 2 2.75 ) 1 1 t d ( 4 ) y( 4 ) 1 ( 1 ) 0 2 w(5) w(4) 2.75 No update, same decision line x wt ( 4 )x 2 2.75 1 2 x1 2.75 x2 0 x2 0.73 x1 x2 The Perceptron Training Algorithm Step 4 (Update 4): Pattern x(4) is input Weight Space Pattern Space w2 x2 w(5) =w(4) wt x( 4 ) 0 x(2) x(4) Initial weight vector wtx(4)>0 w1 wt ( 4 ) x 0 w1 wtx(4)<0 2 w2 2 w1 w2 0 w2 2 w1 1 x(4) wt ( 4 ) x 0 x(1) x(3) x1 5 =4 Line 4 remains decision line The Perceptron Training Algorithm Step 5 (Update5): :Pattern x(1) is input Weight vector to be updated First input vector 2 w( 5 ) w( 4 ) 2.75 1 x( 1) 1 Decision line to be updated x wt ( 4 )x 2 2.75 1 2 x1 2.75 x2 0 x2 0.73 x1 x2 Step 5:Pattern x(1) is input 1 y( 5 ) sgn( wt ( 5 ) x( 1 )) sgn( wt ( 4 )x( 1 )) sgn( 2 2.75 ) 1 1 d ( 1 ) y( 5 ) 1 1 0 2 w(6) w(5) w(4) 2.75 No update, same decision line x wt ( 4 )x 2 2.75 1 2 x1 2.75 x2 0 x2 0.73 x1 x2 The Perceptron Training Algorithm Step 5 (Update5): :Pattern x(1) is input Weight Space Pattern Space w2 x2 w(6) =w(5)= w(4) wt x( 1 ) 0 x(1) w1 Initial weight vector wtx(1)<0 w1 wtx(1)>0 1 w2 w1 w2 0 w2 w1 1 wt ( 4 ) x 0 x(2) x(4) wt ( 4 ) x 0 x(1) x(3) x1 6 5 4 Line 4 remains decision line The Perceptron Training Algorithm Step 6 (Update 6): :Pattern x(2) is input Weight vector to be updated Second input vector Decision line to be updated 0.5 2 w( 6 ) w( 5 ) w( 4 ) x( 2 ) 2 . 75 1 x wt ( 4 )x 2 2.75 1 2 x1 2.75 x2 0 x2 0.73 x1 x2 Step 6:Pattern x(2) is input 0.5 y (6) sgn( wt (6) x(2)) sgn( wt (6) x(2)) sgn( 2 2.75 ) 1 1 d ( 2 ) y( 6 ) 1 1 ) 2 Updated decision line 2 0.5 2.5 t x1 w( 7 ) w( 4 ) x( 2 ) w ( 7 ) x 2 . 5 1 . 75 x 2.5x1 1.75x2 0x2 1.43x1 2 2.75 1 1.75 The Perceptron Training Algorithm Step 6 (Update 6): :Pattern x(2) is input Weight Space Pattern Space w2 x2 w(7) wt x( 2 ) 0 wt ( 7 ) x 0 wtx(3)>0 Initial weight vector w(7) x(3) x(4) w1 x(2) x(1) 7 wtx(3)<0 w1 2 w2 2 w1 w2 0 w2 2 w1 1 x(3) x1 wt ( 7 ) x 0 The Perceptron Training Algorithm Step 7 (Update 7): :Pattern x(3) is input Weight Space Pattern Space w2 x2 w(4) wt x( 3 ) 0 wtx(3)>0 x(2) Initial weight vector x(2) w1 x(4) wtx(3)<0 w1 wt ( 3 ) x 0 3 w2 3w1 w2 0 w2 3w1 1 3 x(1) x(3) w t ( 3 ) x 0 x1 The Perceptron Training Algorithm Step 7:Pattern x3 is input 3 y (7) sgn( wt (7) x(3)) sgn( 2.5 1.75 ) 1 1 d ( 3 ) y( 7 ) 1 1 0 2.5 w(8) w(7) 1.75 The Perceptron Training Algorithm Step 8:Pattern x4 is input 2 y( 8 ) sgn( w ( 8 )x( 4 )) sgn( w ( 7 )x( 4 )) sgn( 2.5 1.75 ) 1 1 t t d ( 4 ) y( 7 ) 1 ( 1 ) 0 2.5 w(9) w(8) w(7) 1.75 The Perceptron Training Algorithm Step 9:Pattern x(1) is input 1 y( 9 ) sgn( w ( 9 )x( 1 )) sgn( w ( 7 )x( 1 )) sgn( 2.5 1.75 ) 1 1 t t d ( 1 ) y( 9 ) 1 1 0 2.5 w(10) w(9) w(8) w(7) 1.75 The Perceptron Training Algorithm Step 10:Pattern x(2) is input 1 y( 10 ) sgn( w ( 10 )x( 2 )) sgn( w ( 7 )x( 1 )) sgn( 2.5 1.75 ) 1 1 t t d ( 2 ) y( 10 ) 1 1 2 2.5 w(11) 1.75 The Perceptron Training Algorithm The Perceptron Training Algorithm Pattern Space x2 Weight Space w2 wt x( 1 ) 0 wt ( 1 ) x 0 w( 1 ) x(1) Initial weight vector 2.5 w(1) 1.75 wt x( 1 ) w1 x(2) w1 x(4) wt ( 1 ) x 0 x(1) wtx(1)>0 wtx(1)<0 1 w2 w1 w2 0 w2 w1 1 x wt ( 1 )x 2.5 1.75 1 2.5 x1 1.75 x2 0 x2 1.429 x1 x2 x(3) x1 The Perceptron Training Algorithm The initial weight vector w(1) and the weight vectors w(2)- w(11) obtained during the training algorithm are given below: 2.5 1.5 1 2 , w(2) , w(3) , w(4) , w(1) 1.75 2.75 1.75 2.75 2.5 w(5) w(4), w(6) w(5) w(4), w(7) , 1.75 3 w(8) w(7), w(9) w(8) w(7), w(10) w(9) w(8) w(7), w(11) 0.75 As can be seen from these vectors, out of the ten vectors w(2)- w(11) only five are different . The Perceptron Training Algorithm These five vectors are given in the MATLAB plot below: 3 2.5 2 1.5 1 0.5 0 -0.5 -1 -1.5 -2 -8 -6 -4 -2 0 2 4 6 8 10 12 The Perceptron Training Algorithm x3 Example: The trained (0,0,1) classifier is required to provide the (0,0,1) classification such that the yellow vertices (0,1,1) 0 of the cube have class (0,0,0) membership d=1 and the blue vertices have (0,1,0) class membership d=2. x2 (1,0,1) x1 (1,0,0) (1,1,0) x3=0.25 plane The Perceptron Training Algorithm The Perceptron Training Algorithm SUMMARY OF CONTINUOUS PERCEPTRON TRAINING ALGORITHM Given are the p training pairs {x1,d1, x2,d2,…………….,xp,dp}, where xi is (N+1) x 1 Di is 1 x 1, i=1,2, ,P. In the following n denotes the training step and p denotes the step counter within the training cycle. Step 1: c>0 is chosen. Step2: Weights are initialized at w at random small values, w is (N+1) x 1. Counters and error are initialized. 1 k ,1 p and 0 E The Perceptron Training Algorithm Step3: The training cycle begins here. Input is presented and output is computed: x p x, d p d , y sgn( w x) t Step4: Weights are updated: → x p x, w 1 w c(d y ) x 2 Step5: Cycle error is computed. 1 E (d y )2 2 The Perceptron Training Algorithm Step 6: If p<P then p p 1, n n 1 and go to Step 3, otherwise go to Step 7. Step 7: The training cycle is completed. For E=0, terminate the training session. Output weights, k and E. If E>0 then enter the new training cycle by going to Step 3. Single-Layer Continuous Perceptron 1 2 Here the activation function is a continuous function of the weights instead of the signum. There are two main objectives of this: To define a continuous function of the weights as the error function so as to obtain finer control over the weights as well as over the whole training procedure; To enable the computation of the error gradient in order to be continuously in a position to know the direction in which the error decreases. Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule The Delta Training Rule is based on the minimisation of the error function which is given by 1 2 E( n ) ( d ( n ) y( n )) 2 where n is a positive integer representing the traning step number, i.e.,the step number in the minimisation process, d(n) is the desired output signal and t y (n) f [ w (n) x(n)] is the actual output. Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule The error function (error surgface) is a function of the weights: E(w1,w2,....,wp)=E(w) which is minimised using an iterative minimisation method which computes the new values of the weights according to w(n 1) w(n) w(n) where w(n) is the increment given to the present weight vector w(n) to obtain the new weight vector w(n+1). Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule Let us now use the steepest descent method for the minimisation of the error function E(w) where it is required that the weight changes be in the negative gradient direction. Therefore we take: ∇ w(n) E ( w(n)) where E(w(n)) is the gradient vector and is called the learning constant. Using this in the equation above, we obtain w( n 1 ) w( n ) E( w( n )) Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule E( w( n )) E( n ) is the error surface at the n’th training step . Therefore the error to be minimised is: 1 E( n ) ( d ( n ) f ( wt ( n )x( n ))2 2 The independent variables for minimisation at each training step are wi, the components of the weight vector. Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule The error minimisation requires the computation of the gradient of the error function: E( w )w w( n ) 1 2 t ( d ( n ) f ( w x( n )) 2 w w( n ) Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule The gradient vector is defined as: E w 1 E w2 E ( w ) . . E w p 1 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule Using E( w )w w( n ) 1 t 2 ( d ( n ) f ( w x( n )) 2 w w( n ) and defining v( w ) w x t we obtain Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule E( w )ww( n ) v( w ) w 1 v( w ) df ( v( w )) w2 d ( n ) f ( v( w )) . dv . v( w ) w p 1 w w( n ) Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule Since and we can write v( w) xi wi f(v) y Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule E ( w)w w( n ) df (v( w)) d ( n) y ( n) x ( n) dv ww ( n ) and E ( w) df (v( w)) x i ( n) d ( n) y ( n) dv ww ( n ) wi ww( n ) Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule If bipolar continuous activation function is used then we have: v and 1 e f (v ) v 1 e v df (v) 2e v 2 dv (1 e ) Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule In fact v df (v) 2e v 2 dv 1 e v 2 1 e 1 1 2 1 (v) 1 f v 2 1 e 2 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule E( w )w w( n ) 1 d ( n ) y( n )1 y 2 ( n ) x( n ) 2 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule Conclusion: The delta training rule for the bipolar continuous perceptron is given as: 1 2 w( n 1 ) w( n ) d ( n ) y( n )1 y ( n ) x( n ) 2 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule If unipolar continuous activation function is used then we have: and 1 f (v ) v 1 e v df (v) e v 2 dv (1 e ) Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule we can write df (v) e v 1 1 ev 1 1 1 (1 ) 2 v v v v v 1 e 1 e dv 1 e 1 e 1 e =f (v)(1 f (v)) Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule Example:We will carry out the same training algorithm as in the previous example but this time using a continuous bipolar perceptron. The error at step n is given by: 2 1 1 e 1 2 d ( n ) 1 E ( n ) d ( n ) v( n ) v( n ) 2 1 e 2 1 e v( n ) 2 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule For the first pattern x(1)=[1 1]t , d(1)=1. The error at step 1 is given by: 2 1 2 E (1) d (1) 1 ( w1 w2 ) 2 1 e 2 2 1 2 2 1 2e 1 1 ( w1 w2 ) ( w1 w2 ) ( w1 w2 ) 2 e e 2 1 e 2 1 (1 ) ( w1 w2 ) Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule For the second pattern x(2)=[-0.5 1]t , d(2)=-1. The error at step 2 is given by: 2 1 2 1 E ( 2 ) d ( 2 ) ( 0.5 w1 w2 ) 2 1 e 2 1 2 1 2 2 1 1 ( 0.5 w1 w2 ) ( 0.5 w1 w2 ) 2 1 e 2 1 e 1 e( 0.5 w1 w2 ) 2 2 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule For the third pattern x(3)=[3 1]t , d(3)=1. The error at step 3 is given by: 2 1 2 1 E ( 3 ) d ( 3 ) ( 3 w1 w2 ) 2 1 e 2 2 1 2 1 2e 2 1 1 ( 3 w1 w2 ) ( 3 w1 w2 ) ( 3 w1 w2 ) 2 2 1 e 1 e 1 e ( 3 w1 w2 ) 2 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule For the fourth pattern x(4)=[-2 1]t , d(4)=-1. The error at step 4 is given by: 2 1 2 1 E ( 4 ) d ( 4 ) ( 2 w1 w2 ) 2 1 e 2 1 2 1 2 2 1 1 ( 2 w1 w2 ) ( 2 w1 w2 ) 2 2 1 e 1 e 1 e( 2 w1 w2 ) 2 2 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule close all; clear all; [w1,w2] = meshgrid(-4:.1:4, -4:.1:4); Z1 = exp(w1+w2); E1=2./(1+Z1).^2 mesh(Z1) Z2 = exp(.5*w1-w2); E2=2./(1+Z2).^2 figure,mesh(Z2) Z3 = exp(3*w1+w2); E3=2./(1+Z3).^2 figure,mesh(Z3) Z4 = exp(2*w1-w2); E4=2./(1+Z4).^2 figure,mesh(Z4) subplot(2,2,1),mesh(E1);title('Error surface for xt(1)=[1 1] and y=f(wt*x(1))'); xlabel('w1'),ylabel('w2'),zlabel('E1(w1,w2)'); Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule subplot(2,2,2),mesh(E2);title('Error surface for xt(2)=[.5 -1] and y=f(wt*x(2))'); xlabel('w1'),ylabel('w2'),zlabel('E2(w1,w2)'); subplot(2,2,3),mesh(E3);title('Error surface for xt(3)=[3 1] and y=f(wt*x(3))'); xlabel('w1'),ylabel('w2'),zlabel('E3(w1,w2)'); subplot(2,2,4),mesh(E4);title('Error surface for xt(4)=[2 -1] and y=f(wt*x(4))'); xlabel('w1'),ylabel('w2'),zlabel('E4(w1,w2)'); E = E1+E2+E3+E4; figure,mesh(E);title('Total Error E(w1,w2)=E1(w1,w2)+E2(w1,w2)+E3(w1,w2)+E4(w1,w2),MESH'); xlabel('w1'),ylabel('w2'),zlabel('E(w1,w2)'); figure,imshow(E,[],'notruesize');colormap(jet);title('Total Error E(w1,w2)=E1(w1,w2)+E2(w1,w2)+E3(w1,w2)+E4(w1,w2),IMSHOW'); xlabel('w1'),ylabel('w2') The error surfaces for the above four cases are shown in the next slide: Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule Error surface for xt(1)=[1 1] and y=f(wt*x(1))Error surface for xt(2)=[.5 -1] and y=f(wt*x(2)) 2 E2(w1,w2) E1(w1,w2) 2 1 0 0 100 50 w1 100 0 1 0 100 100 50 50 w2 w2 0 0 50 w1 Error surface for xt(3)=[3 1] and y=f(wt*x(3))Error surface for xt(4)=[2 -1] and y=f(wt*x(4)) 2 E4(w1,w2) E3(w1,w2) 2 1 1 0 100 100 0 0 50 w1 100 0 50 w2 50 w2 0 0 50 w1 100 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule The total error is defined by: E( w1 , w2 ) E1( w1 , w2 ) E2 ( w1 , w2 ) E3 ( w1 , w2 ) E4 ( w1 , w2 ) The total error surface is shown in the next slide. Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule A contour map of the total error is depicted below: Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule The classifier training has been simulated for = 0.5 for four arbitrarily chosen initial weight vectors, including the one taken from x(1)= 1, x(3) =3, x(2)= -0.5, x(4)=-2, d1= d3 = 1: class C1 d2 = d4 = -1: class C2 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule The resulting trajectories of 150 simulated training steps are shown in the following figure (each tenth step is shown). Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule In each case the weights converge during training toward the center of the solution region obtained for the discrete perceptron case given on the next slide, which coincides with the dark blue region in the contour map of the total error is depicted before and also shown on the next slide. Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule SUMMARY OF CONTINUOUS PERCEPTRON TRAINING ALGORITHM Given are the p training pairs {x1,d1, x2,d2,…………….,xp,dp}, where xi is (N+1) x 1 Di is 1 x 1, i=1,2, ,P. In the following n denotes the training step and p denotes the step counter within the training cycle. Step 1: >0, =1 and Emax>0 chosen. Step2: Weights are initialized at w at random small values, w is (N+1) x 1. Counters and error are initialized. 1 k ,1 p and 0 E Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule Step3: The training cycle begins here. Input is presented and output is computed: x p x, d p d , y f ( w x ) t Step4: Weights are updated: 1 x p x, w w (d y )(1 y 2 ) x 2 Step5: Cycle error is computed. 1 E (d y )2 2 Training Rule for a Single-Layer Continuous Perceptron:The Delta Training Rule Step 6: If p<P then p p 1, n n 1 and go to Step 3, otherwise go to Step 7. Step 7: The training cycle is completed. For E<Emax terminate the training session. Output weights, k and E. If E>Emax then enter the new training cycle by going to Step 3. Delta Training Rule for Multi-Perceptron Layer w11 w12 w21 x1 x2 wjk vk xj w1J xJ -1 v2 wK1 w1j j-th column of nodes v1 wK2 wKJ vK 1 y1 2 y2 k yk K yK Neurons k-th column of nodes Delta Training Rule for Multi-Perceptron Layer The above can be redrawn as: Delta Training Rule for Multi-Perceptron Layer v1 w11 x1 w12 x2 ......... w1 j x j ...... w1J xJ y1 f ( v1 ) v2 w21 x1 w22 x2 ......... w2 j x j ...... w2 J xJ y2 f ( v2 ) vl wl1 x1 wl 2 x2 ......... wlj x j ...... wlJ xJ yl f (vl ) vK wK 1 x1 wK 2 x2 ......... wKj x j ...... wKJ xJ y K f ( vK ) Delta Training Rule for Multi-Perceptron Layer v1 w11 v w 2 21 . . . . vK wK 1 w12 w22 . . wK 2 . . w1J x1 . . w1J x2 . . . . . . . . . . wKJ xJ v Wx v1 y1 f ( v1 ) y f ( v ) v2 2 2 . . Γ . . . . yK f ( vJ ) v J y Γ(v) Delta Training Rule for Multi-Perceptron Layer y1 f (.) y 0 2 . . . . yK 0 0 . . f (.) . . . . . . 0 . . . . v1 v 2 . . . f (.) vJ 0 0 . y Γ[Wx] Delta Training Rule for Multi-Perceptron Layer The desired and actual output vectors at the nth training step are given as: d1 (n) d ( n) 2 d . . d K (n) y1 (n) y ( n) 2 y . . yK (n) The error expression for a single perceptron was given as: 1 2 E( n ) ( d ( n ) y( n )) 2 Delta Training Rule for Multi-Perceptron Layer which can be generalised to include all squared errors at the outputs k=1,2,.....,K 2 1 K 1 2 E( n ) ( d k ( n ) yk ( n )) d( n ) y( n ) 2 k 1 2 where n represents the n-th step which corresponds to a specific input pattern that produces the output error. Delta Training Rule for Multi-Perceptron Layer The updated weight value from input j to neuron k at step n is given by: wkj ( n 1 ) wkj ( n ) Δwkj ( n ) According to the delta training rule for continuous perceptron E Δwkj ( n ) wkj w k 1,2 ,..., K for j 1,2 ,..., J kj wkj ( n ) where Delta Training Rule for Multi-Perceptron Layer E E vk E E (v( w)) w v w k kj kj Using vk wk1 x1 wk 2 x2 ......... wkj x j ...... wkJ xJ we have vk xj wkj Delta Training Rule for Multi-Perceptron Layer The error signal term produced by the kth neuron is defined as: yk Using this yields E vk E yk x j wkj Delta Training Rule for Multi-Perceptron Layer On the other hand we can write: yk E E yk vk yk vk Since 2 1 K 1 2 E( n ) ( d k ( n ) yk ( n )) d( n ) y( n ) 2 k 1 2 we get E ( d k yk ) yk Delta Training Rule for Multi-Perceptron Layer On the other hand using yk f ( vk ) vk vk yields yk E E yk f ( vk ) ( d k yk ) vk yk vk vk Delta Training Rule for Multi-Perceptron Layer which is used to obtain f ( vk ) Δwkj ( n ) ( d k yk ) xj vk For bipolar continuous activation function we already know that f ( vk ) 1 1 2 2 1 ( f ( vk )) 1 yk vk 2 2 Delta Training Rule for Multi-Perceptron Layer Hence 1 2 Δwkj ( n ) ( d k ( n ) yk ( n ))( 1 ( yk ( n )) )x j 2 and 1 wkj ( n 1 ) wkj ( n ) ( d k ( n ) yk ( n ))( 1 ( yk ( n ))2 )x j ( n ) 2 where Delta Training Rule for Multi-Perceptron Layer yk ( n ) ( d k ( n ) yk ( n ))( 1 ( yk ( n )) ) 2 we can write y1 ( n ) w11( n 1 ) . w1J ( n 1 ) w11( n ) . w1J ( n ) . . x ( n ) . x ( n ) . . . . . J 1 yK ( n ) wK 1( n 1 ) . wKJ ( n 1 ) wK 1( n ) . wKJ ( n ) Delta Training Rule for Multi-Perceptron Layer Now defining x1 (n) x ( n) . xJ (n) 1 (n) ( n) . J (n) W (n 1) W (n) δy x t Generalised Delta Training Rule for Multi- Layer Perceptron z1 z2 . t11 u1 t21 u2 . zI x2 uj . t j1 zi . . x1 w11 v1 wk1 v2 wK1 wK2 wkj vk xj tIJ i-th column of nodes uJ xJ wkJ j-th column of nodes (hidden layer) vK 1 y1 2 y2 k K yk yK k-th column of nodes Generalised Delta Training Rule for Multi- Layer Perceptron The weight adjustment for the hidden layer according to the gradient descent method will be: E Δt ji ( n ) t ji where t ji t ji ( n ) j 1,2 ,..., J for i 1,2 ,..., I E E u j t ji u j t ji Generalised Delta Training Rule for Multi- Layer Perceptron Here E xj for u j j 1,2 ,...., J is the error signal term of the hidden layer with output x. This term is produced by the j-th neuron of the hidden layer, where j=1,2,....,J. On the other hand, using u j t j1 z1 t j 2 z2 ............... t jI z I Generalised Delta Training Rule for Multi- Layer Perceptron u j we can calculate t ji as u j Therefore t ji zi E E u j xj zi t ji u j t ji Generalised Delta Training Rule for Multi- Layer Perceptron and Since Δt ji xj zi xj f ( uj ) E E x j xj u j x j u j Generalised Delta Training Rule for Multi- Layer Perceptron E 1 2 d k f ( vk ) x j x j 2 k 1 K and x j u j f ( u j ) u j Generalised Delta Training Rule for Multi- Layer Perceptron E f ( vk ) ( d k f ( vk ) x j x j k 1 K f ( vk ) vk ( d k yk ) vk x j k 1 K Now using vk wk1 x1 wk 2 x2 ......... wkj x j ...... wkJ xJ Generalised Delta Training Rule for Multi- Layer Perceptron we have vk wkj x j Now using this equality and yk E f ( vk ) ( d k yk ) vk vk Generalised Delta Training Rule for Multi- Layer Perceptron in E f ( vk ) ( d k f ( vk ) x j x j k 1 K f ( vk ) vk ( d k yk ) vk x j k 1 K we obtain Generalised Delta Training Rule for Multi- Layer Perceptron E yk wkj x j k 1 K vk wkj in Now using this and x j E E x j xj u j x j u j Generalised Delta Training Rule for Multi- Layer Perceptron we obtain xj Now using f ( u j ) u j K k 1 yk wkj Δt ji xj zi we get Δt ji f ( u j ) u j K zi yk wkj k 1 Generalised Delta Training Rule for Multi- Layer Perceptron f ( u j ) t ji ( n 1 ) t ji ( n ) yk wkj zi k 1 u j for j 1,2 ,..., J K i 1,2 ,..., I Generalised Delta Training Rule for Multi- Layer Perceptron Now defining the j-th column of the matrix w11 w 21 W . . wK 1 w12 w22 . . wK 2 as wj . . w1J . . w1J . . . . . . . . wKJ Generalised Delta Training Rule for Multi- Layer Perceptron and using y1 δy . yK we can write K k 1 yk wkj w δy t j Generalised Delta Training Rule for Multi- Layer Perceptron In the case of bipolar activation function we obtain for the hidden layer f ( u j ) u j 1 2 f (1 x j ) 2 ' xj Now construct a vector whose entries are the above terms for j=1,2,...,J, i.e., Generalised Delta Training Rule for Multi- Layer Perceptron 1 2 ( x ) 1 ' 1 f x1 2 ' 1 2 f x 2 ( 1 x2 ) 2 ' fx f ' 1 xJ ( 1 x 2 ) J 2 Generalised Delta Training Rule for Multi- Layer Perceptron and define z1 z 2 z . . z I We then have ' t ' yk wkj f j zi ( w j δy ) f x z k 1 K Generalised Delta Training Rule for Multi- Layer Perceptron Now defining δx w δy f t j ' x and t11 t12 t 21 t22 T . . . . t J 1 t J 2 . . t1I . . t1I . . . . . . . . t JI Generalised Delta Training Rule for Multi- Layer Perceptron we finally obtain T ( n 1 ) T ( n ) δx z t This updating formula is called the Generalised Delta Rule for adjusting the hidden layer weights. A similar formula was given for updating the output layer weights: W (n 1) W (n) δy x t Generalised Delta Training Rule for Multi- Layer Perceptron Here the main difference is in computing the error signals y and x. In fact, the entries of y are given as yk f ( vk ) ( d k yk ) vk which only contain terms belonging to the output layer. However, this is not the case with x , Generalised Delta Training Rule for Multi- Layer Perceptron δx w δy f t j ' x f x' whose entries are weighted sum of error signals yk produced by the following layer. Here we can draw the following conclusion: The Generalised Delta Learning Rule propagates the error back by one layer which is true for every layer. Generalised Delta Training Rule for Multi- Layer Perceptron - Summary of the Error Back-Propagation Training Algorithm (EBPTA) Given are P training pairs where zi is (1 Xl), di is (K X 1), and i = 1, 2, ... , P. Note that the l'th component of each zi is of value -1 since input vectors have been augmented. Size J - 1 of the hidden layer having outputs y is Error Back-Propagation Training Algorithm (EBPTA) selected. Note that the J'th component of y is of value -1, since hidden layer outputs have also been augmented; y is (J X 1) and 0 is (K Xl). Error Back-Propagation Training Algorithm (EBPTA) Step 1: 'T/ > 0, Emax chosen . Weights W and V are initialized at small random values; W is (K X J), V is (J X /). q f-- 1, p f-- 1, E f-- 0 . Step 2: Training step starts here (See Note 1 at end of list.) Input is presented and the layers' outputs computed [f(net) as in (2.3a) is used]: Error Back-Propagation Training Algorithm (EBPTA) Z f-- zP' d f-- dp Yj f-- f(vjz), for j = 1, 2, ... , J where ~'" a column vector, is the j'th row of V, and J . ok f-- f(w~y), . for k = 1, 2, ... , K where Wk, a column vector, is the k'th row of W. Step 3: Error value is computed: 1 2 E f-- '2(dk - Ok) + E, for k = 1, 2, ... , K Error Back-Propagation Training Algorithm (EBPTA) Step 4: Error signal vectors 80 and 8y of both layers are computed. Vector 80 is (K Xl), 8y is (J X 1). (See Note 2 at end of list.) The error signal terms of the output layer in this step are Error Back-Propagation Training Algorithm (EBPTA) 1 2 Ook = -(dk- 0k)(l - Ok)' for k = 1, 2, ... , K 2 The error signal terms of the hidden layer in this step are 12K 0yj = _ (1 - Yj) L 0okWkj, for j = 1, 2, ... , J k=l Step 5: Output layer weights are adjusted: Wkj f-- Wkj + T/OokYj, for k = 1, 2, ... , K and j = 1, 2, ... ,J Error Back-Propagation Training Algorithm (EBPTA) Step 6: Hidden layer weights are adjusted: Vji f-- Vji + T/OyPi' for j = 1, 2, ... ,J and i = 1, 2, ... ,I Step 7: If p < P then p f-- P + 1, q f-- q + 1, and go to Step 2; otherwise, go to Step 8. Error Back-Propagation Training Algorithm (EBPTA) Step 8: The training cycle is completed. For E < Emax terminate the training session. Output weights W, V, q, and E. If E > Emax' then E f- 0, P f- 1, and initiate the new training cycle by going to Step 2. NOTE 1 For best results, patterns should be chosen at random from the training set Qustification follows in Section 4.5). Error Back-Propagation Training Algorithm (EBPTA) 11IIIII NOTE 2 If formula (2.4a) is used in Step 2, then the error signal terms in Step 4 are computed as follows 80k = (dk - 0k)(l - 0k)ok' for k = 1, 2, ... , K K 8y) = y}(l - y) L 8okWk}' for j = 1, 2, ... , J k=l The Hopfield Network The Hopfield Network We know that the Hopfield Network is a Recurrent (Feedback or Dynamical) Neural Network. Let yi , i=1,2,.....,n, be the outputs of the network and the energy function E satisfy the following: 2 dE dy i i 0 dt dt i 1 n where i>0 , i=1,2,.....,n. The Hopfield Network The above inequlity reveals that the energy decreases with time and becomes zero if and only if dyi 0 i , dt i.e., yi ( t ) cons tan t i , i.e., yi ( t ) reach their stable equilibrium states . The Hopfield Network Now let us assume that 1 df ( yi ) i Ci dyi where Ci 0 The Hopfield Network For the bipolar activation function ax 1 e y f(x) ax 1 e the inverse function is given by: 1 1 y x f ( y ) ln a 1 y 1 The Hopfield Network The Bipolar Activation Function and its Inverse The Hopfield Network The Derivative of the Inverse of the Bipolar Function dx 2 1 2 dy a 1 y The Hopfield Network We can conclude that df 1( yi ) 0 dyi for 1 yi 1 Therefore 2 dE dyi df ( yi ) Ci 0 dt dyi dt i 1 n -1 The Hopfield Network Considering dxi df ( yi ) df ( yi ) dyi dt dt dyi dt -1 -1 we obtain 2 n d y d f ( y ) dxi dyi dE i i Ci Ci dt dyi dt dt dt i 1 i 1 n -1 The Hopfield Network Now defining x1 x 2 x . . xN y1 y 2 y . . y N C diag ( Ci ) yields t t dE dy d x dx dy C C dt dt dt dt dt The Hopfield Network Since dE dy t E( y ) dt dt We can write dx E ( y ) C dt This reveals that the capacitor current vector is parallel to the negative gradient vector. The Hopfield Network N dxi wij ( y j xi ) g i xi I i Ci dt j 1 wi1 y1 Ii wi2 y2 . . . yN gi wiN xi yi Ci N yi f ( xi ) dxi Ci wij y j wij g i xi I i dt xi f 1( yi ) j 1 j 1 N The Hopfield Network Now define N w j 1 ij g i Gi C diag ( Ci ) G diag ( Gi ), i 1,2,..., N w11 w 12 W . . w1N w12 w22 . . w2 N . . w1N . . w2 N . . . . . . . . wNN x1 x 2 x . . xN I1 I 2 I . . I N The Hopfield Network We obtain N dxi Ci wij y j Gi xi I i dt j 1 consequently dx( t ) C Wy Gx I dt and since dx E ( y ) C dt The Hopfield Network we obtain E( y ) Wy Gx I In the case of bipolar activation function we know that 1 1 y x f ( y ) ln a 1 y 1 The Hopfield Network Therefore the state vector is given as: 1 y1 ln 1 y 1 ln 1 y2 1 1 y2 x a 1 yN ln 1 y N The Hopfield Network We already know N dE dxi dyi Ci dt dt dt i 1 therefore N N dyi dE ( wij y j Gi xi I i ) dt dt i 1 j 1 and N N dyi N dyi N dyi dE ( wij y j Gi xi Ii ) dt dt i 1 dt i 1 dt i 1 j 1 The Hopfield Network Now consider: t d dy dy t t Wy y W ( y Wy ) dt dt dt If W W t then t d dy dy t t t ( y Wy ) W y yW dt dt dt The Hopfield Network t t dy dy dy t dy t t t W y Wy ( y W ) yW dt dt dt dt Therefore d dy t t ( y Wy ) 2 y W dt dt and dy 1 d t yW ( y Wy ) dt 2 dt t The Hopfield Network Now consider the first term of N N dyi N dyi dyi N dE ( wij y j Gi xi Ii ) dt i 1 dt i 1 dt dt i 1 j 1 We can write: N N dyi dy t yW wij y j dt dt i 1 j 1 Now using the above equality, we have N N dyi 1 d t wij y j (y Wy) dt 2 dt i 1 j 1 The Hopfield Network Now consider the second term in the same equation: we can write dyi dyi 1 f ( yi ) xi dt dt yi d 1 f ( yi ) ( f ( y )dy ) dyi 0 1 yi yi dyi d dyi d 1 1 f ( yi ) ( f ( y )dy ) ( f ( y )dy ) dt dyi 0 dt dt 0 1 The Hopfield Network yi N N dE d 1 t 1 ( y Wy Gi f ( yi )dy I i yi ) dt dt 2 i 1 i 1 0 N yi N 1 t 1 E ( y Wy Gi f ( yi )dy I i yi ) 2 i 1 i 1 0 The Hopfield Network In order to obtain the state equations in terms of the outputs yi consider once again N dxi Ci wij y j Gi xi I i dt j 1 Using dxi 2 1 2 dyi a 1 yi The Hopfield Network we obtain N dy i 2 Ci wij y j Gi xi I i 2 a (1 y i ) dt j 1 and dy i dt a (1 y i ) 2 2Ci N w j 1 ij y j Gi xi I i The Hopfield Network a( 1 y ) dy 1 Wy GΓ (y) I diag dt 2Ci 2 i The Hopfield Network g12 g11 x1 g1 y1 C1 dx1 g1 x1 C1 g11 ( y1 x1 )+ g12 ( y2 x1 ) dt g21 g22 g2 x2 y2 C2 dx2 g 2 x2 C2 g 22 ( y2 x2 )+g 21 ( y1 x2 ) dt The Hopfield Network C1 0 dx1 0 dt g11 C1 dx2 g 21 dt g12 y1 g1 g11 g12 0 g 22 y2 0 x1 g 2 g 22 g 21 x2 which yields C1 0 g11 C , W 0 C1 g 21 g12 g1 g11 g12 , G g 22 0 0 g 2 g 22 g 21 and 1 E y1 2 The Hopfield Network g11 y 2 g 21 g11 E ( y1 , y2 ) g 21 yi g12 y1 1 Gi f y dy g 22 y2 i1 0 2 g12 y1 1 G1 g 22 y2 a 0 1 y1 ln 0 1 y1 G2 1 y2 ln 1 y 2 The Hopfield Network and 1 1 y1 E y g11 y1 g12 y2 a G1 ln 1 y 1 1 E ( y1 , y2 ) 1 1 y2 E y g 22 y1 g 21 y1 a G2 ln 1 y 2 2 The Hopfield Network y y 1 2 1 2 2 1 E ( g11 y1 g 22 y2 y1 y2 g12 y1 y2 g12 ) G1 f y dy G2 f 1 y dy 2 0 0 y y 2 1 1 1 1 y 1 y 2 2 E ( g11 y1 g 22 y2 y1 y2 g 21 y1 y2 g12 ) G1 ln dy G2 ln dy 2 a 0 1 y 1 y 0 Now consider yi yi yi 1 y I ln dy ln( 1 y )dy ln( 1 y )dy 1 y 0 0 0 The Hopfield Network and yi I1 ln( 1 y )dy 0 Let Hence ln( 1 y ) u 1 y v then dy du 1 y dv dy yi yi yi 0 0 0 I1 ln( 1 y )dy udv ( uv ]0yi vdu ) yi dy { ln ( 1 y )}( 1 y )] ( 1 y ) 1 y 0 yi 0 The Hopfield Network I1 { ln ( 1 yi )}( 1 yi ) yi I 2 {ln ( 1 yi )}( 1 yi ) yi yi yi yi 1 y I ln dy ln( 1 y )dy ln( 1 y )dy 1 y 0 0 0 I I1 I 2 { ln ( 1 yi )}( 1 yi ) {ln ( 1 yi )}( 1 yi ) The Hopfield Network The Hopfield Network y y 2 1 1 1 1 y 1 1 y 2 2 E ( g11 y1 g 22 y2 y1 y2 g12 y1 y2 g12 ) G1 ln dy G2 ln dy a 0 1 y a 0 1 y 2 1 1 E {g11 y12 g 22 y22 (g12 g 21 )y1 y2 } G1 ( 1 y1 ) ln( 1 y1 ) ( 1 y1 ) ln( 1 y1 ) a 2 1 G2 ( 1 y2 ) ln( 1 y2 ) ( 1 y2 ) ln( 1 y2 ) a The Hopfield Network The Hopfield Network The Hopfield Network The Hopfield Network The Hopfield Network Using The Hopfield Network a( 1 yi2 ) dy 1 diag Wy GΓ (y) I dt 2Ci the state equations are obtained as 2 y 1 1 dy1 a dt 2C1 dy2 0 dt 1 1 y1 0 ln a 1 y g g y G 0 1 12 1 1 11 1 y22 g 21 g 22 y2 0 G2 1 1 y2 a a ln 1 y 2C2 2 dy1 1 (1 y 2 )(ag y ag y G ln 1 y1 ) 1 11 1 12 2 1 dt 2C1 1 y1 dy2 1 (1 y 2 )(ag y ag y G ln 1 y2 ) 2 22 2 21 1 2 dt 2C2 1 y2 Discrete-Time Hopfield Networks Consider the state equation of the Gradient-Type Hopfield Network: dx( t ) C Wy Gx I dt We can write dx( t ) -1 C Wy GΓ (y) I dt Discrete-Time Hopfield Networks As the plot of the inverse bipolar activation function shows the second term in the above equation is zero for high gain neurons. Hence: dx( t ) C Wy I dt Discrete-Time Hopfield Networks Now consider dxi df ( yi ) df ( yi ) dyi dt dt dyi dt -1 Using the this plot we can conclude that df -1 ( yi ) 0 dyi for high gain neurons. -1 Discrete-Time Hopfield Networks Hence dx( t ) 0 dt It follows that 0 Wy I Now let us solve this equation using Jacobi’s algorithm. To this end define: Discrete-Time Hopfield Networks where W' W D = L U L,U and D = diag( wii ) Are the lower and upper triangular and diagonal matrices shown in the following decomposition of W. W w11 w 21 . . w N1 w12 w 22 . . w N2 . . . . . . w1N w11 . w 2N 0 . . . . . . . w NN 0 0 w 22 . . 0 . . . . . 0 . 0 0 0 . 0 w 21 . . w 31 w 32 . . . . . w NN w N1 w N2 . . 0 . . . . . . w N,N1 0 0 w12 0 0 0 .. . . . . 0 0 0 . . . . . . w1N . w 2N . . . w N1,N . 0 Discrete-Time Hopfield Networks Now defining we obtain D diag( wii ) -Dy = W'y + I -1 -1 y = -D W'y - D I Now define -D W' = W -1 -D I I -1 Discrete-Time Hopfield Networks y = Wy I Now replace the vector y on the right-hand side by an initial y(0) vector. If the vector y on the lefthand side is obtained as y(0), then y(0) is the solution of the system. If not then call the vector y obtained on the left-hand side y(1), i.e., Discrete-Time Hopfield Networks y(1) = W y(0) I and in general we can write y(k + 1) = W y(k) I Discrete-Time Hopfield Networks The method will always converge if the matrix W is strictly or irreducibly diagonally dominant. Strict row diagonal dominance means that for each row, the absolute value of the diagonal term is greater than the sum of absolute values of other terms: wii wij i j Discrete-Time Hopfield Networks The Jacobi method sometimes converges even if this condition is not satisfied. It is necessary, however, that the diagonal terms in the matrix are greater (in magnitude) than the other terms. Discrete-Time Hopfield Networks Solution by Gauss-Seidel Method In Jacobi’s method the updating of the unknowns is made after all N unknowns have been moved to the left side of the equation. We will see in the following that this is not necessary, i.e., the updating can be made individually for each unknown and this updated value can be used in the next equation. This is shown in the following equations: Discrete-Time Hopfield Networks x1 (n 1) x2 (n 1) 1 a21x1 (n 1) a23 x3 (n) ..... a2 N xN (n) b2 a22 a22 x3 ( n 1) and 1 [ a12 x2 (n) a13 x3 (n) ...... a1N xN (n) b1 ] a11 1 a31x1 (n 1) a32 x2 (n 1) a34 x2 (n) ...... a3 N xN (n) b3 a33 a33 xN (n 1) 1 aN 1 x1 ( n 1) aN 2 x2 ( n 1) ...... aN , N 1 xN 1 ( n 1) bN aNN aNN In vector-matrix form, we can write: D x (n 1) L x (n 1) U x (n) b ( D L) x (n 1) U x (n) b x (n 1) ( D L) 1 (U x (n) b) Discrete-Time Hopfield Networks This matrix expression is mainly used to analyze the method. When implementing Gauss Seidel, an explicit entry-by-entry approach is used: 1 xi (n 1) aii bi aij x j (n 1) aij x j (n) j i j i Discrete-Time Hopfield Networks Gauss-Seidel method is defined on matrices with non-zero diagonals, but convergence is only guaranteed if the matrix is either: 1. diagonally dominant or 2. symmetric and positive definite.