Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Financial Informatics –XIV: Basic Principles Khurshid Ahmad, Professor of Computer Science, Department of Computer Science Trinity College, Dublin-2, IRELAND November 19th, 2008. https://www.cs.tcd.ie/Khurshid.Ahmad/Teaching.html 1 1 Neural Networks Artificial Neural Networks The basic premise of the course, Neural Networks, is to introduce our students to an alternative paradigm of building information systems. 2 Artificial Neural Networks An ANN system can be characterised by •its ability to learn; •its dynamic capability; and • its interconnectivity 3 Artificial Neural Networks: An Operational View Neuron xk x1 Summing Junction wk2 x3 wk3 x4 wk4 Activation Function S yk Output Signal Input Signals x2 wk1 bk Net input or weighted sum : net w1 * x1 w2 * x2 w3 * x3 w4 * x4 Neuronal output identity function y1 net non negative identity function y1 0 if net THRESHOLD ( ) y1 net if net THRESHOLD ( ) 4 Artificial Neural Networks: An Operational View A neuron is an information processing unit forming the key ingredient of a neural network: The diagram above is a model of a biological neuron. There are three key ingredients of this neuron labelled xk which is connected to the (rest of the) neurons in the network labelled x1, x2, x3,…xj. A set of links, the biological equivalent of synapses, which the kth neuron has with the (rest of the) neurons in the network. Note that each link has a WEIGHT denoted by labelled wk1, wk2,…wkj, where the first subscript (k in this case) denotes the recipient neurons and the second subscript (1,2,3…..j) denotes the neurons transmitting to the recipient neurons. The synaptic weight wkj may lie in a range that includes negative (inhibitory) values and positive (excitatory) values. (From Haykin 1999:10-12) 5 Artificial Neural Networks: An Operational View The kth neuron adds up the inputs of all the transmitting neurons at the summing junction or the adder, denoted by S. The adder acts as a linear combiner and generates a weighted average usually denoted by uk: uk = wk1i*x1 + wk2*x2 + wk3*x3 + ………. + wkj*xj; the bias (bk )has the effect of increasing or decreasing the net input to the activation function depending on the value of the bias. (From Haykin 1999:10-12) (From Haykin 1999:10-12) 6 ANN’s: an Operational View Finally, the linear combination, denoted as vk = uk + bk, is passed through the activation function which engenders the non-linear behaviour seen in the behaviour of the biological neurons: the inputs to and outputs from a given neuron show a complex, often non-linear behaviour. For example, if the output from the adder was positive or zero then the neuron will emit a signal, yk = 1 if (vk)0 , however if the output from the adder was negative then there will be no output, yk = 0 if (vk)< 0 . There are other models of the activiation function as we will see later. (From Haykin 1999:10-12) 7 ANN’s: an Operational View Neuron xk x1 wk2 x3 wk3 x4 wk4 Summing Junction Activation Function S yk Output Signal Input Signals x2 wk1 bk Net input or weighted sum : net w1 * x1 w2 * x2 w3 * x3 w4 * x4 Neuronal output cons tan t y1 net y1 0 if net THRESHOLD ( ) y1 C if net THRESHOLD ( ) 8 ANN’s: an Operational View Neuron xk x1 wk2 x3 wk3 x4 wk4 Summing Junction Activation Function S yk Output Signal Input Signals x2 wk1 bk Net input or weighted sum : net w1 * x1 w2 * x2 w3 * x3 w4 * x4 Neuronal output Saturated output y1 0 if net THRESHOLD ( ) y1 1 if net THRESHOLD ( ) 9 ANN’s: an Operational View Discontinuous Output Neuron xk x1 wk2 x3 wk3 x4 wk4 S Threshold (θ) y1 0 if net THRESHOLD ( ) y1 1 if net THRESHOLD ( ) Output f(net) Saturated output yk bk Net input or weighted sum : net w1 * x1 w2 * x2 w3 * x3 w4 * x4 Neuronal output Activation Function Output Signal Input Signals x2 Summing Junction wk1 No output (Normalised) output (eg. 1) net 10 ANN’s: an Operational View Neuron xk x1 wk2 x3 wk3 x4 wk4 Summing Junction S Activation Function yk Output Signal Input Signals x2 wk1 bk The notion of a discontinuous function simulates the fundamental notion that biological neurons usually fire if there is ‘enough’ stimulus available in the environment. But discontinuous is biologically implausible, so there must be some degree of continuity in the output such that an artificial neuron has a degree of biological plausibility. 11 ANN’s: an Operational View Pseudo-Continuous Output Neuron xk x1 wk2 x3 wk3 x4 wk4 Activation Function S yk bk Saturation Threshold (θ’) Net input or weighted sum : f(net) net w1 * x1 w2 * x2 w3 * x3 w4 * x4 Neuronal output Saturated output Output Signal Input Signals x2 Summing Junction wk1 Output β Threshold (θ) Output=α y1 if net THRESHOLD ( ) y1 if net Saturation THRESHOLD ( ' ) y1 f ( , ' , , ) net 12 ANN’s: an Operational View Neuron xk x1 x3 x4 wk2 Summing Junction Activation Function S yk wk3 wk4 bk A schematic for an 'electronic' neuron Output Signal Input Signals x2 wk1 13 ANN’s: an Operational View Neural Nets as directed graphs A directed graph is a geometrical object consisting of a set of points (called nodes) along with a set of directed line segments (called links) between them. A neural network is a parallel distributed information processing structure in the form of a directed graph. 14 ANN’s: an Operational View Input Connections Processing Unit Output Connection Fan Out 15 ANN’s: an Operational View A neural network comprises A set of processing units A state of activation An output function for each unit A pattern of connectivity among units A propagation rule for propagating patterns of activities through the network An activation rule for combining the inputs impinging on a unit with the current state of that unit to produce a new level of activation for the unit A learning rule whereby patterns of connectivity are modified by experience An environment within which the system must operate 16 The McCulloch-Pitts Network . McCulloch and Pitts demonstrated that any logical function can be duplicated by some network of all-ornone neurons referred to as an artificial neural network (ANN). Thus, an artificial neuron can be embedded into a network in such a manner as to fire selectively in response to any given spatial temporal array of firings of other neurons in the ANN. Artificial Neural Networks for Real Neuroscientists: Khurshid Ahmad, Trinity College, 28 Nov 2006 17 The McCulloch-Pitts Network Demonstrates that any logical function can be implemented by some network of neurons. •There are rules governing the excitatory and inhibitory pathways. •All computations are carried out in discrete time intervals. •Each neuron obeys a simple form of a linear threshold law: Neuron fires whenever at least a given (threshold) number of excitatory pathways, and no inhibitory pathways, impinging on it are active from the previous time period. •If a neuron receives a single inhibitory signal from an active neuron, it does not fire. •The connections do not change as a function of experience. Thus the network deals with performance but not learning. 18 The McCulloch-Pitts Network Computations in a McCulloch-Pitts Network ‘Each cell is a finite-state machine and accordingly operates in discrete time instants, which are assumed synchronous among all cells. At each moment, a cell is either firing or quiet, the two possible states of the cell’ – firing state produces a pulse and quiet state has no pulse. (Bose and Liang 1996:21) ‘Each neural network built from McCulloch-Pitts cells is a finite-state machine is equivalent to and can be simulated by some neural network.’ (ibid 1996:23) ‘The importance of the McCulloch-Pitts model is its applicability in the construction of sequential machines to perform logical operations of any degree of complexity. The model focused on logical and macroscopic cognitive operations, not detailed physiological modelling of the electrical activity of the nervous system. In fact, this deterministic model with its discretization of time and summation rules does not reveal the manner in which biological neurons integrate their inputs.’ (ibid 1996:25) 19 The McCulloch-Pitts Network Consider a McCulloch-Pitts network which can act as a minimal model of the sensation of heat from holding a cold object to the skin and then removing it or leaving it on permanently. Each cell has a threshold of TWO, hence fires whenever it receives two excitatory (+) and no inhibitory (-) signals from other cells at a previous time. Artificial Neural Networks for Real Neuroscientists: Khurshid Ahmad, Trinity College, 28 Nov 2006 20 The McCulloch-Pitts Network Heat Sensing Network 1 + + Hot 3 + Heat + B Receptors Cold + + A + 2 + + + 4 Cold 21 The McCulloch-Pitts Network Heat Sensing Network Truth tables of the firing neurons when the cold object contacts the skin and is then removed 1 + + Hot 3 Heat Receptors Cold Cell 1 Cell 2 Cell a Cell b Cell 3 Cell 4 INPUT INPUT HIDDEN HIDDEN OUTPUT OUTPUT 1 No Yes No No No No 2 No No Yes No No No 3 No No No Yes No No 4 No No No No Yes No + + B + + A + + 2 Time + + 4 Cold 22 The McCulloch-Pitts Network Heat Sensing Network ‘Feel hot’/’Feel cold’ neurons show how to create OUTPUT UNIT RESPONSE to given INPUTS that depend ONLY on the previous values. This is known as a TEMPORAL CONTRAST ENHANCEMENT. The absence or presence of a stimulus in the PREVIOUS time cycle plays a major role here. The McCulloch-Pitts Network demonstrates how this ENHANCEMENT can be simulated using an ALL-OR-NONE Network. 23 The McCulloch-Pitts Network Heat Sensing Network Truth tables of the firing neurons for the case when the cold object is left in contact with the skin – a simulation of temporal contrast enhancement Time Cell 1 Cell 2 Cell a Cell b Cell 3 Cell 4 INPUT INPUT HIDDEN HIDDEN OUTPUT OUTPUT 1 2 3 24 The McCulloch-Pitts Network Heat Sensing Network Truth tables of the firing neurons for the case when the cold object is left in contact with the skin – a simulation of temporal contrast enhancement 1 + + Hot Time 3 Heat Receptors Cold Cell 2 Cell a Cell b Cell 3 Cell 4 INPUT INPUT HIDDEN HIDDEN OUTPUT OUTPUT 1 No Yes No No No No 2 No Yes Yes No No No 3 No Yes Yes No No Yes + + B + + A + + 2 Cell 1 + + 4 Cold 25 The McCulloch-Pitts Network Memory Models + + A + + + 1 2 + ++ + + B Three stimulus model 1 + 2 + Permanent Memory model 26 The McCulloch-Pitts Network Memory Models In the permanent memory model, the output neuron has threshold ‘1’; neuron 2 fires if the light has ever been on anytime in the past. Levine, D. S. (1991:16) + 1 + 2 Permanent Memory model 27 The McCulloch-Pitts Network Memory Models Consider, the three stimulus all-or-none neural network. In this network, neuron 1 responds to a light being on. Each of the neurons has threshold ‘3’. In the three stimulus model neuron 2 fires after the light has been on three time units in a row. Time A 1 2 Cell 1 Cell A Cell B Cell 2 1 Yes No No No 2 Yes No Yes No 3 Yes Yes Yes No 4 No Yes Yes B Three stimulus model All connections are unit positive Yes 28 The McCulloch-Pitts Network Why is a McCulloch-Pitts a FSM? A finite state machine (FSM)is an AUTOMATON.An input string is read from left to right; the machine looks at each symbol in turn. At any time the FSM is in one of many finitely interval states. The state changes after each input symbol is read. The NEW STATE depends (only) on the symbol just read and on the current state. 29 The McCulloch-Pitts Network ‘The McCulloch-Pitts model, though it uses an oversimplified formulation of neural activity patterns, presages some issues that are still important in current cognitive models. [..][Some] Modern connectionist networks contain three types of units or nodes – input units, output units, and hidden units. The input units react to particular data features from the environment […]. The output units generate particular organismic responses […]. The hidden units are neither input nor output units themselves but, via network connections, influence output units to respond to prescribed patterns of input unit firings or activities. [..] [This] input-output-hidden trilogy can be seen as analogous to the distinction between sensory neurons, motor neurons, and all other (interneurons) in the brain’ Levine, Daniel S. (1991: 14-15) Artificial Neural Networks for Real Neuroscientists: Khurshid Ahmad, Trinity College, 28 Nov 2006 30 The McCulloch-Pitts Network Linear Neuron: Output is the weighted sum of all the inputs; McCulloch-Pitts Neuron: Output is the thresholded value of the weighted sum Input Vector? X = X (1,-20,4,-2); Weight vector? wji=w(wj1,wj2,wj3,wj4) wj1 =[0.8,0.2,-1,-0.9] x1 x2 wj2 S w yj j3 x3 j wj4 x4 0th input 31 The McCulloch-Pitts Network vj=Swjixi; y=f(v); y=0 if v<=0 or y=1 if v>0 Input Vector? X = X (1,-20,4,-2); Weight vector? wji=w(wj1,wj2,wj3,wj4) =[0.8,0.2,-1,-0.9] wj0=0, x0=0 x1 x2 x3 x4 wj1 wj2 wj3 wj4 S j 0th input yj 32 The McCulloch-Pitts Network Input Vector? X = X (1,-20,4,-2); Weight vector? w=w(wj1,wj2,wj3,wj4) =[0.8,0.2,-1,-0.9] wj0=0, x0=0 vj=Swjixi; y=f(v); f activation function Linear Neuron: y=v McCulloch Pitts: y=0 if v<=0 or y=1 if v>0 Sigmoid activation function: f(v)=1 /(1+exp(-v)) 33 The McCulloch-Pitts Network What are the circumstance in a neuron with a sigmoidal activation function will act like a McCulloch Pitts network? Large synaptic weights What are the circumstance in a neuron with a sigmoidal activation function will act like a linear neuron? Small synaptic weights 34 The McCulloch-Pitts Network The key outcome of early research in artificial neural networks clearly demonstrated the theoretical importance (brain like behaviour and logical basis) and extensive utility (regime switching modes) of threshold behaviour. This behaviour was emulated through the use of the squashing functions and is the basis of many a simulation. 35