Download Lecture 6 - School of Computing | University of Leeds

Bioinspired Computing
Lecture 6
Artificial Neural Networks:
The rise & fall of the perceptron
Netta Cohen
Last time... biological neural networks
We introduced biological neural networks. We found
complexity at every level, from the sub-cellular to the entire
brain. We realised that even with a limited understanding,
cartoon models can be derived for some functions of
neurons (action potentials, synaptic transmission, neuronal
computation and coding). Despite (or perhaps because of)
their simplicity, these cartoon models are priceless.
This time... Artificial neural networks (part 1)
Forget the complexity. Focus on cartoon models of
biological nnets & further simplify them. Build on biology to
design simple artificial networks that perform classification
tasks. Today, we start with a single artificial neuron and
study its computational power.
2
Learning
No discussion of the brain, or nervous systems more generally is
complete without mention of learning.
•
•
•
•
What is learning?
How does a neural network ‘know’ what computation to perform?
How does it know when it gets an ‘answer’ right (or wrong)?
What actually changes as a neural network undergoes ‘learning’?
body
Sensory inputs
brain
Motor outputs
environment
3
Learning (cont.)
Learning can take many forms:
• Supervised learning
• Reinforcement learning
• Association
• Conditioning
• Evolution
At the level of neural networks, the best understood forms of
learning occur in the synapses, i.e., the strengthening and
weakening of connections between neurons. The brain uses its
own learning algorithms to define how connections should
change in a network.
4
Learning from experience
How do the neural networks form in the brain? Once
formed, what determines how the circuit might change?
In 1948, Donald Hebb, in his book, "The Organization of
Behavior", showed how basic psychological phenomena of
attention, perception & memory might emerge in the brain.
Hebb regarded neural networks as a collection of cells that
can collectively store memories. Our memories reflect our
experience.
How does experience affect neurons and neural networks?
How do neural networks learn?
5
Synaptic Plasticity
Definition of Learning: experience alters behaviour
The basic experience in neurons is spikes.
Spikes are transmitted between neurons through synapses.
Hebb suggested that connections in the brain change in
response to experience.
delay
Pre-synaptic cell
Post-synaptic cell
time
Hebbian learning: If the pre-synaptic cell causes the
post-synaptic cell to fire a spike, then the connection
between them will be enhanced. Eventually, this will
lead to a path of ‘least resistance’ in the network.
6
Today... Artificial neural networks (part 1)
Focus on the simplest cartoon models of biological neural
nets. We will build on lessons from today to design simple
artificial neurons and networks that perform useful
computational tasks.
7
The Appeal of Neural Computing
The only intelligent systems that we know of are biological. In
particular most brains share the following feature in their
neural architecture – they are massively parallel networks
organised into interconnected hierarchies of complex
structures.
For computer scientists, many natural systems appear
to share many attractive properties:
• speed, tolerance, robustness, flexibility, self-driven
dynamic activity
In addition, they are very good at some tasks that
computers are typically poor at:
• recognising patterns, balancing conflicts, sensorymotor coordination, interaction with the
environment, anticipation, learning… even curiosity,
creativity & consciousness.
8
The first artificial neuron model
In analogy to a biological neuron, we can think of a virtual
neuron that crudely mimics the biological neuron and performs
analogous computation.
inputs
Σ
output
cell
body
Just like biological neurons, this artificial neuron neuron will have:
• Inputs (like biological dendrites) carry signal to cell body.
• A body (like the soma), sums over inputs to compute output, and
• outputs (like synapses on the axon) transmit the output downstream.
The artificial neuron is a cartoon model that will not have all
the biological complexity of real neurons. How powerful is it?
9
Early history (1943)
McCulloch & Pitts (1943). “A logical calculus of the ideas immanent in
nervous activity”, Bulletin of Mathematical Biophysics, 5, 115-137.
In this seminal paper, Warren McCulloch and Walter Pitts
invented the first artificial (MP) neuron, based on the insight
that a nerve cell will fire an impulse only if its threshold
value is exceeded. MP neurons are hard-wired devices,
reading pre-defined input-output associations to determine
their final output. Despite their simplicity, M&P proved that a
single MP neuron can perform universal logic operations.
A network of such neurons can therefore do anything a
Turing machine can do, but with a much more flexible (and
potentially very parallel) architecture.
10
The McCulloch-Pitts (MP) neuron
• Inputs x are binary: 0,1.
• Each input has an assigned weight w.
• Weighted inputs are summed  in the cell body.
• Neuron fires if sum exceeds (or equals) activation threshold .
• If the neuron fires, the output =1.
The “computation”
• Otherwise, the output=0.
consists of "adders" and a
x1 *
w1
threshold.
x2 *
••
•
x

(
inputs
n

w3

••
•
output
* w
n
*
weights
inputs
x3 *
w2
*
wb
)
over all i
=
bias
1 if   
0 if  < 
Note: an equivalent
formalism assigns =0
& instead of threshold
introduces an extra
bias input, such that
bias * wbias = - 
11
Logic gates with MP neurons
For binary logic gates, with only one input, possible outputs are described
by the following truth tables:
Always 0
IN 1 OUT 1
0
1
IDENTITY
IN 1 OUT 2
0
0
For example:
0
1
x
0
1
NOT
IN 1 OUT 3
0
1
1
0
Always 1
IN 1 OUT 4
0
1
1
1

NOT x
w
w = -1
 = -0.5
Excercise: Find w and  for the 3 remaining gates.
12
Logic gates with MP neurons (cont.)
With two binary inputs, there are 4 possible inputs
and 24 = 16 corresponding truth tables (outputs)!
For example, the AND gate implemented in the MP neuron:
Here is a compact,
graphical representation
of the same truth table:
x1

1
x2
x1 AND x2
1
 = +1.5
IN 2
IN 1 IN 2 OUT
0
0
0
0
1
0
1
0
0
1
1
1
0
1
IN 1
0
1
0
0
0
1
Excercise:
Find w and 
for OR & NAND.
13
Computational power of MP neurons
Universality: NOT & AND can be combined to perform any logical
function; MP neurons, circuited together in a network can solve
any problem that a conventional computer could.
But let’s examine the single neuron a little longer.
Q: Just how powerful is a single MP neuron?
A: It can solve any problem that can be expressed as a
classification of points on a plane by a single straight line.
IN 2
IN 1
AND
0
1
0
0
0
1
0
1
Generalisation to many inputs:
points in many dimensions are
now classified, not by a line,
but by a flat surface.
Even one neuron can successfully handle simple
classification problem.
14
Classification in Action
A set of patients may have a medical problem. Blood
samples are analysed for the quantities of two trace
elements.
x1
w1
x2
w2
inputs
*
weights
bias w3
trace 1 trace 2 problem?
∑xi wi
sum output
2.4
9.8
1.2
0.4
7.9
6.7
etc.
1.0
8.3
0.2
2.1
8.8
7.2
etc.
yes
no
yes
yes
no
no
etc.
output
∑ xi wi
∑ xi wi
∑ xi wi
∑ xi wi
∑ xi wi
∑ xi wi
etc.
+6.6
Yes
-8.1
No
+8.6
Yes
+7.5
Yes
-6.7
No
-3.9
No
etc.
etc.
+ive output = problem w1=-1, w2=-1, w3=+10 & bias=+1
With correct weights, this MP neuron consistently
classifies patients.
15
The missing step
The ability of the neuron to classify inputs correctly hinges
on the appropriate assignment of the weights and
threshold.
So far, we have done this by hand.
Imagine we had an automatic algorithm for the neuron to
learn the right weights and threshold on its own.
In 1962, Rosenblatt, inspired by biological learning rules,
did just that.
16
Frank Rosenblatt (1962). Principles of Neurodynamics, Spartan, New York
 Learning Rule:
Imagine a naive, randomly weighted neuron. One way to train a
neuron to discriminate the sick from the healthy, is by reinforcing
good behaviour and penalising bad. This carrot & stick model is
the basis for the  learning rule:
• Compile a training set of N (say 100) sick and healthy patients.
•
Initialise the neuronal weights (random initialisation is the standard).
Run each input set in turn through the neuron & note its output.
Whenever a wrong output is encountered, alter responsible weights.
•
wi  wi + xi if output too low
wi  wi  xi
if output too high
Repeatedly run through training set until all outputs agree with targets.
•
•
• When training is complete, test the neuron on a new testing set of patients.
• If neuron succeeds, patients whose health is unknown may be determined.
17
Supervised learning
The  learning rule is an example of supervised learning.
Training MP neurons requires a training set, for which the
‘correct’ output is known.
These ‘correct’ or ‘desired’ outputs are used to calculate
the error, which in turn is used to adjust the input-output
relation of the neuron.
Without knowledge of the desired output, the neuron
cannot be trained. Therefore, supervised learning is a
powerful tool when training sets with desired outputs are
available.
When can’t supervised learning be used?
Are biological neurons supervised?
18
A simple example
Let’s try to train a neuron to learn the logical OR operation:
x1
w1
x2
w2 ∑ x w
i i
w3
x3
0
output
w2
w3
0
0
0
0
0
0
OK
WRONG
bias
desired output
0
1
1
OK
x1 OR x2
0
1
1
OK
0
0
1
1
1
0
0
1
0
1
1
1
1
1
1
1
1
0
1
1
0
1
1
0
x1
x2
x3
0
0
1
0
0
1
1
1
1
0
1
1
1
1
1
1
0
w1
w0i  0wi + xi if output low
0 OK
w1i  0wi ∑ xxii wi if output
high
WRONG
OK
WRONG
OK
WRONG
OK
OK
OK
w1= 1, w2= 1, w3= 0
19
The power of learning rules
The  rule is guaranteed to converge on a set of
appropriate weights, if a solution exists. While it might
not be the most efficient of algorithms, this proven
convergence is crucial.
What can be done to improve the convergence rate?
Some common variations on this learning rule:
Adding a learning rate 0<r<1 which “damps” weight
changes (i = rxi or i = -rxi).
Widrow & Hoff recognised that weight changes should be
large when actual output a and target output t were very
different, but smaller otherwise.
They introduced an error term, ∆=t-a, such that i =r∆xi.
20
The Fall of the Artificial Neuron
•
•
Before long researchers had begun to discover the neuron’s limitations.
Unless input categories were “linearly separable”, a perceptron could not
learn to discriminate between them.
• Unfortunately, it appeared that many important categories were not
linearly separable. This proved a fatal blow to the artificial neural
networks community.
In this example, an MP
Successful
neuron would not be
able to discriminate
Few
Many
between the footballers
Hours
Hours
and the academics…
in the
in the
This failure caused the
Gym
Gym
majority of researchers
per
per
to walk away.
Week
Week
Unsuccessful
Footballers
Academics
Exercise: Which logic
operation is described in
this example?
21
Marvin Minsky & Seymour Papert (1969). Perceptrons, MIT Press, Cambridge.
Connectionism Reborn
The crisis in artificial neural networks can be understood, not as
an inability to connect many neurons in a network, but an inability
to generalise the training algorithms to arbitrary architectures. By
arranging the neurons in an ‘appropriate’ architecture, a suitable
training algorithm could be invented. The solution, once found,
quickly emerged as the most popular learning algorithm for nnets.
Back-propagation first discovered in 1974 (Werbos, PhD thesis,
Harvard) but discovery went unnoticed. In the mid-80s, it was
rediscovered independently by three groups within about one year.
Most influential of these was a two-volume book by Rumelhart &
McClelland, who suggested a feed-forward architecture of
neurons: layers of neurons, with each layer feeding its
calculations on to the next.
David E. Rumelhart & James L. McClelland (1986).
Parallel Distributed Processing, Vols. 1 & 2, MIT Press, Cambridge, MA.
22
This time…
•
•
•
•
•
The appeal of neural computing
From biological to artificial neurons
Nervous systems as logic circuits
Classification with the McCulloch & Pitts neuron
Developments in the 60s:
– The Delta learning rule & variations
– Simple applications
– The fatal flaw of linearity
Next time…
The disappointment with the single neuron dissipated as promptly as it
dawned upon the AI community. Next time, we will see why the single
neuron’s simplicity does not rule out immense richness at the network
level. We will examine the simplest architecture of feed-forward neural
networks and generalise the delta-learning rule to these multi-layer
networks. We will also re-discover some impressive applications.
23
Optional reading
Excellent treatments of the perceptron, the delta rule & Hebbian
learning, the multi-layer perceptron and the back-propagation
learning algorithm can be found in:
Beale & Jackson (1990). Neural Computing, chaps. 3 & 4.
Hinton (1992). How neural networks learn from experience,
Scientific American, 267 (Sep):104-109.
24