Download PSY105 Neural Networks 2/5

PSY105 Neural Networks 2/5
2. “A universe of numbers”
Lecture 1 recap
• We can describe patterns at one level of
description that emerge due to rules followed
at a lower level of description.
• Neural network modellers hope that we can
understand behaviour by creating models of
networks of artificial neurons.
Warren McCullock
1943 - First artificial
neuron model
Warren McCulloch
(neurophysiologist) Walter
Pitts (mathematician)
A simple artificial neuron
Threshold logic unit (TLU)
input
weight
activation
Add
Threshold
Multiply inputs by weights and add. If the sum
is larger than a threshold output 1, otherwise
output 0
TLU: the output relation
output
1
activation
0
threshold
The relation is non-linear – small changes in activation give different
changes in the output depending on the initial activation
Model neuron function, reminders…
• Inputs vary, they can be 0 or 1
– Weights change, effectively ‘interpreting’ inputs
• There is a weight for each input
– This can be a +ve number (excitation) or a –ve number
(inhibition)
– Weights do not change when inputs change
• Activation = weighted sum of inputs
– Activation = input1 x weight1 + input2xweight2 etc
• If activation>threshold, output = 1, otherwise
output=0
– Threshold = 1
States, weights & functions
• States: all the possible combinations of inputs
• Weights: how each input is multiplied before
contributing to the activation of the unit
• Functions: a way inputs are combined to
produce outputs
Computing with neurons: identify (1)
input
output
weight
X
Input
State 1
State 2
• 0
• 1
Weight
• 0.7
• 0.7
Act.
Activation
• 0
• 0.7
Threshold = 1
?
Output
• 0
• 0
Computing with neurons: identity (2)
input
output
weight
Act.
Input
State 1
State 2
• 0
• 1
Weight
• 1
• 1
Activation
• 0
• 1
Threshold = 1
?
Output
• 0
• 1
Question: How could you use these
simple neurons (TLUs) to compute the
AND function?
Input 1
•0
•0
•1
•1
Input 2
•0
•1
•0
•1
Output
•0
•1
•1
•1
Computing with neurons: AND
inputs
output
weights
Act.
Input 1
State 1
State 2
State 3
State 4
•0
•0
•1
•1
Threshold = 1,
Input 2
•0
•1
•0
•1
Activation
•0
• 0.5
• 0.5
•1
Weight 1 = 0.5,
?
Output
•0
•0
•0
•1
Weight 2 = 0.5
Networks of such neurons are Turing
complete
1912 - 1954
Semilinear node
input
weight
activation
Add
Squashing
function
Semilinear node: the output relation
(squashing function)
output
1
activation
0
threshold