Download Chapter 1

Introduction to Neural
Networks
John Paxton
Montana State University
Summer 2003
Textbook
Fundamentals of Neural Networks:
Architectures, Algorithms, and Applications
Laurene Fausett
Prentice-Hall
1994
Chapter 1: Introduction
• Why Neural Networks?
Training techniques exist.
High speed digital computers.
Specialized hardware.
Better capture biological neural systems.
Who is interested?
• Electrical Engineers – signal processing,
control theory
• Computer Engineers – robotics
• Computer Scientists – artificial
intelligence, pattern recognition
• Mathematicians – modelling tool when
explicit relationships are unknown
Characterizations
• Architecture – a pattern of connections
between neurons
• Learning Algorithm – a method of
determining the connection weights
• Activation Function
Problem Domains
•
•
•
•
•
Storing and recalling patterns
Classifying patterns
Mapping inputs onto outputs
Grouping similar patterns
Finding solutions to constrained
optimization problems
A Simple Neural Network
x1
w1
y
x2
w2
yin = x1w1 + x2w2
Activation is f(yin)
Biological Neuron
• Dendrites receive electrical signals
affected by chemical process
• Soma fires at differing frequencies
dendrite
soma
axon
Observations
• A neuron can receive many inputs
• Inputs may be modified by weights at the
receiving dendrites
• A neuron sums its weighted inputs
• A neuron can transmit an output signal
• The output can go to many other neurons
Features
• Information processing is local
• Memory is distributed (short term =
signals, long term = dendrite weights)
• The dendrite weights learn through
experience
• The weights may be inhibatory or
excitatory
Features
• Neurons can generalize novel input stimuli
• Neurons are fault tolerant and can sustain
damage
Applications
• Signal processing, e.g. suppress noise on
a phone line.
• Control, e.g. backing up a truck with a
trailer.
• Pattern recognition, e.g. handwritten
characters or face sex identification.
• Diagnosis, e.g. aryhthmia classification or
mapping symptoms to a medical case.
Applications
• Speech production, e.g. NET Talk.
Sejnowski and Rosenberg 1986.
• Speech recognition.
• Business, e.g. mortgage underwriting.
Collins et. Al. 1988.
• Unsupervised, e.g. TD-Gammon.
Single Layer Feedforward NN
x1
w11
y1
w1m
wn1
xn
ym
wnm
Multilayer Neural Network
• More powerful
• Harder to train
x1
xn
z1
y1
zp
ym
Setting the Weight
• Supervised
• Unsupervised
• Fixed weight nets
Activation Functions
• Identity
f(x) = x
• Binary step
f(x) = 1 if x >= q
f(x) = 0 otherwise
• Binary sigmoid
f(x) = 1 / (1 + e-sx)
Activation Functions
• Bipolar sigmoid
f(x) = -1 + 2 / (1 + -sx)
• Hyperbolic tangent
f(x) = (ex – e-x) / (ex + e-x)
History
•
•
•
•
1943 McCulloch-Pitts neurons
1949 Hebb’s law
1958 Perceptron (Rosenblatt)
1960 Adaline, better learning rule (Widrow,
Huff)
• 1969 Limitations (Minsky, Papert)
• 1972 Kohonen nets, associative memory
History
•
•
•
•
1977 Brain State in a Box (Anderson)
1982 Hopfield net, constraint satisfaction
1985 ART (Carpenter, Grossfield)
1986 Backpropagation (Rumelhart, Hinton,
McClelland)
• 1988 Neocognitron, character recognition
(Fukushima)
McCulloch-Pitts Neuron
x1
f(yin) = 1 if yin >= q
x2
x3
y
Exercises
•
•
•
•
2 input AND
2 input OR
3 input OR
2 input XOR