Download Artificial Intelligence Connectionist Models Inspired by the brain

Connectionist Models
In contrast to symbolic models
● Based on the brain paradigm or brain
metaphor
● Have enjoyed much success in recent years
● Synonyms
●
Artificial Intelligence
Introduction to Neural Networks
–
–
–
–
CS 4633/6633 Artificial Intelligence
CS 4633/6633 Artificial Intelligence
Inspired by the brain
●
●
●
●
●
●
●
●
●
neural computing
neural networks
connectionism
parallel distributed processing
Large number of simple processing units (neurons)
Highly connected
Highly parallel, distributed control
Neurons are slow devices compared to digital computers
Can perform complex tasks in a short period of time
Graceful degradation when neurons fail
Handles fuzzy situations very well.
Information accessed on the basis of content
Learns from experience
Brief History
●
●
●
●
●
●
●
●
CS 4633/6633 Artificial Intelligence
1943: Neurophysiologist Warren McCullock and logician Walter
Pitts propose MuCulloch-Pitts model of a neuron
1949: Neurophysiologist Donald Hebb proposes model of dynamic
memory and simple learning rule
1960’s: Frank Rossenblatt advocates perceptron model and proves
convergence of learning algorithm for it
1969: Marvin Minsky and Seymour Papert prove that the perceptron
has severe computational limits. Many abandon field.
1982: Nobel prize winner John Hopfield uses concepts from physics
to analyze neural networks and applies them to new problems
1985: Backpropagation training rule for muli-layer neural networks
is rediscovered
1987: First IEEE conference on neural networks. Over 2000 attend.
The revival is underway!
CS 4633/6633 Artificial Intelligence
A Simple “Neuron”
Receives information from other neurons, performs simple processing,
and sends result to one or more neurons
a1
W1
a2
W2
a3
W3
.
.
an
Activations
●
Input = ΣW jaj
Output = g(input)
Σ
g
Input
Activation
Function
Function
Wn
CS 4633/6633 Artificial Intelligence
●
The activation of a neuron is a numerical value
that represents its “state of excitement”
– May be discrete {-1, 0, 1} or {0 , 1}
– May be continuous [0,1]
Determined by a cell’s activation function
1, if x ≥ t
stept ( x) = 
î 0, if x < t
sigmoid ( x) =
1
1 + e− x
CS 4633/6633 Artificial Intelligence
Threshold (bias)
The minimum total weighted input necessary to cause a
neuron to fire. The t in step function is a threshold. Often
implemented as additional weight, as follows.
1
I1
w1,j
w0,j
n
in j = ∑ w i , j ∗ input i
w2,j
I2
i=0
uj
I3
Neural Network
Neurons are organized into networks.
● Connections are one or two-way
communication links between neurons
● Weights are the strength of connections. A
weight wij is a real number than indicates
the influence neuron ui has on neuron uj
●
w3,j
CS 4633/6633 Artificial Intelligence
CS 4633/6633 Artificial Intelligence
Neural Network: Simple Example
I1
Network Structures
Feed-forward networks
– no cycles
– compute a function from input to output
– two kinds:
●
H1
O1
I2
» perceptron
» multi-layer
H2
Input Layer
Recurrent networks
– cycles are allowed--activation is fed back to the
unit that caused it
– more complex and difficult to train
●
I3
Hidden Layer
Output Layer
CS 4633/6633 Artificial Intelligence
CS 4633/6633 Artificial Intelligence
Representational Power
A feed-forward network computes a
function from the inputs to the outputs
● A feed-forward network with one hidden
layer can approximate any continuous
function of the inputs
● A feed-forward network with two hidden
layers can approximate any function at all
● However, the hidden layers may need lots
of neurons
Learning
●
CS 4633/6633 Artificial Intelligence
A neural net can be trained to represent
any function by adjusting its weights
● A feed-forward network is usually trained by
supervised learning
● We will discuss learning algorithms for
neural networks in the next two classes
●
CS 4633/6633 Artificial Intelligence