Download ppt - IIT Bombay

CS621 : Artificial Intelligence
Pushpak Bhattacharyya
CSE Dept.,
IIT Bombay
Lecture 21
Computing power of Perceptrons and
Perceptron Training
The human brain
Seat of consciousness and cognition
Perhaps the most complex information processing
machine in nature
Brain Map
Forebrain (Cerebral Cortex):
Language, maths, sensation,
movement, cognition, emotion
Midbrain: Information Routing;
involuntary controls
Cerebellum: Motor
Control
Hindbrain: Control of
breathing, heartbeat, blood
circulation
Spinal cord: Reflexes,
information highways between
body & brain
• Maslow’s Hierarchy of Needs
Brain’s algorithms?
• Evolutionarily, brain has developed algorithms
most suitable for survival
• Algorithms unknown: the search is on
• Brain astonishing in the amount of information it
processes
– Typical computers: 109 operations/sec
– Housefly brain: 1011 operations/sec
Brain facts & figures
• Basic building block of nervous system: nerve
cell (neuron)
• ~ 1012 neurons in brain
• ~ 1015 connections between them
• Connections made at “synapses”
• The speed: events on millisecond scale in
neurons, nanosecond scale in silicon chips
Computing Power of
Perceptron
The Perceptron Model
A perceptron is a computing element with input
lines having associated weights and the cell
having a threshold value. The perceptron model is
motivated by the biological neuron.
Output = y
Threshold = θ
wn
w1
Wn-1
Xn-1
x1
y
1
θ
Σwixi
Step function / Threshold function
y
= 1 for Σwixi >=θ
=0 otherwise
Concept of Hyper-planes
• ∑ wixi = θ defines a linear surface in the
(W,θ) space, where W=<w1,w2,w3,…,wn>
is an n-dimensional vector.
y
• A point in this (w,θ) space
θ
defines a perceptron.
w1
x1
w2 w3 . . .
x2
x3
wn
xn
Functions computed by the
simplest perceptron (single input)
True-Function
x
0
1
0-function
θ≥0
w≤θ
f1 f2 f3 f4
0 0 1 1
0 1 0 1
Identity Function
θ≥0
w> θ
θ<0
W< θ
Complement Function
θ<0
w≤ θ
Counting the number of functions
for the simplest perceptron
• For the simplest perceptron, the equation
is w.x=θ.
θ
Substituting x=0 and x=1,
we get θ=0 and w=θ.
w=θ
R4
w
R1
These two lines intersect to
R3
θ=0
R2
form four regions, which
correspond to the four functions.
Fundamental Observation
• The number of TFs computable by a perceptron
is equal to the number of regions produced by 2n
hyper-planes,obtained by plugging in the values
<x1,x2,x3,…,xn> in the equation
∑i=1nwixi= θ
The geometrical observation
• Problem: m linear surfaces called hyperplanes (each hyper-plane is of (d-1)-dim)
in d-dim, then what is the max. no. of
regions produced by their intersection?
i.e. Rm,d = ?
Case of 2-input perceptrons
Output = y
Threshold = θ
w2
w1
x2
x1
Basic equation
• w1x1+w2x2=θ
• There are 4 values of the input: (0,0),
(0,1), (1,0), (1,1)
• The relevant space is the (w1,w2,θ)
coordinate system
θ
w1
w2
4 planes
• All go through the origin
– With (0,0), θ= 0 -- (1)
– With (1,0), w1 = θ --(2)
– With (0,1), w2 = θ --(3)
– With (1,1), w1 + w2 = θ --(4)
• How many regions do they produce?
• That is equal to the number of functions
computable by a 2-input perceptron
How to think about this counting
problem?
• Whenever a new plane comes in, the existing
planes intersect the new plane is a set of lines
all going through the origin
• These lines produce some regions on the new
plane (notice that we are not worrying about the
regions in the space, but are thinking about the
regions on the new plane)
• These regions produced on the new plane is
the additional number of regions produced in
the (w,θ) space
Counting the maximum no. of regions
produced by 4 planes passing through origin
Plane number
Additional Regions produced
1st
2
2nd
(1st plane cuts this plane to form a
line through the origin)
2
3rd
(1st and 2nd planes cut this plane to
form two lines through the origin)
4
4th
(1st, 2nd and 3rd planes cut this
plane to form thre lines through
the origin)
6
It is clear why 2-input perceptron
computes 14 functions
•
•
•
•
•
2+2+4+6= 14
14 regions are produced by the planes.
Hence only 14 functions computed
Terminology:
Functions computable by a perceptron are
called threshold functions
General case
• A perceptron with n weights and the
threshold defines an (n+1)-dimensional
space.
• From the basic equation Σ1 nwixi=θ, 2n
planes passing through origin are
produced
• How many regions do they produce in the
space?
Recurrence relation produced by m
hyperplanes in d-dimension
– C(m,d)= C(m-1,d)+C(m-1,d-1)
Existing regions
Additional regions on
the dth plance
• Boundary conditions:
– C(m,1)=2 (degenerate case of m points
‘passing through’ origin)
– C(1,d)=2
Threshold functions miniscule
compared to Boolean Function
• Solution of the recurrence relation leads to
the max. no. of regions as
2^n2
• vs.
2^2n boolean functions