Download neural_networks

NEURON
Dendrite
Axon Terminal
Soma
Node
10mm
Axon
Nucleus
Schwann Cell
Myelin
Soma: central part of neuron. Contains nucleus protein synthesis
Dendrites: input, some output
Axon: carries signal away from soma
Axon terminal: releases neuro-chemical to communicate with other
neurons
Excitatory neurons: excite their target neurons. Excitatory neurons
in the brain are often glutamatergic.
Inhibitory neurons: inhibit their target neurons. Inhibitory neurons
are often interneurons. The output of some brain structures
(neostriatum, globus pallidus, cerebellum) are inhibitory. The
primary inhibitory neurotransmitters are GABA and glycine.
Modulatory neurons evoke more complex effects termed
neuromodulation. These neurons use such neurotransmitters as
dopamine, acetylcholine, and serotonin.
Image of pyramidal neurons in mouse cerebral cortex expressing greeen
fluorescent protein. The red staining indicates GABAergic interneurons.
(Chemical) SYNAPSE: Electrical signal causes releases of chemicals
that flow across
synapse and
generate
electrical signal
in next neuron.
20 nm
Complexity of human brain and enormous number of possible states:
Synapses in brain: young children 1016 (10,000,000,000,000,000)
Stabilizes in adulthood by ½ to < 5 x1015
TERMINOLOGY
Synapse is asymmetric:
PRE-SYNAPTIC neuron secretes neurotransmitter chemicals
POST-SYNAPTIC neuron binds neurotransmitters at receptors
Pre-synaptic neuron
Release of neurotransmitter triggered by the arrival of a nerve impulse
Action potential
Cellular secretion, exocytosis, is very rapid
Pre-synaptic nerve terminal has docked vesicles docked at
Membrane containing neurotransmitter
Arriving action potential produces influx of calcium ions through
voltage-dependent, calcium-selective ion channels.
Calcium ions trigger biochemical cascade: vesicles fuse with presynaptic
membrane and release their contents to the synaptic cleft.
Post-Synaptic neuron
Receptors on opposite side of synaptic gap bind neurotransmitter
molecules
Respond by opening nearby ion channels in the post-synaptic cell
Ions rush in or out and change local transmembrane potential
Resulting change in voltage is called postsynaptic potential
Result is excitatory, in the case of depolarizing currents
inhibitory in the case of hyperpolarizing currents
Whether a synapse is excitatory or inhibitory depends on what type(s) of
ion channel conduct the post-synaptic current display(s), which in turn is
a function of the type of receptors and neurotransmitter employed at the
synapse.
Following fusion of the synaptic vesicles and release of transmitter
molecules into the synaptic cleft, the neurotransmitter is rapidly cleared
from the space for recycling by specialized membrane proteins in the
pre-synaptic or post-synaptic membrane.
This “re-uptake" prevents “desensitization" of post-synaptic receptors
Re-uptake ensures succeeding action potentials will elicit the same size
post-synaptic potential (PSP)
Necessity of re-uptake and the phenomenon of desensitization in
receptors and ion channels means that the strength of a synapse may
diminish as a train of action potentials arrive in rapid succession
Frequency dependence synapses
Sounds bad but nervous system exploits this property for computational
purposes, and tunes synapses through
Protein phosphorylation; size, number and replenishment rate of vesicles
Example: serotonin re-uptake inhibitors (SSRIs) inhibit serotonin reuptake
Neurotransmitters
Categorized into three major groups:
(1) amino acids (primarily glutamic acid, GABA, aspartic acid &
glycine)
(2) peptides (vasopressin, somatostatin, neurotensin, etc.)
(3) monoamines (norepinephrine NA, dopamine DA & serotonin 5-HT)
plus acetylcholine (ACh).
The major "workhorse" neurotransmitters of the brain are glutamic acid
(=glutamate) and GABA.
Neurotransmitters can be broadly classified into small-molecule
transmitters and neuroactive peptides.
Around 10 small-molecule neurotransmitters are known: acetylcholine, 5
amines, and 3 or 4 amino acids (depending on exact definition used),
Purines, (Adenosine, ATP, GTP and their derivatives) are
neurotransmitters.
Over 50 neuroactive peptides have been found, among them hormones
such as
LH or insulin that have specific local actions in addition to their longrange signalling properties.
Single ions, such as synaptically-released zinc, are also considered
neurotransmitters by some.
It is important to appreciate that it is the receptor that dictates the
neurotransmitter's effect.
Some examples of neurotransmitter action:
•Acetylcholine - voluntary movement of the muscles
•Norepinephrine - wakefulness or arousal
•Dopamine - voluntary movement and emotional arousal
•Serotonin - memory, emotions, wakefulness, sleep and
temperature regulation
•GABA (gamma aminobutyric acid) - motor behaviour
•Glycine - spinal reflexes and motor behaviour
•Neuromodulators - sensory transmission-especially pain
Serotonin
Acetylcholine
Dopamine
Neuronal Patterns
1.
How a neuron can act as a decision-making device;
2.
In what sense does a neuron recognizes a pattern;
3.
4.
How do input weights and threshold interact to determine the set of patterns recognized by
a neuron;
Limitations of the linear discrimination performed by neurons.
Model of a neuron: McCulloch-Pitts neuron
Not biologically neurons per se, but embody fundamental properties of biologically based
information processing (e.g., the weighted summation of excitatory inputs is converted to a
binary signal).
Able to recognize patterns in its environment based on the features of a given stimulus
Cognitive processes that are modeled using these neurons are pattern recognition and decisionmaking.
 Understanding of how the brain can produce cognition.
Use a geometric space (Euclidean distances) to quantify features and patterns
A feature (e.g., a person’s weight, height, hair length, bank account, or grade point average) is a
quantitative aspect of the total pattern we associate with that individual.
A feature can be an axis in a higher dimensional geometric space.
Qualitative features can also be valued, as zero or one, for the absence or presence of a feature
(e.g., ‘has a driver’s license’)
We will only use non-negative values to quantify a feature that corresponds to simple
interpretations of a neuron’s output, i.e., spike or no spike, spike frequency, or inverse interpulse
interval.
Ordered set of such features is called a n-dimensional vector as a point in n-dimensional space.
Re: vector is a point in a geometric space and an ordered set of scalar values
Because these values are ordered, a vector can be called a pattern
Any pattern can be expressed as a vector
Terminology
what goes into a network: input, input pattern, input vector
neural network : an organized system of neurons
Operate on patterns in very high-dimensional spaces—e.g., many neurons in the neocortex
receive 5,000-20,000 excitatory inputs and thus process patterns in 5,000-20,000 dimensions.
Each neuron is part of a network that processes information in an even higher dimensional space.
For organized, functionally delimited brain regions, such as human forebrain
the dimension ranges from 500,000 to 5,000,000 (high dimensional vector)
but
Similarity still based upon separation distance
Individual Neurons Make Decisions.
Synapses made between individual presynaptic inputs and postsynaptic neuron transform that
input.
Neuron’s threshold for action potential generation plays important role in determining a neuron’s
decision.
Assume neuron is dedicated to recognizing a family of closely related similar) patterns
Each active input to a postsynaptic neuron contributes to the excitation of that postsynaptic
neuron in proportion to the synaptic strength or weight, which connects that presynaptic input to
the postsynaptic cell.
Triggering of a postsynaptic neuron: neuronal decision-making is a voting system
Each presynaptic input neuron casts its vote for the firing of the postsynaptic neuron, either by
firing its own action potential or by not firing.
The postsynaptic neuron tallies the votes over the set of inputs—a ‘no’ vote is a zero and a ‘yes’
vote has some positive value (= 1 in simplest model).
If tally is large enough, postsynaptic neuron declares ‘yes’ and it fires
If the tally is not large enough, the postsynaptic neuron says ‘no’ and does not fire.
But, not all votes are equal  neuronal vote is not a true democracy.
Some ‘yes’ votes count more than others do.
Example, an affirmative decision by a postsynaptic neuron may require only a relatively small
number of ‘yes’ votes from its heavily weighted inputs.
Use McCulloch-Pitts neurons to study pattern recognition
Neurons function in discrete time intervals and have no memory of their past excitation.
i.e. assume that successive input sets, and the postsynaptic decisions they drive, occur in discrete,
sequential time intervals (an input set contains one or more input patterns).
Postsynaptic neuron makes a decision at each time interval based solely on its current input
weights, its active inputs, and its threshold.
Net synaptic excitation of a McCulloch-Pitts neuron is a linear function of its inputs.
Each synaptic connection scales its axon’s value and then the postsynaptic neuron adds up these
scaled values.
Each postsynaptic neuron has its own distinct summation process, which converts the current set
of active synapses into a scalar value termed the internal excitation or net input excitation (or just
excitation for short).
However, the output of such a McCulloch-Pitts neuron is either zero or one.
Crude binary encoding of the internal excitation:
neuron fires (output of one) when its net input excitation exceeds a certain value called
‘threshold.’
Threshold is the minimum value of the sum of the weighted active inputs needed for the
postsynaptic neuron to fire.
All neurons in model operate on same temporal cycle.
One computational cycle of a network for the weighted summation of active inputs and the
thresholding to occur.
On the next computational cycle of the simulation, this process is repeated.
Remember that a postsynaptic neuron does not remember what happened on the last cycle (note
that we use the phrase ‘computational cycle’ to describe computations by the network model)
NEURON NETWORK
Assess performance on pattern recognition problems
Look at simple network:
Single postsynaptic neuron receives connections from three different presynaptic inputs.
Computational elements of a McCulloch-Pitts neuron
Neuron, including its inputs, is a parallel computational device 
A neuron with n inputs performs n multiplications and n-1 additions at each discrete
time step.
The multiplications are performed in parallel—each synapse multiplies the incoming
excitation from each input by its weight.
McCulloch-Pitts neurons: these products are summed instantaneously with the total
ending up in the cell body.
Neuron then compares this summated excitation with its threshold, and this comparison
decides whether or not the neuron fires.
On the next time step, the whole process begins again.
Re: a McCulloch-Pitts neuron forgets its summated internal excitation from one time
step to the next.
The minimum, quantitative description of a McCulloch-Pitts neuron requires:
1. The set of all synaptic weights which defines the potency of each presynaptic input
to the neuron
2. The threshold for firing this neuron
Recapitulate: a McCulloch-Pitts postsynaptic (output) neuron
1. Has no memory from one time step to the next
2. Within a time step, it linearly sums its inputs as scaled by their respective synaptic
weights
3. It fires (output=1) if the value of its internal excitation exceeds a preset threshold
value of excitation, otherwise it does not fire (output=0).
presynaptic
postsynaptic
dendrite
A single McCulloch-Pitts neuron with:
three inputs (1, 2, and 3 that take on values x1, x2, and x3, respectively).
Each synapse is a connection formed between a presynaptic axon (input line) and the
single postsynaptic dendrite (indicated by the darkened circular symbol, .
The
 indicates the summation of internal excitation (analogous to the altered
polarization of neuronal cell body).
The output of the postsynaptic neuron is another axon that transmits any action
potentials of the postsynaptic neuron.
Summarize: McCulloch-Pitts neuron consists of:
 Its synaptic weights that scale its excitatory inputs,
 A summation process that adds up the scaled inputs,
 A spike generator that crudely encodes the value of this summation.
Two Types of Variables in Simulations
1. Variables whose values you choose
2. Variables whose values the simulation generates
In the first case: we parameterize the computational elements
In the second case: variables are continuously changed at successive time steps by the
computational steps of the simulation.
Parameterized variables that we choose for each postsynaptic neuron includes:
1. the value of each synaptic weight, and
2. the firing threshold.
The category of the calculated, changing values of a neuron includes time itself and
1. the internal excitation of each neuron, and
2. the neuron’s output (fire or not fire).
Simple Simulation: computational parameters that you specify are fixed for any one
simulation (pattern recognition), and there is no synaptic modification (no learning).
Internal excitation—a linear computation
What happens when an input vector arrives at a postsynaptic neuron?
The internal or postsynaptic excitation of a McCulloch-Pitts neuron is the linearly scaled
sum of its inputs
Individual synaptic weights specify the scaling on a connection-by-connection basis (the
terms synaptic weight and synaptic strength are interchangeable).
Example: input at time step one
x 1
1
0
0 .5
synaptic weights of a postsynaptic neuron named j
Figure on next slide 
wj
0 .1
0. 3
0.2
A single McCulloch-Pitts neuron
On one time step the inputs take on values xi where i{1,2,3}
Each synapse, wij, is formed by a presynaptic input line i connecting to postsynaptic
neuron j.
Here each input xi(t) makes a single synapse with the postsynaptic neuron called j.
Inputs x depend on state of presynaptic neurons and are time dependent.
To keep notation simple, the time notation, t, has been suppressed in diagram.
Excitation  (next slide)
EXCITATION
yi(1): the internal excitation of postsynaptic neuron i at time step one (reverse subscripts
meaning)
Excitation: the sum of the inputs scaled by their respective synaptic weights
3
yi ( 1 ) =  x j ( 1 )wij = 1  0.1+ 0  0.3 + 0.5  0.2 = 0.1+ 0 + 0.1 = 0.2.
j=1
n
yi =  x j wij
Time notation is suppressed
j=1
Notation for a synapse: wij  woutput,input (note subscript notation)
Synapse is the connection between two neurons  requires two integers to specify
Subscript j indicates a specific input neuron
Subscript i indicates a specific output neuron
This example has three synapses:
w11, the synapse connecting input neuron 1 with output neuron 1;
w12, the synapse between input neuron 2 and output neuron 1;
w13, the synapse between input neuron 3 and output neuron 1.
THRESHOLD
Threshold: parameter of a neuron that defines the minimum excitation needed for that
neuron to fire.
Logical operation that compares two values
if excitation of neuron is greater than its threshold value (logical true) then that neuron
fires an action potential;
if it is false, the neuron does not fire.
PATTERN RECOGNITION
Pattern: x-vector giving state (x values) of group of pre-synaptic neurons
Pattern Recognition: Post-synaptic neuron fires in response to input from x-vector
THOUGHT PROCESS (Consciousness?)
Parameterize the synaptic weights and the threshold of the post-synaptic neuron to
recognize a pattern(s) successfully
First Example (trivial)
All synaptic weights (w) are positive and threshold=0
x1
w1
x2
x3
w2
w3
x4
w4
y
 neuron fires in response to all input patterns (even x=0)
Trivial and uninteresting because neuron can only make one decision —
it recognizes everything (fires at every time step)
 neuron transmits no information (response is independent of input pattern x)
Neuron is not wrong, just boring
How to parameterize model to recognize different patterns
Example: Two different simple patterns
x1>x2
Pattern recognition: neuron y1 fires if x1>x2
neuron y2 fires if x1<x2
x1
w11
x2
w21 w12
y1
Re:
y  w .x
or x1<x2
w22
y2
n
i.e.
yi   wij x j
i 1
To recognize x-patterns, Parameters that can be varied are:
weights (w) and thresholds (h)
Define problem: which parameter is fixed versus which do you need to determine
Example: assume w are pre-determined based upon considerations such as
physiological information
Method for determining thresholds h1 and h2 to recognize patterns
Extremely simple example:
Normalize w and x each to 1 i.e.  xi2 =1 and i wij2 = j wij2 = 1
(This is not a real restriction, just normalization)
Simple weights
 w11 w12  = 1 0

  
 w21 w22  01 
Neuron x1 can only excite y1  w11=1, w21=0
Neuron x2 can only excite y2  w12=0, w22=1
x1
x2
w11
w22
y1
y2
Click mouse on screen for slide to evolve:
Draw unit circle and place w1j and w2j
Next: on circumference, locate point that corresponds to boundary between two patterns
Boundary conditions xb: x1=x2 and x12+x22=1  x12+x12=2x12=1
 x1=1/2=0.707  x2=0.707  tan=.707/.707=1  =45o
Drop perpendicular from xb to w1 vector: intersection is h1=(1) cos
Drop perpendicular from xb to w2 vector: intersection is h2=(1) sin
w2=(0,1)
xb
h2= 1/2 =.707
x2
45o
x1
h1= 1/2 =.707
w1=(1,0)
Graphical Method becomes complicated for wij0, and hard to visualize for > 2 input
neurons (> 2-D graph)
General Mathematical Way for Determining Thresholds
Relax restrictions that required i wij2 =j wij2 =1 but maintain  xi2 =1 (can relax later)
Still have basic relationship between input (x), weights w, and output excitation y
n
y=w•x
i.e.
yi   wij x j
j 1
Calculating thresholds h requires knowledge of boundary input vector xb
Iff xb is known then thresholds h are calculated from
n
h=w•xb
i.e.
hi   wij xbj
j 1
xbi are elements of boundary vector
Example: next page
Click mouse on screen for slide to evolve:
Determine h1 and h2 for a network
y1 to recognize the pattern (fire) if : x1>2x2
and

y2 to recognize the pattern (fire) if : x1<2x2
xb =
but
.894 
 
.447 
New wiring diagram (weights) with all (forward) connections
x1
w11
x2
w21 w12
y1
h1=w11xb1+ w12xb2
Example: w11=0.75 w12=0.23 
w21=0.45 w22=0.62
w22
y2
h2=w21xb1+ w22xb2
h1 = (0.75)(.894)+(0.23)(0.447) = 0.773
h2 = (0.45)(.894)+(0.62)(0.447) = 0.679
Check: x1=0.80 x2=0.60  x1<2x2
y1= (0.75)(.80)+(0.23)(0.60) = 0.738 <h1  does not fire
y2= (0.45)(.80)+(0.62)(0.60) = 0.732 >h2  fires
x1=0.95  x2=0.31: x1>2x2
y1=0.784 fires
y2=0.62 no fire