Download Self-Organizing Visual Cortex Model using Homeostatic Plasticity

Self-Organizing Visual Cortex Model using
Homeostatic Plasticity Mechanism
Veldri Kurniawan
S0565942
Master of Science
School of Informatics
University of Edinburgh
2006
Abstract
Homeostatic plasticity is regulatory mechanism that modulates both synaptic
connections and intrinsic properties of individual neuron in order to maintain stability
of neuronal activity. This type of plasticity is considered to be an important
complement to Hebbian plasticity, especially during learning and development
process of nervous system. In this project, the role of homeostatic plasticity in selforganizing visual cortex model (LISSOM) will be investigated. Two computational
model of homeostatic plasticity, gradient rule for intrinsic plasticity proposed by
Triesch [27] and homeostatic synaptic scaling proposed by Sullivan and de Sa [26]
will be applied to LISSOM model of V1. Several experiments will be conducted to
evaluate the self-organization process and the organized map of the model with
homeostatic mechanism. The result of this initial exploration indicates that
homeostatic plasticity, particularly intrinsic plasticity, can be successfully integrated
into LISSOM model of visual cortex to improve some of its behavior and make it
more biologically realistic.
i
Acknowledgement
I would like to express my sincere gratitude to my supervisor, Dr. James A. Bednar
for his help, guidance, support and inspiration during the course of this project. I
would also like to thank Lewis Ng and Tikesh Ramtohul for fruitful discussions on
many technical aspects related to this project.
ii
Declaration
I declare that this thesis was composed by myself, that the work contained herein is
my own except where explicitly stated otherwise in the text, and that this work has not
been submitted for any other degree or professional qualification except as specified.
(Veldri Kurniawan)
iii
Table of Contents
Chapter 1 Introduction................................................................................................1
1.1 Objectives ............................................................................................................2
1.2 Document Structure .............................................................................................2
Chapter 2 Background ................................................................................................4
2.1 Biological Basis of Homeostatic Plasticity..........................................................4
2.1.1 Homeostatic Regulation in Biological Systems............................................4
2.1.2 Hebbian Learning and Problem of Stability .................................................5
2.1.3 Evidence of Homeostatic Regulation in Nervous Systems...........................7
2.1.4 Mechanisms of Homeostatic Plasticity.........................................................9
2.2 Computational Models of Homeostatic Plasticity .............................................13
2.2.1 Homeostatic Synaptic Scaling Model of Sullivan and de Sa......................14
2.2.2 Triesch’s Intrinsic Plasticity Learning Rule ...............................................16
2.3 The LISSOM Model ..........................................................................................18
2.3.1 Architecture of the LISSOM Model of V1 .................................................19
2.3.2 Threshold Adaptation in LISSOM..............................................................24
2.3.3 LISSOM without Shrinking Lateral Excitatory Radius..............................26
Chapter 3 Method & Parameters.............................................................................29
3.1 Choice of the Models .........................................................................................29
3.2 Experiment with Intrinsic Plasticity Applied to LISSOM Models....................30
3.2.1 Intrinsic Plasticity in Simplified LISSOM Network...................................30
3.2.2 Sensitivity of Intrinsic Plasticity Parameter Settings..................................33
3.2.3 Intrinsic Plasticity Applied to LISSOM with LGN ON/OFF Channel.......34
3.2.4 Intrinsic Plasticity Applied to LISSOM without Shrinking Lateral
Excitatory Radius.................................................................................................35
3.3 Experiment with Homeostatic Synaptic Scaling Applied to LISSOM Models.35
Chapter 4 Results.......................................................................................................37
4.1 Intrinsic Plasticity Applied to Simplified LISSOM Network............................37
4.1.1 Receptive Fields and Orientation Maps......................................................37
4.1.2 Average Firing Rate, Firing Rate Distribution, and Speed of Threshold
Adaptation............................................................................................................44
4.2. Sensitivity of Intrinsic Plasticity Parameter Settings........................................48
4.2.1. Initial Threshold Values.............................................................................49
4.2.2. Intrinsic Plasticity Learning Rate...............................................................52
iv
4.2.3. Target Average Firing Rate .......................................................................54
4.3. LISSOM with LGN ON/OFF and Intrinsic Plasticity ......................................55
4.4 Intrinsic Plasticity Applied to LISSOM with no Shrinking Lateral Excitatory
Radius ......................................................................................................................58
4.5. Homeostatic Synaptic Scaling ..........................................................................61
Chapter 5 Discussion and Future Works.................................................................66
5.1. Discussion .........................................................................................................66
5.1.1. Intrinsic Plasticity and LISSOM Models...................................................66
5.1.2. Synaptic Plasticity and LISSOM Models ..................................................71
5.2. Future Works ....................................................................................................71
Chapter 6 Conclusions...............................................................................................73
Bibliography ...............................................................................................................74
v
Chapter 1
Introduction
Homeostatic plasticity is a recently proposed regulatory mechanism that modulates
both synaptic connections between neurons and intrinsic electrical properties of
individual neurons in order to maintain stability of neuronal activity in spite of change
during learning by Hebbian mechanism. This kind of plasticity is considered to be an
important complement of Hebbian plasticity, especially during learning and
development process of nervous system. [13, 29, 30, 31, 36]
Self-organizing visual cortex model that are going to be used in this project is
LISSOM (Laterally Interconnected Synergetically Self-Organizing Map). LISSOM is
one of the most biologically realistic, laterally connected self-organizing map model
of the visual cortex. This model can demonstrate wide variety of phenomena
occurring in visual system including input-driven self-organization, patchy
connectivity, ocular dominance, orientation selectivity, tilt-aftereffect, contour
integration, preference for faces in newborn, and more. [2, 3, 4, 18, 19, 20, 23]
An automatic threshold adaptation is one of the desirable extensions to the LISSOM
model for enhancing its reliability and making it more relevant biologically [19,
chapter 17]. In current LISSOM model, activation thresholds (the upper and lower
bound of neuron’s activation function) should be tuned manually by hand through
trial and error. Besides time consuming for the modeler, this process also likely to be
subjective. Comparison of experiment results under different conditions is also
difficult. Moreover, different types of input pattern will require different threshold
settings, depending on how strongly it activates the network. High-contrast abstract
patterns might require lower threshold values as opposed to low-contrast natural
images which require higher threshold settings. LISSOM model currently use
Hebbian learning rule combined with divisive weight normalization to prevent
unconstrained synaptic weight growth. While this normalization rule is useful in
practice, there is little evidence from biological experiment that confirm the existence
of such mechanism. Recent experiments by Turrigiano et. al. [29, 30] suggest that
neurons might use a kind of homeostatic plasticity to regulate its firing rate by
multiplicatively scaling all synapses strength depending on recent history of inputs.
The Topographica cortical map simulator [2] will be used for testing and
implementing various homeostatic mechanisms in the LISSOM model. Topographica
1
is a large-scale, general purpose simulator for topographic map formation in the brain.
As a testbed for initial testing of various homeostatic plasticity mechanisms, a very
simplified model of visual system which consists only of one input layer and one
output layer will be set up in Topographica. After the implementations have been
verified to be correct, more complete visual models will be tested using the new
homeostatic plasticity rules. It is expected that with addition of homeostatic plasticity,
the original functionality of LISSOM such as ability to self-organize and form
orientation selectivity will still be working well. Several aspect of stability will be
evaluated such as sensitivity of threshold setting, stability of neurons activity over
long period of time, and long term stability of the weight values. The effect of
homeostatic plasticity mechanism (if any) on the original functionality of LISSOM
(such as self-organization, orientation and direction selectivity, and other features)
will be also investigated.
1.1 Objectives
The objective of this project is: 1) to extend the LISSOM model to use automatic
threshold adaptation based on homeostatic plasticity mechanism. A mechanism of
homeostatic threshold adaptation will be implemented and tested using Topographica
simulator [2]. If this mechanism can be successfully implemented, the need of
manually tuning the threshold function can be avoided and therefore make the
simulator more reliable and more applicable in real situations where the visual system
receives inputs from natural scenes. 2) to replace the weight normalization rule used
in Hebbian learning process in LISSOM with a form of homeostatic synaptic scaling,
which is more relevant biologically. It is hoped that by addition of automatic
threshold adaptation and homeostatic synaptic scaling, the resulting system will be
less sensitive to threshold settings, have more stability to cope with different inputs,
and also more biologically plausible.
1.2 Document Structure
Chapter 2 presents the background information relevant to the project concerning the
biological evidence and mechanism of homeostatic plasticity that recently found in
nervous system, followed by overview of computational model of homeostatic
plasticity. Lastly, the architecture of basic LISSOM model of V1 will be described.
2
Chapter 3 describes the method and parameters of the experiments that are conducted
to investigate the behavior of LISSOM models with homeostatic plasticity mechanism.
It starts by explanation of the reason behind the choice of the models, and then
followed by two sections describing detailed experiment with intrinsic plasticity and
homeostatic synaptic scaling.
Chapter 4 presents the results obtained from the experiments of applying homeostatic
plasticity mechanism into LISSOM. The resulting self-organized orientation maps and
other related plots that show the detailed properties of the network during selforganization will be examined.
Chapter 5 presents a discussion and analysis of the experiment results and suggestions
for future works related to this topic.
Chapter 6 presents the conclusions.
3
Chapter 2
Background
This chapter presents a literature review of homeostatic plasticity and LISSOM model.
The first part explains the biological evidence and mechanism of homeostatic
plasticity that recently has been found in nervous system. The second part gives an
overview of some computational model of homeostatic plasticity and detailed
exposition of the model that are going to be used in this project. The third part
explains the basic LISSOM model of V1.
2.1 Biological Basis of Homeostatic Plasticity
2.1.1 Homeostatic Regulation in Biological Systems
Homeostasis is a regulatory mechanism of an open system to maintain its internal
equilibrium by means of continuous dynamic adjustments. The term was coined by
Walter Cannon in 1932, derived from the Greek words ‘homoios’ which means
‘similar’/‘same’ and ‘stasis’ which means ‘standing still’. He suggested a new term
homeostatic instead of using the term equilibria which is commonly used in simple
control systems and explains the reason behind this in his seminal book, The Wisdom
of the Body (1932):
The constant conditions which are maintained in the body might be termed
equilibria. That word, however, has come to have fairly exact meaning as
applied to relatively simple physico-chemical states, in closed systems, where
known forces are balanced. The coordinated physiological processes which
maintain most of the steady states in the organism are so complex and so
peculiar to living beings – involving, as they may, the brain and nerves, the
heart, lungs, kidneys and spleen, all working cooperatively – that I have
suggested a special designation for these states, homeostasis. The word does
not imply something set and immobile, a stagnation. It means a condition – a
condition which may vary, but which is relatively constant.
Canonn’s insight into the nature of stability behind biological systems originated from
his observation that complex self-righting adjustment (self-organization) might be the
way used by an organism to regulate its internal mechanism in face of tremendous
amount of changes and perturbations. He proposed four tentative general features of
homeostasis in his book:
4
1. Constancy in an open system, such as our bodies represent, requires mechanisms that
act to maintain this constancy.
2. Steady-state conditions require that any tendency toward change automatically meets
with factors that resist change. An increase in blood sugar results in thirst as the body
attempts to dilute the concentration of sugar in the extracellular fluid.
3. The regulating system that determines the homeostatic state consists of a number of
cooperating mechanisms acting simultaneously or successively. Blood sugar is
regulated by insulin, glucagons, and other hormones that control its release from the
liver or its uptake by the tissues.
4. Homeostasis does not occur by chance, but is the result of organized self-government.
There are many known examples of homeostatic regulation in living organism such as:
regulation of body temperature, regulation of the amounts of water and minerals in the
body, regulation of blood glucose level, regulation of blood pressure, and the recently
proposed regulation of neural activity. Homeostatic regulation mostly occurs through
negative feedback mechanism, in which the system reacts in such a way to reverse the
direction of change and thus maintain stability. One examples of negative feedback is
Thermoregulation. When body temperature falls, receptors in the skin and
hypothalamus register a change and send signals to part of the brain which controls
the temperature. The brain will then respond by sending signals to other parts of the
body to increase the temperature.
2.1.2 Hebbian Learning and Problem of Stability
Nervous systems are highly plastic. During the development stage and lifetime of an
individual, neurons can dramatically change in shape/size, synaptic connections, and
their intrinsic properties. Neuron’s plasticity is important for learning and adaptation
to the continuously changing environment. The outcome of many years of research in
neuroscience suggests that plasticity and learning are facilitated by modification in the
number and strength of synaptic connections and also by alternations in the internal
properties of neurons (such as ion channels). [13, 29, 30, 31]
The most celebrated theory of activity-dependent plasticity, Hebbian Learning, has
dominated neuroscience research since its introduction by Donald Hebb in 1949.
Hebbian rule states that synaptic strength between pre-synaptic and post-synaptic
5
neurons will be strengthened when both neurons fire in correlation and weakened
when they fire out of correlation.
This rule is extremely powerful in explaining the phenomena of information storage
and learning in neural system. However, because Hebbian plasticity relies on positive
feedback mechanism, it is also highly unstable and could drive neurons to become
hyper-activated or silenced altogether. Suppose we have a hypothetical pre-synaptic
neuron connected with a bunch of synapses to one post-synaptic neuron. Consider the
case of positive correlation in which the firing of pre-synaptic neuron caused
potentiation of post-synaptic neuron. According to Hebbian rule, synapses between
these two neurons should be strengthened, which in turn would allow the pre-synaptic
neuron to drive the post-synaptic one more strongly. The increase in post-synaptic
activation would tend to strengthen the synapses even more, which lead to even more
activation and so on, until it reach maximum saturation point. The reverse case with
uncorrelated input is also similar. The synapses strength would be weakened
progressively and selectivity of post-synaptic neuron falls to zero.
Computational modelers who were working with Hebbian rule in the early times
realized that this stability problem can prevent the application of this rule in larger
models so they incorporated some kind of normalization rule to limit the growth of
synaptic strength within bound. Although simple form of normalization usually works
well in practice, there is no biological evidence that support such operation in real
organism. This early form of ad-hoc normalization is practically a simplification of
more complicated homeostatic regulation that counterbalances Hebbian plasticity [31].
Turrigiano and Nelson [31] give a very helpful illustration of the importance of
regulating excitability of neural circuit in feedforward neural network architecture
(figure 2.1). Suppose there are several layers of networks connected in hierarchical
fashion by feedforward connections from the input layer to several higher layers
above. We can view this simplified network topology as resembling connection of
neurons in primate visual systems starting from photoreceptors, bipolar cells, ganglion
cells, LGN, V1, V2, and so on. Input pattern is received by neurons in the first layer
and propagated through synaptic connections to the second layer neurons. From the
second layer, the signal is forwarded to the third layer neurons, and so on up to the
highest layer. In this simplified model, it is obvious that the gain of signal
transmission from one layer to the next should be close to one in order to retain the
specificity of the pattern. If the gain is higher than one, the signal will be continuously
amplified every time it passed through one layer of neurons, and at the highest layer
the firing rate will reach saturation and information about the pattern is lost. If the
6
gain is lower than one, the signal will be weakened over time and fail to reach the
highest layer neurons. This example shows that the ability of neurons to maintain
stable/constant rate of activity is very important for conserving the information that is
being propagated through the network of neurons.
Figure 2.1. This figure illustrates the problem of propagating information in feedforward network. (a)
Abstract feedforward network consisting of 5 layers of neurons. (b) The graph shows expected firing
rates of the red neuron in each layers when given sinusoidal input. If the activity is subsequently
amplified when passing from lower layer to higher layer, in the end it will saturates and the information
about the input is lost. (c) In this case, the activity is subsequently weakened in each layer, and fails to
propagate to highest layer. (d) When every neuron maintain stable firing rate, the signal can be
transmitted up to the highest layer. (adapted from [31])
2.1.3 Evidence of Homeostatic Regulation in Nervous Systems
Recently, an increasing amount of experimental evidence suggests that several
different kinds of homeostatic regulation are being exercised by nervous systems in
order to maintain neural excitability within a normal physiological range [12, 29, 31].
Nervous System face tremendous amount of perturbations in the early development
stage and during the life of an organism, including changes in cell structure and size,
fluctuations in synaptic strengths, and innervations. In face of such distress, neurons
can still maintain stable level of excitation by regulating several diverse properties
such as synaptic size, synaptic strength, synaptic number, and ion channels that
controls neuronal excitability [13, 12, 31].
The results of many experiments suggest that the ability of neurons to maintain their
excitation levels might implies the capability of monitoring their own internal activity
and transduce this information into regulated changes in synaptic and ion channel
7
properties. This kind of monitoring and feedback mechanism is thought to be similar
with other homeostatic regulation found in other biological systems and may provide
a basis for understanding how stability in neural system is maintained [12].
Experimental evidence of homeostatic regulation has been found in systems ranging
from invertebrate neuromuscular junction (NMJ) to synapses in the vertebrate cortex.
During each of these experiments, the neurons were artificially perturbed to alter the
innvervation, synaptic function, and cell growth.
In the recent study which involves advanced genetic manipulations, it has been shown
that homeostatic regulation plays role in Drosophila neuromuscular junction to keep
neuromuscular transmission relatively constant in the midst of altered synaptic
properties. In this experiment, the number of post-synaptic receptors of Drosophila
NMJ is impaired or reduced genetically. By doing this, we expect abnormal muscle
depolarization, but contrary to the expectation, the muscle can still depolarize
normally because impaired post-synaptic receptor is precisely compensated by
increased presynaptic transmitter release. A similar homeostatic regulation is also
observed at neuromuscular junction of mouse with genetically modified post-synaptic
acetylcholine receptor. The reduction of post-synaptic receptor is precisely
compensated by increase in transmitter release so the muscle can still function
normally. [12, 29, 31]
Another evidence of homeostatic response was observed in lobster stomatogastric
ganglion (STG) during in vitro experiment. In its natural place, STG neurons fire in
bursts due to both synaptic inputs and intrinsic conductance. Remarkably, when STG
neurons are isolated from their synaptic circuit and placed in a cell culture, they can
regenerate their bursting pattern in absence of any synaptic input by coordinating
changes in its voltage-gated channel densities. This homeostatic adaptation can be
stopped by supplying rhythmic pattern similar with neuron’s normal synaptic inputs
or by preventing Ca2+ entry to the neuron. Several theoretical and experimental works
suggest that chemical signals such as Ca2+ play important role in balancing intrinsic
electrical properties of neurons in homeostatic way. [12, 29]
Many experimental results also confirm existence of homeostatic regulation in central
nervous system (CNS). Neurons in CNS are much more complex compared to NMJ
or STG neurons. These neurons receive inputs from thousands of other neurons and
carry out fluctuating signal over short time scale to transfer information. Over longer
time scale these neurons should also regulate themselves homeostatically to prevent
runaway excitation or die out. One evidence of this regulation is found in the
8
experiment of blocking some inputs to neurons in invertebrate networks and
vertebrate spinal cord. Initially the neurons cease to have spontaneous activity but
over time the activity recovers to normal state. Another experiment with cortical
pyramidal neurons using cortical cultures also suggests that these neurons can
maintain a set-point firing rate in the face of changing synaptic input. Similarly,
regulation of firing rates was also found when Kir potassium channel was expressed in
individual cultured hippocampal neurons. This channel create a hyperpolarizing
current that lowers firing rates in the beginning, but over time firing rates return to
normal state even with continued expression of the channel. This result demonstrates
that homeostatic regulation of firing rate can take place at the level of individual
neuron. [31]
2.1.4 Mechanisms of Homeostatic Plasticity
The exact mechanisms of homeostatic regulation in nervous systems remain to be
investigated in details. Current evidences also indicate the possibility that various
different types of regulation might actually taking place at the same time.
Homeostatic regulation not only occurs at the level of individual neuron but also
involve complex intercellular signaling. Some hypotheses even suggest that it may
also take place at larger circuit or system level. [12]
Several different mechanisms that promote homeostatic plasticity in central nervous
system have been proposed in recent years. These mechanisms include multiplicative
synaptic scaling, the regulation of intrinsic excitability of neurons, the BCM model,
and the change in the brain derived neurotrophic factor (BDNF) [13, 14, 29].
Experiments on these mechanisms indicate that neurons have an ability to utilize their
own activity as a feedback signal to modify intrinsic excitability and total synaptic
strength with the aim of maintaining long term stability. However, the exact activity
that is being regulated is still not clearly defined. It could be average firing rate,
average calcium concentration, or some statistical measure of network activity (such
as maximization of information transfer) [30].
2.1.4.1 Multiplicative Synaptic Scaling
One way to maintain neuron stability is using synaptic scaling mechanism that
preserves the average neurons activity level by multiplicatively increasing or
decreasing the strength of all synaptic connections, depending on previous activities.
When the neuron firing rate increases, the strength of all excitatory connections will
9
be scaled down, and conversely when the neuron firing rate fall, they will be scaled
up [13, 29, 30]. Experimental evidences of synaptic scaling can be found in a work on
neuromuscular junction and also from cultured networks of central neurons. More
recent experiments also found similar behavior in living brain. [13]
One important difference between synaptic scaling with much studied form plasticity
such as LTP/LTD is the speed of change. LTP/LTD can induce change in synaptic
strengths quite rapidly, while synaptic scaling is much slower, requiring hours or days
of modified activity to produce measurable changes. This also shows that neurons
integrate average activity over long period of time, instead of responding to momentto-moment fluctuations. If homeostatic regulation occurs very rapidly, it would
destroy signal variations that are used for information transmission. Conversely, if it
is too slow, it will not have enough power to counterbalance destabilizing effect
caused by other type of plasticity. The exact rate of this balance is still unknown. [13,
30, 31]
The other important distinction between LTP and synaptic scaling is that instead of
modifying specific-synapse individually, synaptic scaling will scale all synaptic
connections up or down by the same multiplicative factor. This characteristic ensures
that modification of synapses will still preserve the relative difference between
individual synaptic inputs, which means that information learned by LTP/LDP
plasticity such as Hebbian rule will be kept intact while at the same time the neurons
can maintain stable firing rate. Therefore we can see that Hebbian plasticity and
synaptic plasticity works together complementary to encode information and
regulating overall neurons activity within acceptable limit [13, 29, 30]. The figure
below illustrates the effect of synaptic scaling on the number of receptors at each
synapses.
Figure 2.2. Illustration of multiplicative scaling effect on the number of receptor in synapses. The left
picture shows two synapses with equal strength. When one presynaptic neuron is potentiated, the
corresponding synapse in postsynaptic terminal will have an increase in the number of receptors which
result in increased firing rate. After multiplicative scaling (right picture), the firing rate returns to
10
normal again because removal of some AMPA receptors in postsynaptic terminal. However, the
relative difference between the two synapses is preserved. (adapted from [31])
How can a neuron regulate its own excitability ? We might suspect that in order to do
this task, neuron should have some mechanism to monitor its own history of
activation and regulate the number of ion channels accordingly. But where is this
activity monitor located. One possible answer is that activity monitors are located in
the cell soma and they are capable to integrate action-potential activity over time. This
central location would allow genetic transcriptional response at the nucleus during
homeostatic signaling. The signal could then be transmitted to all synapses. The
second possible answer suggests that homeostatic mechanism could be locally
managed post-synaptically at each synapse. Every synapse would regulate its own
excitability using information about sum total excitation received through action
potential back propagation. [12]
2.1.4.2 Intrinsic Plasticity
Another form of homeostatic plasticity which can promote stability in neurons
activity is Intrinsic Plasticity. Many experimental evidences suggest that somehow
neurons can also regulate their intrinsic electro-chemical properties (which govern the
way they integrate synaptic inputs and produce spikes) according to previous history
of their activations. These intrinsic properties are influenced by the amount of
intrinsic ion channels and not by channels associated with synaptic terminal.
Changing the quantity and distribution of these channels can influence the neural
excitability, synaptic integration, firing rate and firing pattern, and synaptic plasticity.
Experimental evidences of intrinsic plasticity have been found in the culture of
neocortical cells and also demonstrated in the functioning of cerebellar granule cells
(neurons that coordinate movements). [13, 31, 36]
Molecular mechanism of intrinsic plasticity might be related to chemical signaling
process (such as intracellular Ca2+ concentration) that regulate the balance of neuron’s
outward and inward conductances depending on the level of excitation (figure 2.3).
When the neuron activity is higher than the target activity, Ca2+ flows into the cell and
raises intracellular Ca2+ concentration which in turn will change the proportions of
ion-channels and adjust conductances to reduce the activity level. On the other hand,
when the neuron activity is low, intracellular Ca2+ concentration is reduced and the
proportions of ion-channels and conductances will be regulated to increase the level
of activity. [29]
11
Figure 2.3. Schematic showing regularization of internal Ca2+ concentration depending on the activity
of the neuron (adapted from [29])
Figure 2.4. Intrinsic plasticity effect can be seen as shifting the F/I curve to the left or right depending
on the neuron’s level of activity. (figure adapted from [13])
The figure above shows the typical F/I curve which relates the synaptic input current
with the neuron’s firing rate. This figure illustrates that intrinsic plasticity works by
changing the way the input is transformed into firing rate, instead of modifying
synaptic connections. The curve in bold shows the initial condition of the neuron. In
this condition, if the given input is too low, the neuron will not fire, and if it is too
high it will reach saturation. The neuron will only be selective in the middle range
between these two states. When the average neuron’s activity over long period of time
became very low, intrinsic plasticity mechanism will increase the neuron’s
excitability and effectively translate the F/I curve to the left. Conversely, when the
average neuron’s activity became too high, plasticity mechanism will make the
neuron less excitable and effectively translate the F/I curve to the right. The shifting
of F/I curve will prevent the neuron to fall into complete silent or complete saturation,
even when the input became very weak or very strong. [13]
2.1.4.3 Unifying Framework
It would be very interesting to see one unifying framework which explains every
kinds of homeostatic plasticity in one general basic mechanism. Results from current
experiments have shown that both synaptic scaling and intrinsic plasticity seems to be
governed by common underlying subcellular pathways which is the activity12
dependent brain-derived neurotrophic factor (BNDF) [14]. However, detailed
mechanism of how BNDF operates is still not well understood.
Another interesting question to be asked is about connection between homeostatic
plasticity and Hebbian plasticity in deeper subcellular level. Whether both of them are
governed by the same biochemical mechanism such as BDNF or not is still an open
question. [13]
Further issues concerning homeostatic plasticity is the question about what activities
are exactly being regulated by this mechanism. Current experiments suggest that the
number of regulated activities seems to be very diverse. Usual form of homeostatic
plasticity deals with regulation of firing rates and maintaining synaptic strengths
within bounds. However there are some evidences which show that other kinds of
activity are also being regulated, such as the regulation of firing pattern in
stomatogastric ganglion (STG) of decapod crustaceans [31]. Another experiment on
organotypic rat spinal culture also demonstrates that regulation is not only present in
individual neuron but also take place in larger network-wide activity. [13]
2.2 Computational Models of Homeostatic Plasticity
Several theoretical and computational models of homeostatic plasticity mechanisms
have been proposed by some researchers based on the results and evidences from
biological experiments. Most of the models that exist in current literature are very
detailed low level molecular models. Some of them explain the phenomenon of
homeostatic in the level of single neuron using voltage-gated ion-channel model [24]
and some others use integrate-and-fire neurons model [17, 32, 34]. For the purpose of
this project, we need more abstract homeostatic plasticity models which can be
applied to rate-based neurons model. Quite recently, several models like this have
been emerged.
Sullivan and de Sa, in their recent papers [25, 26], proposed one possible
implementation of homeostatic synaptic scaling (together with Hebbian learning) in
one dimensional Self-Organizing Maps neural network. They applied multiplicative
synaptic scaling by replacing normally used weight normalization scheme in Hebbian
rule with normalization based on the average activity of the neurons over period of
time. The result of the experiment is quite promising as the network is able to selforganize and also regulate Hebbian learning from overrun at the same time. Moreover,
13
they also reported that the network can still maintain stability of firing rate in the face
of network growth and neurons die-out.
Another interesting model which implements intrinsic plasticity was recently
proposed by Triesch [27]. Basically, the author use information theory and apply the
hypothesis that neurons might use intrinsic plasticity to obtain an exponential
distribution in firing rates and therefore maximize the entropy. Maximization of
entropy was thought to be beneficial because it allows neurons to process maximum
amount of information using minimum amount of energy. Triesch develop a gradient
descent rule for adjusting the shape and position of the sigmoid F/I curve. The
gradient rule is derived by measuring Kullback-Leiber divergence between
distribution of current firing rate and desired exponentially distributed firing rate. This
model was applied to a continuous activation model neuron and proven to be
successful in producing approximately exponential firing rate distribution. In another
publication [28], Triesch reported that precursor of this intrinsic plasticity model
which uses simpler proportional-control-law rules also works similarly. More recently,
Triesch and Butko [6, 7] presented the result of applying this model together with
SOM and Hebbian Learning to learn V1-like maps in visual cortex.
Model of homeostatic plasticity based on well known BCM (Bienenstock, Cooper,
Munro) model have also begin to emerge. BCM model was originally proposed to
supplement Hebbian learning with competition mechanism, but experiments show
that it can also promote stability in neurons while learning using Hebbian rule.
Stability in BCM model is achieved by sliding the threshold of
depression/potentiation curve depending upon the history of neuron’s activity. More
recently, a model called UCM (Unified Calcium Model) has been proposed by
Shouval, et. al. [21] as the common mechanism of various bi-directional synaptic
plasticity. A neuron that uses this model together with BCM has been shown to
develop input selectivity and stability. [8, 34, 35].
2.2.1 Homeostatic Synaptic Scaling Model of Sullivan and de Sa
Sullivan and de Sa [25, 26] proposed an alternative normalization rule based on
homeostatic synaptic scaling to replace commonly used weight normalization in
Hebbian learning. In order to test this rule, they set up linear SOM (Self Organizing
Map) network with Hebbian rule for learning and homeostatic weight normalization,
then trained it with series of random Gaussian input patterns. The input and output
14
neurons of SOM are arranged in ring arrangement to eliminate edge effect. At each
G
iterations, the input vector, x , is presented to the network by multiplication with
feedforward connection weights, WFF. This feedforward activity is then passed
through Mexican hat shape convolution kernel to emulate the effect of lateral
connections.
G
G
y = f ( g ∗ WFF x )
(2.1)
G
Where f(x) = max (0, x), g is Mexican hat shape kernel, and y is the output vector. In
this version of SOM, lateral connection is assumed to be fixed but feedforward
connections are adapted with Hebbian learning rule.
w ijt = wijt −1 + α x tj yit
(2.2)
Where α is learning rate, xj is presynaptic activity, yi is postsynaptic activity, and
w ijt is weight value from input neuron j to output neuron i before weight normalization.
Homeostatic normalization based on recent activity of the neurons is used for limiting
the growth of the weights.
wijt +1
=
w ijt
ActivityNormit
,
ActivityNormit
t
⎛ Aavg
⎞
,i − Atarget
= 1+ βN ⎜
⎟
⎜
⎟
A
target
⎝
⎠
(2.3)
Atarget is preferred target activity level for the neurons, Aavg,i is the average of recent
activity of neuron i, and β N is homeostatic learning rate. Aavg,i is computed as a
running average of the recent activity of the neuron.
t
t
t −1
Aavg
,i = β c yi + (1 − β c ) Aavg ,i
(2.4)
The parameter β C determines the speed over which previous activity of the neuron is
averaged. This normalization rule can be seen as a simplified abstraction of some
internal homeostatic regulation process involving some intercellular chemicals such
as calcium concentration. Since the detailed mechanism of this molecular process is
not yet known, this model uses the running average firing rate and desired target
firing rate directly as a substitute.
15
A linear SOM network with 150 inputs and 15 outputs was used to test this
homeostatic rule [26]. After the network was trained using randomly positioned 1-D
Gaussian pattern, the map was computed by presenting every possible input to the
network and selecting one neuron with the highest activation as the winner for each
input. The resulting self-organized map is good because it forms regular topographic
ordering of the input position (figure 2.5, left). The average activity of the neurons
keeps oscillating around the target value during early training iterations but slowly
dampens over time and finally converged to predefined average firing rate (figure 2.5,
right).
Figure 2.5. The resulting self-organized map and average neuron activities during self-organization.
(adapted from [26])
Sullivan and de Sa also performed an experiment of changing the network
architecture in the middle of learning process to simulate synapses proliferation
during cortical development [26]. Initially, they started with 75 input neurons and
trained the network until average activity is settling down. After that, another 75 input
neurons are added to the network and the learning process continues until the activity
reach stable point. Then 75 neurons are removed again. During this abrupt
modification, they showed that the network can still retain the continuity of the map
and homeostatic mechanism can regulate the firing rate within bound.
2.2.2 Triesch’s Intrinsic Plasticity Learning Rule
It has been shown in biological experiment that neurons in visual cortex exhibit sparse
exponential firing rate distribution when presented with natural scenes [1].
Exponential distribution of firing rate is believed to be an important aspect of the
neurons in order to achieve maximum information processing capacity with limited
16
amount of energy. The reason behind this is because exponential distribution has the
highest entropy among all distribution of a non-negative random variable with a fixed
mean.
Initial experiment conducted using Hodgkin-Huxley model neuron suggests that
intrinsic plasticity mechanism might plays an important role in achieving
approximately exponential distribution firing rate [24]. In that experiment,
unsupervised learning rule that adapts membrane conductance of voltage gated
channels to match the desired exponential firing rate distribution was used. More
recently, Triesch proposed similar learning rule that can be applied to more abstract
rate-based model neuron. The learning rule was initially formulated as proportional
control laws based on low order moments of a neuron’s firing rate [28]. Later on, a
gradient rule based on Kullback-Leiber divergence (KL-divergence) was derived, and
proved to be similar with the previous learning rule [27].
Triesch’s intrinsic plasticity learning rule works by directly adapting the neuron’s
activation function to achieve approximately exponential firing rate distribution. The
activation function used in this rule is the sigmoid function:
y = g( x) =
1
1 + exp( −( ax + b ))
(2.5)
y is the neuron’s output (firing rate) and x is total synaptic current received by the
neuron. a determines the slope/gradient of the sigmoid and b determines its position.
The distribution of firing rate, f y ( y ) is related to the distribution of x, f x ( x ) , by
equation:
f y ( y) =
f x ( x)
∂y
∂x
(2.6)
The gradient rule was constructed by utilizing the KL-divergence between the current
firing rate distribution (denoted by f y ( y ) ) and the desired exponential distribution,
f exp = (1/ µ )exp( − y / µ ) :
17
f y ( y)
⎛
⎞
D ≡ d ( f y || f exp ) = ∫ f y ( y ) log ⎜
⎟ dy
⎝ (1/ µ )exp( − y / µ ) ⎠
⎛ y
⎞
= ∫ f y ( y ) log( f y ( y )) dy − ∫ f y ( y ) ⎜ − − log µ ⎟ dy
⎝ µ
⎠
1
= − H ( y ) + E ( y ) + log µ
(2.7)
µ
Where D is KL distance, H(y) is the entropy of y, E(y) is its expected value, and µ is
desired average firing rate level.
In order to bring the current firing rate distribution as close as possible to the
exponential distribution, the value of D should be minimized. This can be achieved by
constructing a stochastic gradient descent rule that slowly adapts the parameters of
sigmoid function, a and b. By taking partial derivatives of D with respect to a and b,
the following rule was obtained (the detailed derivations is available in [27]).
⎛1
⎞
⎛
1⎞
1
∆a = η ⎜ + x − ⎜ 2 + ⎟ xy + xy 2 ⎟
µ⎠
µ
⎝
⎝a
⎠
⎛ ⎛
1⎞
1 ⎞
∆b = η ⎜ 1 − ⎜ 2 + ⎟ y + y 2 ⎟
µ⎠
µ ⎠
⎝ ⎝
(2.8)
Where η is intrinsic plasticity learning rate.
Application of this rule to single model neuron was proven to be successful in
bringing the neuron’s firing rate close to exponential distribution [27]. The rule also
has been successfully applied to solve the “bars” problem, which is a standard nonlinear ICA problem [6, 28]. On larger network scale, this rule has been applied to
model the self-organization process of V1 oriented maps using SOM-like network and
Hebbian learning. It was shown that after being trained with natural image patches,
the network can form oriented, bandpass, and localized receptive fields that ordinarily
found in V1 maps [6, 7].
2.3 The LISSOM Model
LISSOM (Laterally interconnected synergetically self-organizing map) is one of the
most biologically realistic computational map model of visual cortex that has been
proven to be successful in modeling the structures and processes that take place in V1
and several higher visual areas [18, 19, 20, 23]. This model has been successfully
18
used to demonstrate wide variety of phenomena occurring in visual cortex including
input-driven self-organization, patchy lateral connectivity, ocular dominance,
orientation selectivity, tilt-aftereffect, contour integration, preference for faces in
newborn, and more. [3, 4, 10, 19, 20].
The basic component of LISSOM model is a two-dimensional sheet of neural units
that represent the vertical columns of the cortex. Each unit in the sheet is an
abstraction of six layers of neurons inside the vertical column of real biological cortex.
Such an abstraction is reasonable because neurons in a column generally respond to
similar input. Each cortical unit receives input from neurons in LGN ON/OFF
channels and the LGN neurons in turn receive inputs from retina. The cortical neurons
are connected with lateral excitatory and lateral inhibitory connections that are
adapted throughout self-organization. These lateral connections play an important role
in the development of orientation map and in mediating functions like illusions and
perceptual grouping. Each unit in LISSOM model is modeled after rate-based neuron
model. The response of each neural unit is computed by taking a weighted sum of
their input and squashing it through a sigmoid function. The afferent and lateral
connections of the cortical units are all adapted through Hebbian learning with
divisive normalization. [18, 19, 20, 23]
2.3.1 Architecture of the LISSOM Model of V1
For the purpose of experiments conducted in this project, basic LISSOM model of V1
will be used. Figure 2.6 shows the architecture of basic LISSOM model which
consists of one retinal receptors, LGN ON/OFF channels, and V1. The retina is
represented as RxR array of photoreceptor cells that receive grayscale inputs. LGN
ON and LGN OFF consist of LxL array of neurons corresponding to ON/OFF
processing in retinal ganglion cells and the LGN. In real biological visual pathway,
LGN ON and OFF are not always physically separated, however they are divided into
two separate sheets for simplicity. Each LGN unit receives input from a local
anatomical receptive field in the retina so the topographic organization of the LGN is
the same as the retina. V1 is NxN array of cortical neurons that receive afferent input
from LGN ON/OFF and also have reciprocal excitatory and inhibitory lateral
connections with other neurons. [19]
19
Figure 2.6. Architecture of basic LISSOM model. The model is a hierarchy of two dimensional array of
neural units, starting from the retinal receptors at the bottom, two sheets that represent LGN ON and
OFF channels, and finally a single sheet representing V1.
The LGN ON/OFF neurons receive input from retina through afferent connections.
The strength of this afferent connections are not adapted through any learning, instead
they are set permanently according to the standard difference-of-Gaussian (DoG)
model of LGN receptive fields. The weights for the ON-center units are computed by
equation:
L xy , ab
⎛ ( x − xc )2 + ( y − yc )2 ⎞
exp ⎜ −
⎟
σ c2
⎝
⎠ −
=
⎛ (u − xc )2 + ( v − yc )2 ⎞
⎟
∑ uv exp ⎜ −
σ c2
⎝
⎠
⎛ ( x − xc )2 + ( y − yc )2 ⎞
exp ⎜ −
⎟
σ s2
⎝
⎠
⎛ (u − xc )2 + ( v − yc )2 ⎞
⎟
∑ uv exp ⎜ −
σ s2
⎝
⎠
(2.9)
where Lxy,ab is the weight from retinal receptor (x,y) in the receptive field of an ONcenter neuron (a,b) with center (xc,yc). σ c defines the width of the central Gaussian
and σ s the width of the surround Gaussian. The OFF-center weights are the negative
of the ON-center weights. Using this weights setting, the ON-center of LGN units will
respond to light areas surrounded by dark, and the OFF-center units to dark areas
surrounded by light [19]. After receiving input from retina, LGN units compute their
response as a squashed weighted sum of the total received activation:
⎛
⎞
ξab = σ ⎜ γ L ∑ χ xy Lxy ,ab ⎟
⎜
⎟
⎝
xy
⎠
(2.10)
where ξab is the response of LGN ON/OFF unit (a,b) , χ xy is the activation of retinal
unit (x,y) in the receptive field of (a,b), Lxy,ab is the afferent weight from (x,y) to (a,b),
20
and γ L is a constant scaling factor. σ is a piecewise-linear squashing function (figure
2.7) :
0
s ≤ θl
⎧
⎪
σ ( s ) = ⎨( s − θl ) /(θu − θl ) θl < s < θu
⎪
1
s ≥ θu
⎩
(2.11)
Figure 2.7. Piecewise-linear squashing function that approximates the sigmoid activation function.
V1 units are connected to the LGN ON/OFF units by excitatory afferent connections.
Besides afferent connections, V1 neurons also have short range lateral excitatory
connections and long range lateral inhibitory connections to other neurons. All of
these connections are initialized with normalized random value in the beginning and
then modified by learning from the inputs presented to the network.
The initial response of V1 neurons is computed by taking weighted sum of activations
received from LGN and passing it through sigmoid function:
⎛
⎞
sij = γ A ⎜ ∑ ξab Aab,ij + ∑ ξab Aab,ij ⎟
ab∈OFF
⎝ ab∈ON
⎠
(2.12)
ηij (0) = σ ( sij )
where η ij (0) is initial response of V1 neuron (i,j), sij is afferent activation of V1
neuron (i,j), ξab is the activation of LGN ON/OFF neuron (a,b) in the receptive field
of V1 neuron (i,j), Aab,ij is the afferent weight, and γ A is a constant scaling factor.
After receiving afferent input, V1 neurons propagate the activations through the
lateral excitatory and inhibitory connections for several time steps until the activity
settles down. At each time step, the neuron combines the afferent activation with
lateral excitation and inhibition:
21
⎛
⎞
ηij (t ) = σ ⎜ sij + γ E ∑ηkl (t − 1) Ekl ,ij − γ I ∑ηkl (t − 1) I kl ,ij ⎟
⎝
kl
kl
⎠
(2.13)
where ηkl (t − 1) is the activity of another V1 neuron (k,l) in previous time step, Ekl,ij is
the excitatory lateral connection weight connecting neuron (i,j) and (k,l), Ikl,ij is the
inhibitory connection weight, γ E and γ I are scaling factor that determine the strength
of excitatory and inhibitory lateral interactions. The initial activity of V1 starts out
diffuse but slowly converges into a small number of stable focused patches of activity,
or activity bubbles.
After the activity settles, the afferent and lateral connection weights of V1 neurons are
modified according to Hebbian rule with divisive postsynaptic normalization:
w 'pq ,ij =
w pq ,ij + α X pqη ij
∑ uv ( wuv ,ij + α X uvηij )
(2.14)
where wpq,ij is the current connection weight from (p,q) to (i,j), w'pw,ij is the new
connection weight, α is the learning rate for each type of connection, Xpq is the
presynaptic activity after settling, and ηij is the activity of neuron (i,j) after settling.
After the map begins to organize, some lateral inhibitory connection weights become
very weak and they can be periodically removed to model connection death in biology,
resulting in patchy lateral connectivity similar to that observed in animal visual cortex.
Hebbian learning rule facilitates the cortical neurons to learn correlations in the input
patterns and develop representation of the correlation structures in the afferent and
lateral weights value. Divisive normalization is necessary for Hebbian rule to prevent
the weights value from increasing out of bound. One aims of this project is to replace
this normalization with more biologically plausible regulation based on homeostatic
synaptic scaling.
Several parameters like lateral excitation, sigmoid curve, and the learning parameters
are gradually adjusted during self-organization to enhance the resulting self-organized
map. This adjustment is currently done manually and it is hoped that some of this
manual adjustment could be replaced with homeostatic mechanism.
For the purpose of this project, the network will be trained with oriented Gaussian
pattern with random position and random orientation, using the following equation:
22
⎛ [( x − xc,k )cos(φ) − ( y − yc,k )sin(φ)]2 [( x − xc,k )sin(φ) + ( y − yc,k )cos(φ)]2 ⎞
−
⎟⎟ (2.15)
2
2
⎜
σ
σ
a
b
⎝
⎠
χxy = maxexp⎜ −
k
Where χ xy is the activity of retinal receptor (x,y), σ a and σ b determine the width
along the major and minor axes of the Gaussian, and φ is the orientation.
After being trained for several thousands iterations (usually between 10000 to 20000
iterations), the network develop an orientation map which is similar to the maps
observed in animal visual cortex (figure 2.8). This orientation map was obtained by
presenting the sine grating with various orientations and frequencies and then
measuring the activity of the neuron that most strongly respond to particular
orientation. If we look at the self-organized afferent weights, some neurons develop
Gabor-shaped RFs that are strongly selective to particular orientation and the some
others develop spherical RFs that are not orientation selective (figure 2.9.a). Lateral
inhibitory connections are patchy and connecting neurons with similar orientation
preference, consistent with neurobiological experimental result (figure 2.9.b).
Figure 2.8. Orientation preference map and orientation selectivity of LISSOM network trained with
oriented Gaussian pattern. Every neuron in the map is colored according to the orientation that it
prefers. Nearby neurons generally prefers similar orientations, forming groups called iso-orientation
patches. The map contains all the features found in real animal maps, such as pinwheel (marked by
white circle): points around which orientation changes continuously, linear zones (long rectangle):
straight lines along which orientations changes continuously, fractures (small square): sharp transition
between two very different orientation, and saddle point (marked with bowtie): a long patch where one
orientation is nearly bisected by another. Orientation selectivity measures how closely the input must
match the neuron’s preferred orientation for it to respond (coded by grayscale, with white means highly
selective, and black means low selecvitiy). (adapted from [19])
23
Figure 2.9. Self-organized afferent weights and lateral inhibitory weights of LISSOM network trained
with oriented Gaussian pattern (adapted from [19]).
2.3.2 Threshold Adaptation in LISSOM
In LISSOM model, the global activation threshold should be adapted gradually to
enable the development of smooth orientation map and increasing selectivity of the
neurons to particular orientation. In the beginning, the threshold is set within a certain
range so that all the neurons can easily response to the given input pattern and
Hebbian learning can associate the activity of input and output to change the synaptic
weights accordingly. During the course of self-organization, the activation threshold
should be gradually increased so that the neurons become harder to fire and retain its
selectivity to certain orientation that was previously learned. If the threshold is kept
unmoving, later presentations of input pattern will disturb the learned weights value
so the neurons cannot learn the association with specific orientation and form good
receptive field.
With the intention to observe the effect of self-organization without threshold
adaptation, the simplified LISSOM model (with one retinal sheet and one V1 layer)
was trained with fixed threshold setting. After 20000 iterations, we get the following
orientation map and afferent weights.
24
Figure 2.10. orientation map and afferent weights of LISSOM without threshold adaptation.
Figure 2.11. Average firing rates of several neurons of LISSOM without threshold adaptation.
It can be seen clearly that the network failed to organize properly. The orientation
map looks totally unordered and the receptive fields of most neurons are not selective
to particular orientation. If we plot the average firing rate of several neurons, we also
notice that the neurons tend to increase the activity level in the beginning and leveled
up after about 5000 iterations. This happened because with fix threshold setting the
neuron could not limit the increase in excitability due to Hebbian learning, so instead
of becoming more selective to particular orientation, it will end up preferring several
orientations at once.
The preceding example illustrates the importance of adjusting the neuron’s
excitability in the development of orientation map. In typical LISSOM models, the
threshold is changed several times by manually setting upper and lower bound values
at certain predefined iterations. This process is usually done by trial and error, based
on previous experience of the modeler. Figure 2.12 shows the plot of threshold
settings that are usually used in typical LISSOM models.
25
Figure 2.12. The piecewise-linear activation function during several iterations of learning.
2.3.3 LISSOM without Shrinking Lateral Excitatory Radius
LISSOM models have been very successful in simulating the self-organization of
orientation map in primary visual cortex. However, there are still some discrepancies
when we compare the self-organization process in LISSOM and the development of
orientation map in real animal visual cortex. Measurements in ferret visual cortex
during development, reveals that the structure of the orientation map is already
established in the early stage of development and remains stable throughout selforganization [9]. On the other hand, in LISSOM model, the structure of the
orientation map is repeatedly changing during the first several thousands iterations,
and the orientation column spacing also shrinks dramatically as the map develops
(figure 2.13). Reduction of orientation column spacing is not observed in the
development of animal orientation map.
100 iterations
200 iterations
500 iterations
1000 iterations
2000 iterations
4000 iterations
6000 iterations
20000 iterations
Figure 2.13 Self-organization of orientation map in LISSOM model. It can be observed that the map
changes its structure several times during the first 4000 iterations and column spacing shrinks
considerably in the first 1000 iterations.
26
The main cause of this problem was speculated to be the adjustment of lateral
excitatory radius and reduction of learning rate in the beginning of self-organization.
Initial attempts have been made to solve this problem, using LISSOM model with
fixed lateral excitatory radius and faster learning rate reduction. The orientation map
can still develop using this method and the map formation is smoother, more stable,
and closer to biology (figure 2.14). However, new problem arise with this approach.
Some neurons fail to organize properly and their afferent weights still look random
(figure 2.15). This might happen because the learning rate was reduced very fast in
the beginning and these neurons did not receive enough activation while the learning
rate was still high. They also miss the opportunity of learning during later iterations
because the activation threshold was already too high, reducing their excitability.
It is hoped that homeostatic plasticity mechanism might helps in this case by
regulating individual neurons excitability depending on the activations they receive.
So the neurons that fail to learn in the beginning can still have an opportunity to learn
later because they still retain their high excitability.
Figure 2.14. Orientation map obtained from LISSOM model without shrinking lateral excitatory radius
and faster learning rate reduction. White circles indicate areas where groups of neurons fail to organize
properly.
27
Figure 2.15. Afferent RFs and retinotopic mapping of LISSOM model without shrinking lateral
excitatory radius. It can be noticed that RFs of some neurons are still random looking and the
retinotopic mapping clearly shows several areas with discontinuities.
28
Chapter 3
Method & Parameters
This chapter describes the method and parameters of various experiments that are
conducted to investigate the behavior of LISSOM models with homeostatic plasticity
mechanism. It is hoped that with inclusion of homeostatic adaptation several aspects
of LISSOM based models can be improved and makes it more biologically plausible.
Two models of homeostatic plasticity, intrinsic plasticity model of Triesch [27, 28]
and homeostatic synaptic scaling model of Sullivan & de Sa will be implemented and
applied to various LISSOM models of visual cortex self-organization. The result of
these experiments will be presented in the next chapter.
3.1 Choice of the Models
In current computational neuroscience literature, there are only few proposed models
of homeostatic plasticity mechanism that are appropriate for rate-based neural models.
Most computational models of homeostatic plasticity explain the phenomena in higher
level of details involving Hodgkin-Huxley models or spiking neuron models.
Although there is a variant of LISSOM model that is based on spiking neurons
(SLISSOM [10]) which was primarily developed to investigate segmentation and
binding, most LISSOM based models are still using rate-based neurons, mainly for
speed and simplicity.
Several models of homeostatic plasticity mechanism might be applicable to rate-based
LISSOM models. These models are Triesch intrinsic plasticity [27, 28], Sullivan and
de Sa homeostatic synaptic scaling [25, 26], variants of BCM rules, and threshold
scaling using plastic facilitation function [15, 16, 33]. It is very tempting to try
implementing and applying all these different models into LISSOM and analyzes its
effect. However, given limited time, I only choose the first two models to be
implemented and analyzed in details. The main reason behind this choice is because
these two models have been proven to be working reasonably well in SOM based
models (which are precursor of LISSOM) [6, 7, 25], so there is higher possibility that
these models will also work well with LISSOM. Moreover, these two models
represent two primary aspects of homeostatic plasticity that are currently
hypothesized, which are intrinsic plasticity [36] and synaptic scaling [29, 30].
Intrinsic plasticity mechanism might be a promising alternative to replace manual
29
threshold adaptation in LISSOM models and homeostatic synaptic scaling might be
an alternative to replace divisive normalization in Hebbian learning.
3.2 Experiment with Intrinsic Plasticity Applied to LISSOM Models
This experiment is conducted to study and analyze the effect of applying intrinsic
plasticity mechanism proposed by Triesch [27] (described in section 2.2.2) to the
LISSOM models. It is hoped that implementation of intrinsic plasticity can eliminate
the need of manual threshold adjustment and improve several aspect of LISSOM
models.
Four different experiments are conducted to test this homeostatic mechanism. The
first experiment is carried out to study the basic behavior of simplified LISSOM
model with intrinsic plasticity in quite details. Several features of self-organized
network and changes during self-organization will be carefully examined. This first
experiment is also intended to find out whether LISSOM with intrinsic plasticity can
perform as well as (or better than) the one with manual threshold adaptation. In the
second experiment, sensitivity of intrinsic plasticity parameters will be examined by
running several simulations with different parameter settings and analyze the result of
self-organized network. The third experiment is devised to test intrinsic plasticity with
more complete LISSOM network with LGN ON/OFF channels. In the forth
experiment, intrinsic plasticity will be applied to LISSOM network without shrinking
lateral excitatory boundary, which are thought to be more realistic in simulating selforganizing process during the development of visual cortex. The following
subsections explain the methods and parameter settings for all these experiments.
3.2.1 Intrinsic Plasticity in Simplified LISSOM Network
A simplified LISSOM model with one retinal sheet and one V1 layer will be used in
this experiment (figure 3.1). The retina density is 24x24 and the V1 density is 84x84
neurons. Afferent connections from retina to V1 and lateral excitatory/inhibitory
connections of V1 neurons are all adapted through Hebbian learning. Divisive
normalization is used to constraint the growth of the weights value. In later
experiment, homeostatic weight normalization will be tried together with intrinsic
plasticity to make the model closer to biology. The network is trained using elongated
Gaussian spot (figure 3.2) with random position and random orientation, presented to
the retina one pattern at each time step.
30
Figure 3.1. Simplified LISSOM model with one retina sheet and V1 layer.
Figure 3.2. Example of elongated Gaussian spot presented to the retina as training pattern.
For analysis and comparison, two slightly different LISSOM networks are
implemented in Topographica. The first network is LISSOM with manual threshold
adaptation and the second one is LISSOM with automatic threshold adaptation based
on intrinsic plasticity model of Triesch [27, 28]. Parameters setting (such as learning
rate, input patterns, and adaptation of lateral connections) for both models are
identical except their activation functions.
The first network is the typical LISSOM implementation that uses piecewise linear
approximation of sigmoid activation function to control the gain and implement
saturation behavior of V1 neurons (figure 2.7; equation 2.11). There is only one
globally shared threshold function for every V1 neuron. The upper and lower bound
of this threshold is adjusted manually several times during self-organization to ensure
smooth global organization and well-tuned receptive field. Using single activation
function for every neuron might not be very realistic biologically but it is practically
effective and simpler to implement.
The second network is LISSOM with intrinsic plasticity that maintain individual
sigmoid activation function (equation 2.5) for each V1 neuron. These sigmoid
functions are adapted automatically based on recent activity of the neurons using
gradient descent algorithm to maximize entropy / information transfer [27]. Using
nonlinear sigmoid functions for individual neurons instead of one global linear
activation function might impact the speed of simulation since exponentiation is
31
slower than division. The gradient rule that update the sigmoid threshold also involves
quite complex arithmetic operations that might consume additional computation time.
However, based on recent evidence of homeostatic regulation in neuron’s intrinsic
plasticity, keeping track separate activation function for each neuron is more
biologically realistic and might also give additional benefits to the whole process of
self-organization.
Extra parameters are also needed for this gradient rule (equation 2.8). Separate
experiments have been carried out (by trying different parameter settings
systematically) to find out approximately the best value of these parameters for this
particular model. The parameter settings for this experiment are: initial sigmoid
threshold (a=13, b=-4), homeostatic learning rate ( η =0.0002), target average firing
rate of individual neuron ( µ =0.01). The initial sigmoid threshold is chosen such that
initially V1 neurons will respond quite strongly to the input pattern and enable rapid
learning in the beginning. The position and slope of initial sigmoid threshold is
similar with initial threshold setting of the piecewise linear in the first model. The
homeostatic learning rate is set to be much slower than Hebbian learning, consistent
with biological observation. Although the exact speed of homeostatic adaptation is
still unknown, this value works quite well. Target average firing rate is set based on
measurement of the average firing rate of the first model after the network stabilized.
Sensitivity of the model to these homeostatic parameters is explored in the next
experiment.
After running the simulation for both networks until reaching convergence (about
20000 iterations), several aspects of them can be analyzed by observing some
properties that emerge after self-organization such as the orientation maps, shape of
receptive fields, retinotopic mapping, and lateral inhibitory connections. The
similarities and differences of these properties between the two models will be
discussed and compared.
Another aspects of the models which are more numerical such as average firing rate
of individual neuron, distribution of firing rate, activity pattern of individual neuron,
and the speed of threshold adaptation will be studied by plotting time-series graphs
and analyzing its pattern. The average firing rate will not be computed using all
previous activities of the neuron; instead only recent activities of the neurons are used
to calculate the running average, using the following equation:
Yi t = α yit + (1 − α )Yi t −1
32
Yi t is the running average of neuron i at time t; yit is the firing rate of neuron i at time t,
and α determines the speed over which previous activity of the neuron is averaged.
For all experiments in this report, α = 0.0003 is used. This equation is used because
homeostatic adaptation is hypothesized to be related to recent past activities of the
neuron. So instead of averaging all activities from the beginning to the end, the
average is computed by taking into account only recent activities in specific time
window.
3.2.2 Sensitivity of Intrinsic Plasticity Parameter Settings
The second experiment is performed to test the effect of choosing different parameter
values for intrinsic plasticity model to the learned self-organized LISSOM network.
The simplified LISSOM model with intrinsic plasticity threshold adaptation that is
described in the first experiment will be used again in this experiment. There are three
different parameters that control the behavior of intrinsic plasticity mechanism
(equation 2.8) : the initial position and slope of sigmoid function (a and b), the
homeostatic adaptation rate ( η ), and the desired average firing rate ( µ ). As a
baseline for comparison, the values in the previous experiment will be used (a=13,
b=-4, η =0.0002, µ =0.01). Separate experiment has demonstrated that using this
parameter setting for intrinsic plasticity can produce good self-organized LISSOM
map.
At first, using fixed homeostatic adaptation rate ( η =0.0002) and fixed target average
firing rates ( µ =0.01), several simulations will be carried out using different initial
position and slope of sigmoid activation function, and then the result of self-organized
map and all its properties will be observed. Secondly, the initial sigmoid curve and
target average firing rates are held fixed (a=13, b=-4, µ =0.01), but homeostatic
adaptation rate will be varied. Finally, using fixed initial sigmoid curve and
homeostatic adaptation rate, several values of target average firing rates will be tried.
Table 3.1 lists all parameter values that will be tried in the simulations.
simulation #
a
b
η
µ
1
15
-3
0.0002
0.01
2
18
-3
0.0002
0.01
3
10
-4
0.0002
0.01
4
13
-6
0.0002
0.01
33
5
13
-4
0.0001
0.01
6
13
-4
0.0004
0.01
7
13
-4
0.001
0.01
8
13
-4
0.0002
0.005
9
13
-4
0.0002
0.02
Table 3.1. Different intrinsic plasticity parameter settings what will be used in simulation.
3.2.3 Intrinsic Plasticity Applied to LISSOM with LGN ON/OFF Channel
In the third experiment, intrinsic plasticity will be applied to more complete LISSOM
model with LGN ON/OFF layers in-between retina and V1, as shown in figure 3.3.
Intrinsic plasticity is applied only to V1 neurons, not to LGN neurons. In reality,
perhaps all neurons, including retina and LGN neurons are regulated by some kind of
homeostatic mechanism, but this fact is ignored in this experiment for simplicity.
After being trained for about 20000 iterations using oriented Gaussian patterns, the
result of the network with homeostatic mechanism will be analyzed and compared
with result of the network with manual threshold adaptation.
Figure 3.3. LISSOM model with LGN ON and LGN OFF sheets
Parameter setting for this experiment is similar with the first experiment. The size of
retina sheet and LGN On/Off is 24x24, and the size of V1 is 84x84 neurons. Using
higher density for V1 will allow better visualization of the orientation map. Values for
intrinsic plasticity parameters are exactly the same with the first experiment (a=13,
b=-4, η =0.0002, µ =0.01). It is expected that using the same parameter values the
network can still organize properly.
34
3.2.4 Intrinsic Plasticity Applied to LISSOM without Shrinking Lateral
Excitatory Radius
LISSOM models have been quite successful in simulating the self-organization of
visual cortex map. However, as described in section 2.3.4, the detailed development
of orientation map in this model is rather different from development of orientation
map in animal. The difference was thought to be caused by shrinking lateral
excitatory radius in the beginning of self-organization. Initial experiments have been
conducted to simulate LISSOM with fixed lateral excitatory radius. Although the
orientation map can still form, there are some neurons that fail to learn because they
didn’t get enough activation in the beginning and the global activation threshold
become too high afterwards. In this experiment, intrinsic plasticity will be applied to
simplified LISSOM model without shrinking lateral excitatory radius. It is hoped that
with intrinsic plasticity the process of orientation map formation will be more similar
to animal map development. The parameters for this simulation (in Topographica) are
given in table below.
Parameter
Value
Retina density
24 x 24
V1 density
84 x 84
Afferent learning rate
Initial value: 0.9590; after 100 iterations: 0.548;
after 1000 iterations: 0.274; after 3000 iterations: 0.137
Lateral excitatory radius
0.03
Lateral excitatory learning rate
0
Lateral inhibitory radius
0.244
Lateral inhibitory learning rate
1.7871
Input pattern
Oriented gaussian: scale=1.0, size=0.0936, aspect_ratio=4.0
Intrinsic plasticity learning rate
0.002
Initial sigmoid position
a = 13, b = -4
Intrinsic plasticity target average
firing rate
0.01
Table 3.2. Parameter for simulation of LISSOM with intrinsic plasticity and no shrinking excitatory
radius
3.3 Experiment with Homeostatic Synaptic Scaling Applied to
LISSOM Models
In this experiment, homeostatic synaptic scaling model of Sullivan and de Sa [25, 26]
will be implemented and applied to LISSOM model. This synaptic scaling mechanism
will replace the divisive normalization that is used in Hebbian learning to constraint
35
the growth of synaptic weights. It is hoped that with homeostatic synaptic scaling the
network can still organize properly and good orientation map can be developed.
The simplified LISSOM network that was described in section 3.2.1 will be used
again in this experiment. For initial experiment, the simulation will not use any
threshold function. Only simple bounding function to constraint the firing rate within
0.0 (minimum value) to 1.0 (maximum value) will be applied, similar to what has
been done in [26]. If this approach cannot work properly, manually adjusted
piecewise-linear threshold will be tried to regulate the neurons activation.
In the last experiment, homeostatic synaptic scaling will be applied together with
intrinsic plasticity mechanism to the LISSOM network. It will be observed whether
the network can still organize well in this case. The network will be trained for several
thousands iterations, and then several properties like orientation map and receptive
fields will be observed and compared with the standard LISSOM network. For
observing the stability of this algorithm, the average firing rate and the sum of
weights value of one projection will be plotted in time-series graph.
The parameters for LISSOM is exactly the same with the first experiment. According
to equation 2.3 and 2.4, the synaptic scaling itself needs three additional parameters,
which are β N (homeostatic learning rate), β c (control the speed of averaging the
neurons firing rate), and Atarget (desired target average firing rate). Several different
values will be tried for these parameters and the resulting self-organized network will
be evaluated. Further analysis such as sensitivity of these parameters will be carried
out if the basic result of this algorithm on LISSOM network is demonstrated to be
good enough.
36
Chapter 4
Results
4.1 Intrinsic Plasticity Applied to Simplified LISSOM Network
This section presents the result of experiment that was described in section 3.2.1. The
experiment was conducted to study the effect of applying intrinsic plasticity
mechanism to the simplified LISSOM model. Two networks are compared in this
experiment, one is LISSOM with manual activation threshold adjustment and the
other is LISSOM with intrinsic plasticity.
4.1.1 Receptive Fields and Orientation Maps
In this section we will compare the orientation map and receptive fields for LISSOM
model with manual threshold adjustment and model with intrinsic plasticity. After
being trained for 20000 iterations, several properties of the network can be visualized
graphically in Topographica. One of this property is orientation preference map. This
map is computed by presenting the sine grating with different orientations and phases
to the network and measuring the activity of V1 neurons to this input. Each neuron
will be assigned particular color that represent particular orientation that activate it
most strongly.
It can be observed from figure 4.1 that the network with intrinsic plasticity can
develop properly organized orientation map with visual appearance and features (such
as pinwheel, linear zones, saddle point, and fractures) similar to the original model
using manual threshold adaptation. The map with intrinsic plasticity also has larger
column spacing and looks more similar to real orientation map found in animal visual
cortex [5]. Orientation selectivity also improves considerably with intrinsic plasticity.
Most of the neurons are highly selective to specific orientation with some exception
on the neurons in the upper-right and bottom-right corners.
37
(a) Orientation map of LISSOM with manual threshold adjustment
(b) Orientation map of LISSOM with intrinsic plasticity
Figure 4.1. This figure shows the orientation map and selectivity of LISSOM with (a) manual threshold
adaptation and (b) with intrinsic plasticity, after 20000 training iterations. Orientation preference map
show the preference of individual neuron to particular orientation. Each orientation is represented by
different color coding. Orientation selectivity map shows the degree of selectivity of specific neuron to
particular orientation. The grayscale from black to white represent increasing selectivity (i.e. black: non
selective, white: highly selective). Orientation preference & Selectivity is combination of orientation
preference map and orientation selectivity. Certain features that exist in typical orientation map also
appear in map with intrinsic plasticity, such as linear zones (long rectangles), fractures (square),
pinwheel (circle), and saddle point (marked with bowtie).
With manual threshold adaptation, almost every neuron in the corners fails to
organize properly therefore we saw random noise on the orientation preference map in
this area (most noticeable on top-right and bottom-left corner of figure 4.1.a). This
happens because these corner neurons receive less activations in the beginning, so
while they are still learning an association with some oriented pattern in later
iterations, the activation threshold might be already too high so they cannot learn
much and ends up to be selective to some random orientation.
38
If we observe both orientation maps closely, it is apparent that there are some
discontinuities around the borderline. This happens because the oriented Gaussians
spot that are used as input pattern did not cover the neurons around the borders as
much as neurons in the middle area. Therefore the neurons around the border receive
less training input than other neurons. This unpleasant artifact can be easily removed
by extending the coverage area of the oriented Gaussian input (i.e. by increasing the
upper and lower bound of random number generator for x & y position of Gaussian
input pattern from +/- 0.5 to +/- 0.65 in Topographica). However, in this experiment,
this fact might also help to discern the benefit of using intrinsic plasticity. In the
orientation and selectivity map of the model using manual threshold adaptation, we
can observe some neurons around the border (especially neurons in the corners) that
are completely unselective and did not organize properly, whereas in the model using
intrinsic plasticity, these neurons are still selective even though they receive very few
inputs. Using homeostatic mechanism, when the neurons receive less input, their
intrinsic excitability will be lowered to achieve homeostatic average firing rate so it
can still operate effectively. This is beneficial because there will be less neurons that
are useless and die out due to receiving less input during the development. In the next
section this observation will be verified more closely by plotting the graph of average
firing rate and threshold value of these neurons.
(a) RFs and lateral inhibitory weights of LISSOM with manual threshold adjustment
39
(b) RFs and lateral inhibitory weights of LISSOM with intrinsic plasticity
Figure 4.2. This figure shows the afferent RFs (left) and lateral inhibitory connections (right) of
LISSOM with manual threshold adaptation and with intrinsic plasticity, after 20000 training iterations.
The afferent RFs show the pattern of synaptic weights that connect V1 neurons to retina neurons. Each
little square represent synaptic weights of one V1 neuron. The weight value is coded by grayscale from
black (zero value) to white (high value). The lateral inhibitory plots show the pattern of lateral
inhibitory connections between V1 neurons. Each small square represent lateral connection weights of
one V1 neuron to its neighbor’s neurons (represented as single pixel in that square). The colors
represent orientation preference of the neurons that are laterally connected to the current neuron. The
intensity/brightness of the color represents the strength of connection (higher intensity means higher
strength).
The above figure compares the afferent weights of V1 neurons in the network with
manual threshold adaptation and network with intrinsic plasticity. Qualitatively, the
afferent receptive fields of both networks do not differ much. The final afferent RFs
of the network with intrinsic plasticity are Gabor-shaped and strongly selective to
particular orientation. Lateral connections of both networks are patchy and tend to
follow the RF shape, connecting neurons with similar orientation preference, similar
to result found in biology [22].
One obvious difference is that neurons with intrinsic plasticity have slightly broader
receptive field size and less unselective RFs. This might happen because with this
model of intrinsic plasticity, the neurons try to achieve a desired average firing rate
and exponential firing rate distribution at the same time. In order to achieve
exponential firing rate, the frequency of weak firing should be more often than strong
firing. With broader receptive field each neuron has more chance to be weakly
activated by patterns that are not aligned to its receptive field orientation.
40
(b)
(a)
Figure 4.3. This figure shows the retinotopic organization of the orientation map for LISSOM with (a)
manual threshold adaptation and (b) with intrinsic plasticity, after 20000 training iterations. This map is
computed by taking the center of gravity of the afferent weights of each V1 neurons and connecting it
with the center of gravity of neighboring neurons.
The above figure shows the retinotopic organization of the orientation map which is
computed by taking the center of gravity of the afferent weights of the neuron and
connecting it to the center of gravity neighboring neurons with lines. There are some
distortions in this mapping because orientation map represent both position and
orientation smoothly across the same surface. The retinotopic map of the network
with intrinsic plasticity (figure 4.3.b) covers the whole input space uniformly which is
good indication that the network has learned to represent all positions in the input
space properly. Moreover, there are fewer distortions in this map with intrinsic
plasticity which indicates that the network has managed to learn more accurate
mapping than the network without intrinsic plasticity.
The figure below shows the detailed self-organization process of LISSOM with
intrinsic plasticity and LISSOM without intrinsic plasticity. It is apparent that with
intrinsic plasticity the development of the map is more stable. The pattern of the
orientation map did not change much during self-organization.
41
Iterations
LISSOM With Intrinsic Plasticity
LISSOM Without Intrinsic Plasticity
100
200
300
500
1000
42
2000
4000
6000
10000
20000
Figure 4.4. Comparison of orientation map self-organization process between LISSOM with intrinsic
plasticity and LISSOM with manual threshold adaptation. It is apparent that with intrinsic plasticity the
development of the map is more stable and the structure of orientation map did not change much during
self-organization.
43
4.1.2 Average Firing Rate, Firing Rate Distribution, and Speed of
Threshold Adaptation
In order to gain better insight into the mechanism of intrinsic plasticity, the behavior
of individual neurons is analyzed by plotting the graph of the firing rates, average
firing rates, distribution of firing rates, and the speed homeostatic threshold adaptation.
Figure 4.5 show the average firing rates of several neurons on different position of V1
sheet during self-organization from 0 to 20000 iterations.
(a) average firing rates of several neurons in LISSOM with manual threshold adaptation.
(b) average firing rates of several neurons in LISSOM with intrinsic plasticity.
Figure 4.5 Average firing rates of neurons in V1 (Blue: neuron in the top-left corner position, Red:
neuron in the center of V1, Green & Cyan: neurons halfway between the corner and the center)
The average firing rates curves of the neurons in both networks follow similar trends.
Initially, the average firing rates is quite high (ranging from 0.04 to 0.10) because the
activation threshold is low in the beginning to enable the neurons to learn as much as
possible from the given inputs. As the network self-organize and learns better
associations with input patterns, slowly the activation threshold is increased so the
neurons becomes more selective to specific orientation and activated less often than
44
before. So gradually, the average firing rates also decrease until reaching stable value
around 0.01. This plot also shows that intrinsic plasticity can perform as good as
manual threshold adjustment in slowly regulating the average firing rate of the
neurons to reach desired value.
An interesting difference can be observed in the average firing rates curves of the
neuron in the top-left corner (in blue color). As described in section 4.1.1, this neuron
receives fewer inputs compared to other neurons so it also activated less. In the
network with manual threshold adjustment (figure 4.5.a), the average firing rate of
this neuron is slowly rising for the first 500 iterations (after it receives some input
patterns) but then it decays rapidly and silenced out after 5000 iterations. This
happens because the activation threshold is adjusted globally so the activity of all
neurons will be decreased over time with the same speed regardless of individual
neuron’s activity history. Therefore the neuron that receives less activation in the
beginning will be even harder to be activated later and finally it fails to learn
meaningful association with the inputs and sometimes even becomes completely
silenced. This situation can be improved using intrinsic plasticity for individual
neuron (figure 4.5.b, blue curve). The same neuron can now slowly increase its
activity (by decreasing its own sigmoid activation threshold) to achieve the target
average firing rate. After only about 500 iterations this neuron can reach reasonable
average firing rate (around 0.03) so it can learn correlations with input pattern more
properly.
(a) Firing activity of single neuron in network without intrinsic plasticity
(b) Firing activity of single neuron in network with intrinsic plasticity
Figure 4.6. Firing activity of single neuron (located in the center of V1) of LISSOM with manual
threshold adaptation (a), and LISSOM with intrinsic plasticity (b) during 20000 training iterations.
45
Figure 4.6 shows the plots of firing pattern for single neuron located in the center of
V1 for the network with manual threshold adjustment and the network with intrinsic
plasticity. It is interesting to find out that although both networks have similar average
firing rate curves (figure 4.5), the pattern of firing of individual neuron is
comparatively different. Without intrinsic plasticity (figure 4.6.a), the neuron’s firing
pattern is dense and most activities are dominated by higher firing rate level (between
0.5 – 1.0). In the beginning of self-organization (the first 5000 iterations), the neuron
is activated more often because the activation threshold is lower and learning rate is
high. After 5000 iterations, the activity become sparser and the peak activity level is
also slightly reduced. Different firing behavior is observed in the network with
intrinsic plasticity. The neuron’s activity is already sparse since the beginning, and
most activities are dominated by lower firing rate level (below 0.1).
The difference can be seen more clearly by plotting the histogram of output of single
neuron before and after passing through threshold function (figure 4.7). Before
passing through the threshold function, the neuron’s output is already exponentially
distributed. This might happen because oriented Gaussian is used as the training
pattern. With other training pattern such as natural images this statistics might be
different. After passing through piecewise-linear threshold function (figure 4.7.a,
right), the distribution of firing rates becomes almost flat with an exception of single
peak around 0, which means that every possible value of firing rate occurs equally
often. Using intrinsic plasticity, the neuron retains the exponential distribution of
firing rates (figure 4.7.b, right), with most activations dominated by small firing rate
activity (below 0.05, with an exception of small peak around 1.0). This shows that
model of intrinsic plasticity of Triesch [27] can work well on LISSOM model in
regulating the neurons activity to achieve sparse and exponentially distributed firing
rates.
46
(a) Histogram showing distribution of outputs from single V1 neuron before thresholding (left) and
after thresholding (right) for the network with manually adjusted piecewise-linear threshold.
(b) Histogram showing distribution of outputs from single neuron before thresholding (left) and after
thresholding (right) for the network with intrinsic plasticity.
Figure 4.7. The plots compare the histogram of output from single neuron before and after passing
through threshold function for the network with manual threshold adjustment (a) and network with
intrinsic plasticity (b). The histogram is computed using 100 bins using data collected during 20000
iterations.
Intrinsic plasticity adapts the position and gradient of the sigmoid threshold function
(variable a and b in equation 2.5) adaptively depending on recent past activity of the
neuron during self-organization process. The speed of threshold adaptation can be
observed by plotting the change of variable a and b during the course of learning
(figure 4.8). The value of a, which controls the gradient, is gradually decreasing
following linear path, which means that the gradient of sigmoid is slowly increasing.
The value of b, which controls the position of sigmoid, is decreasing rapidly in the
first 500 iterations. This signifies that the curve is translated to the right pretty quickly
in the beginning to reduce the activation of the neurons to sparser regime (figure 4.9).
47
Figure 4.8. The change of sigmoid threshold parameter (a and b) during the course of self-organization
(from 0 to 20000 iterations). Intrinsic plasticity mechanism adjust this value adaptively based on the
activity of the neurons. It can be observed that the value of a is linearly decreasing from 13 to 10 while
the value of b is decreasing rapidly in the first 500 iterations and then leveled up at -6.
Figure 4.9. Plot of sigmoid threshold adaptation during 20000 iterations. The curve shifts very rapidly
in the first 1000 iterations and then constantly increases the slope afterwards.
4.2. Sensitivity of Intrinsic Plasticity Parameter Settings
In order to simulate realistic self-organization process of visual cortex map, several
parameters in LISSOM network such as Hebbian learning rate and lateral connection
strengths should be tuned properly. With inclusion of intrinsic plasticity mechanism,
the number of parameters that should be specified is growing. Now we should
determine four additional parameters: the initial sigmoid curve (a, b), the homeostatic
learning rate ( η ), and the target average firing rate ( µ ).
This experiment was performed to observe the sensitivity of intrinsic plasticity
parameter settings to the resulting self-organized LISSOM map. Simplified LISSOM
network that was used in the last experiment was simulated several times with
different values of intrinsic plasticity parameters and after several thousands iterations,
the resulting map is observed.
48
4.2.1. Initial Threshold Values
Intrinsic plasticity mechanism adjusts the sigmoid activation function of individual
neuron automatically, based on the history of their past activities. The sigmoid
activation function (equation 2.5) is determined by two variables a and b. a controls
the slope of the curve and b controls its position. Initially, this value should be
specified by the modeler. Choosing different initial values for a and b might affect the
result of self-organization. Four different combinations of initial a and b values were
tested in this simulations (figure 4.10). All simulations use the same homeostatic
adaptation rate ( η =0.0002) and target average firing rate ( µ =0.01). The initial
threshold value that was used in previous experiment (a=13, b=-4) will be used as a
baseline for comparison.
Figure 4.10. The plot shows the initial sigmoid activation curves that are going to be tried in this
experiment. The blue curve is the sigmoid activation function that was used in the first experiment.
This function will be used as a reference point for comparison.
No.
Initial
Threshold
Results and Comments
1
a=15, b=-3
(red curve,
figure 4.10)
This initial threshold value is set to be lower than the baseline threshold.
Because the threshold is low, initially, V1 neurons will be highly active, but
intrinsic plasticity mechanism can still bring down the activation level at
proper time, so in the end the network can still develop good orientation map.
The figure below is the orientation map after 15000 iterations.
49
2
a=18, b=-3
(green curve,
figure 4.10)
The curve is located at the same position with the previous one but has
smaller gradient. The network failed to develop good orientation map using
this initial threshold because V1 activity is very high in the beginning and
homeostatic adaptation cannot bring down the neurons activity to the target
level (0.01) on time. After 20000 iterations, the neurons average firing rate is
still quite high (0.03). Perhaps better map can be obtained if using faster
homeostatic adaptation rate. The figure below shows the map after 20000
iterations.
3
a=10, b=-4
(black curve,
figure 4.10)
This threshold is located at the same position with the baseline threshold but
has considerably larger gradient. The network totally failed to develop good
orientation map using this initial threshold. During the training process, the
threshold did not adapt too much because V1 activity has already reach near
the target value. The value of a did not change at all, and b was only slightly
decreasing from -4 to -5.5 for the first 2000 iterations and then leveled up.
After 10000 iterations the following orientation map is obtained.
4
a=13, b=-6
(purple curve,
figure 4.10)
This curve has the same gradient with the baseline curve but positioned
slightly farther to the right. Using this threshold, initial activity of V1 was
very low, but slowly the homeostatic mechanism shifted the threshold to the
left and restored V1 activity to higher level so Hebbian learning can operate
more effectively. After 20000 iterations, good orientation map was developed.
50
An interesting difference to be noticed is the pattern of average firing rates of
V1 neurons during self-organization (the figure below this text). The average
firing rates is low in the beginning (around 0.0005 – 0.0008), almost constant
for the first 2000 iterations, then increasing quite rapidly for the next 6000
iterations. After 8000 iterations, it begins to decrease slowly and reaching the
stable level. We observe that this trend of average firing rates is completely
different from the experiment reported in section 4.1. In that experiment, the
average firing rates was very high in the beginning and then slowly
decreasing until reaching the target value.
The neurons are less excitable in the beginning, so intrinsic plasticity
mechanism will lower the activation threshold slowly. As learning progressed,
the neurons activity is gradually increasing and after 5000 iterations it reached
high firing rate level. Because the firing rate is larger than target value,
intrinsic plasticity mechanism will bring them down gradually to the desired
level of activity by increasing the slope of the sigmoid function.
The figure below shows the pattern of activity of single V1 neuron. Initially
the activity is low because the activation threshold is high, but gradually
intrinsic plasticity increase the neurons excitability so it can slowly restore its
activity.
51
4.2.2. Intrinsic Plasticity Learning Rate
The speed of intrinsic plasticity learning rate might also affect the self-organization
process of LISSOM models. In this section, different values of learning rate will be
tried using simplified LISSOM network that was used previously. All experiments
will use the same initial threshold value (a=13, b=-4) and same target average firing
rate ( µ =0.01). η =0.0002 will be used as a baseline for comparison. The intrinsic
plasticity learning rate ( η ) is held fixed throughout the self-organization.
No.
Intrinsic Plasticity
Learning Rate
Results and Comments
1
η =0.0001
The network can still develop good map using learning rate which is 2
times slower than the baseline value. The figure below shows the
orientation map after 10000 iterations.
Interestingly, although good orientation map can still form using slower
learning rate, the step-by-step formation process of the orientation map is
rather different than using faster learning rate. With slower learning rate,
the orientation map substantially reorganizes itself during early stages of
self-organization, just like the self-organization process of the network
with manual threshold tuning shown in section 4.1.3. The figures below
show several snapshots of the map during the first 4000 iterations:
Iterations
Orientation map and orientation selectivity
500
52
1000
2000
4000
2
η =0.00002
Using 10 times slower adaptation rate, orientation map can still form but
the quality is not very good. There are lots of fractures and discontinuities.
The figure below shows the orientation map after 20000 iterations.
3
η =0.0004
With 2 times faster adaptation rate, the network develop some patterns in
the orientation map but the patterns look a bit strange and unusual. The
following figure shows the orientation map after 15000 iterations.
53
4
η =0.001
With 5 times faster learning rate, the network totally failed to develop
good orientation map. The orientation selectivity is low and most RFs
become spherical-shaped. This happens because when intrinsic plasticity
is adapting too fast, the neuron’s excitability will be reduced very rapidly
in the beginning of self-organization. Hence, the neurons will have very
low activation and as a consequence they will not be able to learn properly
through subsequent Hebbian learning.
The following figure shows orientation map and afferent RFs after 10000
iterations.
4.2.3. Target Average Firing Rate
The following results shows the orientation map obtained using different value of
target average firing rate. All experiments will use the same initial threshold value
(a=13, b=-4) and same learning rate ( η =0.0002).
No.
Target Average
Firing Rate
Results and Comments
1
µ =0.005
With lower target average firing rate, the network fails to develop good
orientation map and the orientation selectivity also very low.
2
µ =0.02
With twice higher target average firing rate, the network also fails to
develop good orientation map.
54
4.3. LISSOM with LGN ON/OFF and Intrinsic Plasticity
Previously, several experiments using simplified LISSOM network have shown that
intrinsic plasticity mechanism can be applied successfully to regulate the excitability
of V1 neurons during self-organization. In order to test whether this mechanism can
still work in more complex network, a LISSOM network with LGN ON/OFF layer
will be used. The description of the network is given in section 3.2.3. After being
trained for 20000 iterations using oriented Gaussian pattern with random positions
and random orientations, the following results are obtained.
Figure 4.11 shows the final orientation map and orientation selectivity of the network
with manual threshold adaptation and network with intrinsic plasticity. Essentially,
both networks developed good orientation maps that are similar to real maps
measured in animal visual cortex. They contain structures such as linear zones,
pinwheel, fractures, and saddle point.
If we observe the detailed structure of the maps more closely, we can find several
distinct features between them. The orientation column spacing of the first map is
narrower than orientation column spacing of the second map. The transitions between
different orientations also appear more smoothly in the second map. Measurements of
orientation map in animal visual cortex also produce maps with wide column spacing
[5], so in certain sense, the second map is more similar to real animal maps than the
first one.
55
(a) Orientation map of the network with manual threshold adaptation
(b) Orientation map of the network with intrinsic plasticity
Figure 4.11. Orientation map and orientation selectivity of V1 neurons in LISSOM with LGN ON/OFF
layer, trained with oriented Gaussian input pattern for 20000 iterations. The upper figure is the result
using manually adjusted activation threshold and the lower figure is the result using intrinsic plasticity.
Features like pinwheel, linear zones, fractures, and saddle point can be found on both maps.
Another interesting difference can be observed when we plot the average firing rate of
V1 neurons for both networks (figure 4.12). The average firing rate of neurons with
manual threshold adjustment rapidly drops after about 4000 iterations and then
reached constant level at very low value (around 0.0001 – 0.0002) in 15000 iterations.
This might be caused by inappropriate threshold setting or the speed of threshold
adjustment was too fast. Although the network can still produce good orientation map,
the excitability of V1 neurons become very low, so when an input pattern is presented
to the trained network, the response of V1 neurons will be very weak (figure 4.13.a).
This problem does not happen when intrinsic plasticity is applied. The neurons can
still maintain stable homeostatic firing rate level and V1 response to the input pattern
is strong (figure 4.13.b). This might be very beneficial, especially when V1 is
connected to the network in higher layer such as V2 and V4, because the signal that
56
propagates from V1 will be strong enough to activate neurons in higher layers and the
information will not be lost in the midst of transmission.
(a) Average firing rate of neurons in V1 in LISSOM with manual threshold adaptation.
(b) Average firing rate of neurons in V1 in LISSOM with intrinsic plasticity.
Figure 4.12. This plot shows the average firing rate of several V1 neurons in LISSOM with LGN
ON/OFF for period of 20000 iterations.
(a) network with manual threshold adaptation
(b) network with intrinsic plasticity
Figure 4.13. Activity of retina, LGN Off, LGN On, and V1 when presented with oriented line pattern.
Both networks were previously trained for 20000 iterations with oriented Gaussian inputs. V1 activity
of the first network (a) is substantially weaker than V1 activity of the network with intrinsic plasticity
(b).
57
4.4 Intrinsic Plasticity Applied to LISSOM with no Shrinking Lateral
Excitatory Radius
Intrinsic plasticity was applied to LISSOM model with no shrinking lateral excitatory
radius (described in section 3.2.4). For this simulation, the intrinsic plasticity learning
rate was set 10 times faster ( η =0.002) than the learning rate used in previous
experiments because the Hebbian learning rate was reduced almost 10 times faster in
the beginning of self-organization. This indicates that the speed of intrinsic plasticity
adaptation is directly related to Hebbian learning rate adaptation.
It is apparent that after applying intrinsic plasticity, better orientation map and
selectivity is obtained and there is no more neuron that fails to learn (figure 4.14 &
4.15). The orientation map development is relatively stable (figure 4.16). In the first
1000 iterations, pattern of oriented patches emerges smoothly without any noticeable
jumps and the pattern does not change dramatically afterward. The patches that are
red/orange in the beginning are still mostly red/orange in the end. Some patches that
have green/blue color often change to other colors later. We can also observe that the
orientation column spacing shrinks a little bit from 2000 to 5000 iterations but not
significantly.
Figure 4.14. Final orientation map and orientation selectivity obtained from simulation of LISSOM
network with intrinsic plasticity and no shrinking excitatory radius (after 25000 iterations)
58
Figure 4.15. Afferent receptive fields and retinotopic map obtained from simulation of LISSOM
network with intrinsic plasticity and no shrinking excitatory radius (after 25000 iterations). It can be
observed that all neurons could develop ring-shaped or spherical-shaped RFs. Smooth retinotopic map
also indicate that the network has properly organized.
Iterations
Orientation Map
50
150
300
59
500
1000
2000
5000
10000
15000
60
20000
25000
Figure 4.16. Snapshots of orientation map and orientation selectivity taken at specific iterations during
self-organization of LISSOM network with intrinsic plasticity and no shrinking lateral excitatory radius.
4.5. Homeostatic Synaptic Scaling
This section presents the result of experiment that was described in section 3.3. In
these experiments, homeostatic synaptic scaling model proposed by Sullivan and de
Sa [25, 26] was implemented to constraint the growth of synaptic weights that
undergo changes through Hebbian learning in LISSOM network. The experiments
were divided intro three parts. In the first part, synaptic scaling was applied to
LISSOM network with fixed activation threshold, in the second part, it was applied to
network manual threshold adjustment, and in the last part it was tried in combination
with intrinsic plasticity.
With the aim to test whether synaptic scaling can works with fixed activation
threshold setting, several simulations were carried out using simplified LISSOM
network that uses fixed linear threshold (with lower bound=0.0 and upper bound=1.0).
Different combinations of homeostatic learning rate and target average firing rate
were tried, but all of them did not give satisfying result. Figure 4.17 shows an
example of orientation map and afferent weights pattern obtained from one of the
simulation. We can observe that there are still some regular structures in the
orientation map but the shape of the afferent RFs is completely disorganized. With the
speculation that perhaps homeostatic learning rate and target average firing rate
should be reduced gradually throughout the self-organization process (to match the
speed of Hebbian learning that is also reduced gradually), several additional
61
experiments were performed, but there is not much improvement either. This result
indicates that it is difficult to apply synaptic scaling into LISSOM if we use fixed
activation function.
Figure 4.17. Orientation map and afferent RFs of simplified LISSOM network with homeostatic
synaptic scaling. The network did not use any threshold function to regulate the neuron’s output. The
neuron’s activation function is replaced by simple bounding function to restrict the firing rate within
range 0.0 – 1.0. (synaptic scaling parameters:
β N =0.1, β C =0.003, target average firing rate=0.05)
In the second trial, more simulations were performed to test if the result can be
improved by manually adapting the activation threshold throughout self-organization.
Several different combinations of homeostatic learning rate ( β N ) and target average
firing rate (Atarget) were tried but the network always fail to develop good orientation
map. Figure 4.18 shows an example of the result obtained. The orientation map is
totally disordered and most of the RFs are spherical and unselective to any orientation.
If we observe the change of average firing rate during self-organization (figure 4.5),
we notice that it is not constant but slowly decreasing over time. With this insight,
several simulations were performed with gradually decreasing target average firing
rate (Atarget). The result using this approach is slightly better than the result of previous
experiment (figure 4.19). It can be observed that some structures begin to emerge in
the orientation map and several elongated shapes appeared in the afferent RFs.
Inspired by this slightly better result, another set of experiments were carried out
using slowly decreasing homeostatic learning rate ( β N ) together with decreasing
target average firing rate. It is reasonable to decrease homeostatic learning rate as selforganization progresses in order to match Hebbian learning rate which is also
gradually decreased. Several different scheduled adjustments of homeostatic learning
rate were tried without success. However, after numerous trials and errors (by trying
different learning rate adjustment that minimizes the oscillations of the weights and
average firing rates) better orientation map could be obtained (figure 4.20). The
structure of the map looks better now, however there are still lots of fractures and
62
discontinuities in some areas. The orientation selectivity also increased considerably,
and most neurons have elongated-shaped RFs. Although slightly better orientation
map could be obtained using this approach, it is very difficult to manually tune the
homeostatic learning rate in this way.
The stability of afferent weights value was also observed by plotting the absolute sum
of afferent weights value of single V1 neuron (figure 4.21). Initially, the weights
value oscillates quite strongly, alternating within range 0.5 to 2.0. Gradually, the
oscillations become weaker but the frequency did not change much. After 20000
iterations, the frequency of oscillation begins to drop because at this time the
homeostatic learning rate was reduced significantly.
Figure 4.18. Orientation map and afferent RFs of simplified LISSOM network with manual threshold
adjustment and homeostatic synaptic scaling after 13000 iterations. (using constant synaptic scaling
parameters:
β N =0.01, βC =0.005, target average firing rate=0.03)
Figure 4.19. Orientation map and afferent RFs of simplified LISSOM network with manual threshold
adjustment and homeostatic synaptic scaling after 20000 iterations. (synaptic scaling parameters:
β N =0.01, βC =0.01, Atarget=0.05 for the first 1000 iterations, after 1000 iterations Atarget=0.03, after
2500 iterations Atarget=0.01)
63
Figure 4.20. Orientation map and afferent RFs of simplified LISSOM network with manual threshold
adjustment and homeostatic synaptic scaling after 30000 iterations. (using gradually adjusted synaptic
scaling
parameters:
βC
=0.0085;
βN
iterations β N =0.0075, after 2000 iterations
iterations
=0.0085
for
the
first
500
iterations,
after
500
β N =0.0065, after 4000 iterations β N =0.005, after 20000
β N =0.003; Atarget=0.05 for the first 1000 iterations, after 1000 iterations Atarget=0.025, after
2000 iterations Atarget=0.02, after 4000 iterations Atarget=0.015, after 8000 iterations Atarget=0.01)
Figure 4.21. This plot shows the absolute sum of afferent weights value of one neuron (in the center of
V1) during 30000 iterations. The plot was obtained from the same simulation that produced the result
shown in figure 4.20.
It is interesting to find out whether LISSOM can still develop good orientation map if
homeostatic synaptic scaling is applied together with Triesch’s intrinsic plasticity
mechanism. After trying several different parameter settings, finally pretty good
looking orientation map could be obtained (figure 4.22). However, this map still looks
a bit strange compared to normal maps. There are lots of fractures and transition
between different orientations is not as smooth as ordinary maps. The orientation
selectivity is low and the shape of the afferent RFs also looks rather blurred. Figure
4.23 shows the stability of weight value during self-organization. It can be observed
that initially there are some large oscillations but after 5000 iterations the weights
become quite stable. Compared to previous experiment (figure 4.23) the weights
value is more stable in this case.
64
Although the result is not as good as expected, this experiment shows that orientation
selective RFs and structured orientation map could still emerge from LISSOM model
with combination of intrinsic plasticity and homeostatic synaptic scaling.
Figure 4.22. Orientation map and afferent RFs of simplified LISSOM network with intrinsic plasticity
and homeostatic synaptic scaling after 38000 iterations of training. Intrinsic plasticity parameters are: a
=
13,
b= -4, learning rate=0.0002, target average firing rate=0.01. Synaptic scaling parameters are adjusted
gradually:
β C =0.0085; β N =0.0085 for the first 500 iterations, after 500 iterations β N =0.0075, after
2000 iterations
20000 iterations
βC =0.0065,
after 4000 iterations
βC =0.005,
after 8000 iterations
β N =0.003,
after
βC =0.002; Atarget=0.04 for the first 1000 iterations, after 1000 iterations Atarget=0.035,
after 2000 iterations Atarget=0.03, after 3000 iterations Atarget=0.025, after 4000 iterations Atarget=0.020,
after 5000 iterations Atarget=0.015, after 6000 iterations Atarget=0.013, after 7000 iterations Atarget=0.011,
after 8000 iterations Atarget=0.01.
Figure 4.23. Plot of the absolute sum of weights value of the center V1 neuron during 38000 iterations.
65
Chapter 5
Discussion and Future Works
5.1. Discussion
5.1.1. Intrinsic Plasticity and LISSOM Models
Regularization of neuron’s excitability plays an important role in the self-organization
process of LISSOM models. In typical models, neuron’s excitability is adapted by
gradually adjusting the piecewise-linear threshold function during self-organization.
Initial investigation presented in section 4.1 indicates that intrinsic plasticity
mechanism can be successfully integrated into LISSOM model to replace the manual
threshold adjustment and regulate the neuron’s excitability more robustly. The plots
of V1 neurons activity and distribution of firing rate shown in figure 4.5 and 4.6 also
verify that the neurons with intrinsic plasticity can produce sparser firing activity and
achieve approximately exponential distribution of firing rate. This is consistent with
observations conducted in biological visual cortex when presented with natural scenes
[1].
The models with intrinsic plasticity can develop good orientation map that has smooth
transition between different orientations and contain features similar to maps found in
animal visual cortex (figure 4.1.b, 4.11.b, 4.14). Moreover, compared to the maps
with manual threshold tuning, increased selectivity was observed in the maps with
intrinsic plasticity. Orientation selectivity was increased primarily because each
neuron maintains its own separate activation function so it can regulate its own
excitability depending on the amount of input received. The neurons that receive less
input will have lower activation threshold and vice versa. On the other hand, the
neurons with globally tuned threshold function should regulate their excitability
uniformly no matter how much input they receive, therefore some of the neurons
might not be very selectivity because they didn’t have much chance to learn while the
threshold was low. The use of global threshold adaptation implies that all neurons
should learn and adapt at the same speed throughout the self-organization process.
This assumption might be too restrictive and limiting, because once the neurons lag
behind their counterpart in the beginning of learning stage, they will never recover
their learning ability and lost their plasticity forever. Intrinsic plasticity can increase
the neurons selectivity to single orientation because each neuron can now adapt their
66
activation threshold individually independent of the others. So the neurons which did
not have chance to learn in the beginning can still learn an association with input
pattern in later stage. This might be beneficial in terms of information maximization
because every neuron will be utilized up to its maximum capacity so it can still
contribute to the whole network [1, 24, 27]. Moreover, the ability to regulate
individual neuron’s excitability also proven to be effective in solving the problem
found in LISSOM with no shrinking excitatory radius described in section 4.4.
Another distinct feature of the maps with intrinsic plasticity compared to the old maps
is the size of orientation column spacing (the thickness of orientation-selective
patches). The maps with intrinsic plasticity tend to have wider orientation column
spacing (i.e. thicker orientation-selective patches). Measurements from animal visual
cortex also reveal similar maps with wide orientation column spacing [5].
Consequently, it can be said that the maps with intrinsic plasticity is more closely
resemble real animal maps than the map with manual threshold adjustment. The exact
reason of why the map with intrinsic plasticity could have wider column spacing is
remain to be investigated further.
Comparison of detailed step-by-step formation of the orientation map presented in
section 4.1.3 reveals another benefit of using intrinsic plasticity. For the first 500
iterations, there is not much noticeable difference in the map of both networks. Some
oriented column patterns begin to emerge on both maps and orientation selectivity
also increasing. During the next iterations, the map formation process differs greatly,
leading to different final orientation map. With intrinsic plasticity, the structure of the
orientation map is relatively stable and does not fluctuate greatly during selforganization. Large orientation-selective patches appear in the beginning 1000
iterations and then gradually shrink into smaller patches with slightly different
orientations. The orientation selectivity of the neurons is also increasing constantly
with the largest rise in the first 2000 iterations. Different pattern of self-organization
is observed in the network without intrinsic plasticity. During the first 5000 iterations,
the orientation map reorganizes several times into substantially different pattern
before it begins to settle down and form quite stable pattern.
This difference might be related to the speed and timing of threshold adaptation. As
can be seen in figure 4.7 and 4.8, the speed of threshold adaptation using intrinsic
plasticity is not linear. For the first 1000 iterations, the threshold was translated
rapidly to the right in order to counteract the rapid increase of activity due to high
Hebbian learning rate, and thus substantially reducing the neuron’s excitability. This
is reflected in the formation of orientation map for the first 1000 iterations, in which
67
large orientation-selective patches appear and orientation selectivity increases rapidly.
At this stage, most neurons have already learn an association with specific orientation,
and because the neuron’s excitability is not as high as before, subsequent associative
learning did not change the neuron’s orientation preference dramatically. Hence,
during the next iterations relatively smooth transitions can be observed in the
formation of the orientation map. In order to test whether using slower learning rate
would result in unstable transitions of the orientation map, simulations with several
times slower learning rate was performed. As reported in section 4.2.2, with slower
intrinsic plasticity learning rate, the network indeed went through substantial
reorganization during the first several thousands iterations, similar to the network with
manual threshold adjustment. This result shows that smooth transitions during selforganization did not miraculously appear just by applying intrinsic plasticity into
LISSOM network, but it also depends on careful choice of intrinsic plasticity learning
rate (and also initial threshold position).
Results reported in section 4.2.1 indicate that initial threshold setting of intrinsic
plasticity also affect the process of self-organization. If we set the initial threshold too
low or too high, the network might fail to organize properly. The experiments also
point out that the choice of initial gradient/slope of the sigmoid (determined by
variable a) greatly affects the resulting map more than the choice of initial position of
the sigmoid (variable b).
An interesting behavior can be observed in the fourth experiment reported in section
4.2.1. In this experiment, the initial threshold is positioned slightly farther to the right
(using b=-6 instead of b=-4, and the initial gradient value, a=13, is not changed). This
means that initially V1 neurons are less excitable and have very weak activity because
of high threshold value. Surprisingly, even with very weak initial V1 activity, the
network can still learn correlations in the input patterns and develop good orientation
map. Initially, the average firing rate of V1 neurons are very low (below 0.01, as can
be seen in the plot in section 4.2.1). During the first 2000 iterations, intrinsic plasticity
mechanism slowly restores the neurons excitability and at the same time the neurons
also learns quickly from input patterns because the Hebbian learning rate was high at
that moment. Gradually the activity becomes stronger and reached quite high level of
average firing rate (around 0.02) after 6000 iterations, before they slowly decreasing
again to reach the target value of 0.01. This pattern of neurons activity is different
with the normal pattern of activity in typical LISSOM model, in which the neurons
have very high activity in the beginning and gradually decreasing as self-organization
progresses. The relevance of this result to real biological development of cortical
neurons is not clear, because to our knowledge there is no definite literature that
68
report the exact pattern of neurons activity during the development, prenatally and
postnatally. On the other hand, this result indicates that with intrinsic plasticity
mechanism, self-organization of cortical neurons is not very sensitive to initial
neuron’s excitability.
Another intrinsic plasticity parameter that should be set correctly is the target average
firing rate. The result of simulations reported in section 4.2.3 shows that this
parameter affects the result of self-organization greatly. The network failed to develop
good map when the setting of this value was higher or lower than 0.01. This value
cannot be set arbitrarily because it is constrained by input activity. The statistics of
input pattern and normalization of synaptic weights performed after applying Hebbian
rule might also determine the possible range of average activity. During input
presentations, the oriented Gaussian input patterns with fixed intensity (scale) cover
the whole retina space uniformly, so the strength of overall activations coming from
retina to each V1 neuron is roughly the same. The afferent connection weights from
retina to each V1 neuron is adapted through Hebbian learning and normalized to 1.
Because of this normalization, the activation of V1 neurons (before passing through
threshold function) is constrained to be within certain range, proportional to the
strength of the inputs coming from retina. Therefore, when the activation threshold is
moved to increase or decrease the average activity of V1 neurons during the learning
process, the previously learned receptive field will also be disrupted.
To illustrate this point more clearly, suppose that we have a trained network that
already form a good orientation map and receptive fields. The average activity of V1
neurons in this network is 0.01. Then, we retrain this network using Hebbian learning
and intrinsic plasticity with twice higher target average activity (0.02) for several
iterations. Because the target average activity is higher than current average activity,
intrinsic plasticity mechanism will increase the neuron’s excitability by shifting the
sigmoid threshold to the left, so the neurons can have more activation. This also
means that the neurons can be easily activated by other oriented patterns different
from their RFs orientation. Hebbian learning is still operating here, so when the
neurons output activity increase, the connection weights also become strongly adapted
to new input patterns. The weights value is normalized to 1, so the contribution of
new pattern to the weights value is significant. In this way, some of previously
learned receptive fields will be blurs out forming spherical unselective RFs. Higher
average activity can be achieved but also destroying the map organization at the same
time. From this observation, we might speculate that in order to control the target
average activity arbitrarily, we should adjust both the activation threshold and the
weights normalization value. One possibility is to use homeostatic synaptic scaling
69
mechanism to replace the divisive normalization. However, because of limited time,
the sensitivity of the network with intrinsic plasticity and synaptic scaling to the value
of target average firing rate has not been tried.
This issue is also related to the problem of training the network using input with
different scale (intensity). Every time we train the network using input with different
scale, we should use different initial threshold and target average activity for the
intrinsic plasticity mechanism. Several attempts have been made to find an algorithm
which can automatically find the proper initial value of this parameter for any input
scales, however none of them is proven to be good enough.
Even though it is rather difficult to set the parameters of intrinsic plasticity so it can
work properly, other experiments reported in section 4.3 and 4.4 have shown that this
mechanism is really beneficial to improve several aspects of LISSOM models. Better
orientation map could be obtained when intrinsic plasticity was applied to LISSOM
with LGN ON/OFF channel (section 4.3). Moreover, in that experiment it has been
shown that even though the network with manual threshold adaptation could develop
good orientation map, the excitability of V1 neurons might become very low after
self-organization. This problem might happen because the threshold was not carefully
tuned. On the other hand, the network with intrinsic plasticity could maintain stable
firing rate level throughout self-organization process, so in the end, V1 neurons can
still respond strongly to given input pattern. Maintaining strong response and stable
average activity is important, especially when we consider transmitting the activations
to higher layers like V2 or V4. If the signal that propagates from V1 is strong enough
then it will not be lost in the midst of transmission to higher layers.
It has been observed in real biological experiment that the structure of the orientation
map in ferret primary visual cortex is remarkably stable during the development stage
from the earliest age until maturation. This suggests that orientation map in animals is
already established in very early stage of the development and remain stable over time
[9]. As described in section 2.3.4 and section 4.4, although the orientation map
produced by LISSOM model is similar with the map measured in animal, the detailed
self-organization process of the orientation map in LISSOM is quite different with the
development of the map in animals. In the beginning of self-organization, the
orientation map of LISSOM model repeatedly reorganizes itself and the orientation
column spacing shrinks dramatically as the map develops.
Initial attempts have been made to improve this by keeping the lateral excitatory
radius fixed and speed up the Hebbian learning rate reduction. Orientation map still
70
develop using this approach, however some neurons fail to learn and their afferent
weights are still random because they didn’t get enough activation in the beginning
and the global activation threshold become too high afterwards. Results presented in
section 4.4 shows that after applying intrinsic plasticity to this network, better
orientation map was obtained and there are no more dead neurons. This improvement
could be gained partly because intrinsic plasticity mechanism adjusts the activation
for each neuron separately. The development of the orientation map is also relatively
stable. Pattern of oriented patches appears smoothly without any noticeable jumps and
the pattern does not change dramatically afterward. This result is, to a certain extent,
looks similar to the development of the map in animal.
5.1.2. Synaptic Plasticity and LISSOM Models
The results reported in section 4.5 have shown that, to some extent, it is possible to
apply homeostatic synaptic scaling mechanism of Sullivan and de Sa [25, 26] to
replace the divisive normalization used in LISSOM model. Nevertheless, it is very
difficult to tune the parameters of the algorithm. If the homeostatic learning rate is too
high, the weights value will decrease very quickly. If the learning rate is too slow, the
weights value will become very large and the network becomes saturated. Pretty good
result could be obtained after gradually adjusting the learning rate and target average
firing rate during the course of self-organization. However, the resulting orientation
map still does not look as good as usual. The last experiment demonstrates that with
the proper parameter settings, it is possible to combine both intrinsic plasticity and
homeostatic synaptic scaling in LISSOM visual cortex model, even though the result
is not as good as when using intrinsic plasticity and divisive normalization.
5.2. Future Works
From the results discussed above it is clear that there are still lots of issues concerning
homeostatic plasticity in LISSOM models that needs further investigations. In this
project, only two computational models (intrinsic plasticity and homeostatic synaptic
scaling) that were implemented and applied to LISSOM model. There are other
models such as BCM [8, 34, 35] (including some of its variants) and STDP (Spike
Timing Dependent Plasticity) [17, 32] that can be implemented in the future. STDP
will need spiking neuron model, so in principle it should be possible to integrate this
mechanism into SLISSOM model (Spiking LISSOM) [10].
71
Intrinsic plasticity mechanism of Triesch [27] has been shown to be applicable to
replace manual threshold adjustment in LISSOM and could improve some aspect of
the model. This mechanism was only applied to V1 neurons, but in principle, intrinsic
plasticity can be also applied to other neurons including retinal receptors and LGN
neurons. It is possible that by using this kind of regulation on retina or LGN, the
ability of the network to deal with variability in input scale (intensity) can be
improved considerably. This ability is important, especially when using natural
images as training pattern. Currently, LISSOM model trained with natural images
does not develop the orientation map as good as the model trained with artificial
pattern. Better result should be possible if some kind of homeostatic regulation is
applied when training the model using natural images. The role of intrinsic plasticity
in other LISSOM models such as model with two retinal sheets, model of color,
ocular dominance, disparity, and direction selectivity also remains to be investigated.
It has been shown that orientation map self-organization in LISSOM model with
intrinsic plasticity and no shrinking lateral excitatory radius looks very similar to the
process of map development in animal visual cortex. This result was obtained when
the model was trained using oriented Gaussian. More realistic training pattern such as
combination of retinal waves and natural images (to simulate prenatal and postnatal
training pattern in animal) should be tried to produce more biologically convincing
result.
One difficulty of applying intrinsic plasticity to LISSOM model is choosing the right
parameter values. It might be possible to devise an algorithm that adapts some of
these parameters automatically based on other values that reflect the state of the
network. For example intrinsic plasticity learning rate could be adjusted depending on
how fast the network connections weights changes and target average activity could
be determined from the statistics of inputs coming to the neurons.
Initial experiments shows that it is possible to integrate homeostatic synaptic scaling
of Sullivan and de Sa [25] into LISSOM model. However, it is even more difficult to
tune the parameters so that the network could produce good orientation map. The
homeostatic learning rate and target average activity should be decreased gradually
during the course of self-organization. Currently, it was done manually by trial and
error. It might be possible in the future works to regulate these parameters
automatically by another learning process.
72
Chapter 6
Conclusions
The experiments conducted in this project have demonstrated that homeostatic
plasticity mechanism, particularly intrinsic plasticity, can be successfully integrated
into LISSOM model of visual cortex to make it more biologically realistic and also
improves some of its behavior. Two computational model of homeostatic plasticity,
intrinsic plasticity mechanism proposed by Triesch [27] and homeostatic synaptic
scaling proposed by Sullivan and de Sa [25, 26] have been implemented and applied
to basic LISSOM model. Several experiments were conducted to test the resulting
self-organized map and the behavior of the network under this homeostatic
mechanism.
The orientation map that is more closely resemble the map measured in animal visual
cortex was obtained after applying intrinsic plasticity mechanism into LISSOM model.
Moreover, this mechanism also proven to be effective in solving the problem of deadneurons that fails to organize properly in LISSOM model without shrinking lateral
excitatory radius. The use of intrinsic plasticity also eliminates the need of manually
tuning the activation threshold and therefore makes the model more reliable and more
biologically plausible. Quite extensive experiments have been performed to test the
sensitivity of intrinsic plasticity parameters setting to the resulting self-organized map.
It has been found that parameter settings affect the process of self-organization quite
significantly and it is rather difficult to find the optimal parameters that work well
with specific model. Moreover, in LISSOM model, certain values such as Hebbian
learning rate is not constant but gradually adjusted during self-organization. This
might indicate that perhaps intrinsic plasticity learning rate should also be adapted
regularly to balance the speed of Hebbian learning. It might be possible to devise
another kind of regularization rule that automatically adjust intrinsic plasticity
parameters based on current state of the network.
Initial experiments with homeostatic synaptic scaling in LISSOM model indicate that
it is very difficult to apply this algorithm directly because there are too many
fluctuations in the weights values during self-organization. The fluctuations can be
slightly reduced by adapting the homeostatic parameters gradually, but this process is
currently done manually and based on trial and error. In order to integrate it into
LISSOM more fully, some kind of automatic adjustment of the parameters should be
invented.
73
Bibliography
[1] Baddeley, R., et. al., Responses of Neurons in Primary and Inferior Temporal
Visual Cortices to Natural Scenes. Proc. Roy. Soc. of London. B, 264:1775-1783,
1998.
[2] Bednar J. A., Choe Y., De Paula J., Miikkulainen R., Provost J., and Tversky T.,
Modeling cortical maps with Topographica. Neurocomputing: 1129-1135, Presented
at
the
2003
Computational
Neuroscience
Meeting,
2004.
http://www.topographica.org
[3] Bednar J. A. and Miikkulainen R., Self-organization of spatiotemporal receptive
fields and laterally connected direction and orientation maps. Neurocomputing, 5254:473–480, 2003.
[4] Bednar J. A., and Miikkulainen R., Tilt Aftereffects in a Self-Organizing Model
of the Primary Visual Cortex. Neural Computation, 12:1721–1740, 2000.
[5] Blasdel, G. G. Orientation selectivity, preference, and continuity in monkey striate
cortex. Journal of Neuroscience, 12:3139—3161, 1992
[6] Butko N. and Triesch J., Exploring the Role of Intrinsic Plasticity for the Learning
of Sensory Representations. ESANN 2006 proceedings - 14th European Symposium
on Artificial Neural Networks Bruges, 2005.
[7] Butko N. and Triesch J., V1 maps from sparse neurons with intrinsic plasticity.
12th Joint Symposium on Neural Computation, 2005.
[8] Castellani G. C., Quinlan E. M., Cooper L. N., Shouval H. Z., A biophysical
model of bidirectional synaptic plasticity: Dependence on AMPA and NMDA
receptors. PNAS vol. 98, no 22: 12772-12777, 2001.
[9] Chapman B., et. al., Development of Orientation Preference Maps in Ferret
Primary Visual Cortex. The Journal of Neuroscience, 16(20):6443–6453, 1996.
[10] Choe Y. and Miikkulainen R. Contour Integration and Segmentation with SelfOrganized Lateral Connections, Biological Cybernetics 90:75-88, 2004.
74
[11] Davis, G.W., Homeostatic Control of Neural Activity: From Phenomenology to
Molecular Design, Annual Review Neuroscience, 29:307-323, 2006
[12] Davis, G.W., Bezprozvanny, I., Maintaining The Stability of Neural Function: A
Homeostatic Hypothesis, Annual Review Physiology, 63:847-69, 2001
[13] Desai N., Homeostatic plasticity in the CNS: synaptic and intrinsic forms.
Journal of Physiology – Paris 97:391-402, 2003.
[14] Desai N., Rutherford L. C., Turrigiano G. G., BDNF regulates the intrinsic
excitability of cortical neurons. Learning & Memory 6:284-291, 1999.
[15] Di Paolo E. A., Homeostatic adaptation to inversion of the visual field and other
sensorimotor disruptions. From Animals to Animats 6. Proceedings of the VI
International Conference on Simulation of Adaptive Behavior: 440-449.
[16] Di Paolo, E. A., Organismically-inspired robotics: homeostatic adaptation and
teleology beyond the closed sensorimotor loop, Dynamical systems approach to
embodiment and sociality, 2003.
[17] Gutig R., et. al., Learning Input Correlations through Nonlinear Temporally
Asymmetric Hebbian Plasticity, The Journal of Neuroscience, 23(9):3697-3714, 2003.
[18] Miikkulainen, R. Self-organizing process based on lateral inhibition and synaptic
resource redistribution. In Kohonen, T., et. al., editors, Proceedings of the 1991
International Conference on Artificial Neural Networks, 415–420, 1991
[19] Miikkulainen R., Bednar J. A., Choe Y., and Sirosh J., Computational maps in
the visual cortex. Springer, Berlin, 2005.
[20] Miikkulainen, R., Bednar, J. A., Choe, Y., and Sirosh, J. Self-organization,
plasticity, and low-level visual phenomena in a laterally connected map model of the
primary visual cortex. In Goldstone, R. L., et. al., editors, Perceptual Learning, vol.
36 of Psychology of Learning and Motivation, 257–308, 1997
[21] Shouval H. Z., Bear M. F., Cooper L. N., A unified model of NMDA receptordependent bidirectional synaptic plasticity. Proc Natl Acad Sci USA, 2002.
75
[22] Sincich, L. C., and Blasdel, G. G. Oriented axon projections in primary visual
cortex of the monkey. The Journal of Neuroscience, 21:4416–4426, 2001
[23] Sirosh J., Miikkulainen R., and Bednar J.A. Self-organization of orientation maps,
lateral connections, and dynamic receptive fields in the primary visual cortex. In
Lateral Interactions in the cortex: Structure and Function, UTCS Neural NetRes Grp.
Electronic book, ISBN 0-9647060-0-8, 1996.
[24] Stemmler, M., Koch, C., How Voltage-Dependent Conductances can Adapat to
Maximize the Information Encoded by Neuronal Firing Rate. Nature Neuroscience,
2:521-527, 1999.
[25] Sullivan T. and de Sa. V. R., A Self-Organizing Map with homeostatic synaptic
scaling. 14th Annual Computational Neuroscience Meeting, CNS, 2005.
[26] Sullivan T. and de Sa. V. R., Homeostatic synaptic scaling in Self-Organizing
Maps. Workshop on Self-Organizing Maps, Paris, 2005.
[27] Triesch J., A gradient rule for the plasticity of a neuron's intrinsic excitability.
ICANN 2005, LNCS 3696:65-70, 2005.
[28] Triesch J., Synergies between intrinsic and synaptic plasticity in individual
model neurons. Advances in Neural Information Processing Systems 17. MIT Press,
2005.
[29] Turrigiano, G. G., Homeostatic plasticity in neuronal networks: the more things
change, the more they stay the same. Trends in Neurosciences 22: 221-227, 1999.
[30] Turrigiano G. G. and Nelson S., Hebb and homeostasis in neuronal plasticity.
Current Opinion in Neurobiology 10:358-364, 2000.
[31] Turrigiano G. G. and Nelson S., Homeostatic Plasticity in the Developing
Nervous System. Nature Review: Neuroscience, Vol. 5, 2004.
[32] van Rossum M. C. W., Bi, G. Q., Turrigiano, G. G., Stable Hebbian Learning
from Spike Timing-Dependent Plasticity, The Journal of Neuroscience, 20(23):88128821, 2000.
76
[33] Williams H. and Noble J., Homeostatic plasticity improves signal propagation in
Continuous-time Recurrent Neural Networks. Sixth International Workshop on
Information Processing in Cells and Tissues, IPCAT, 2005.
[34] Yeung L. C., Harel Z. S., Blais B. S., Cooper L. N., Synaptic homeostatis and
input selectivity follow from calcium-dependent plasticity model. PNAS vol. 101, no
41:14943-14948, 2004.
[35] Yeung L. C., Blais B. S., Cooper L. N., Shouval H.Z., Metaplasticity and the
Unified Calcium Model lead to input selectivity in spiking neurons.
[36] Zhang W. and Linden D. J., The Other Side of the Engram: ExperienceDependent Changes in Neuronal Intrinsic Excitability. Nature Reviews Neuroscience,
4:885-900, 2003.
77