Download Notes on Learning to Compute and Computing to Learn

Notes on Learning to Compute and Computing to Learn
Khurshid Ahmad
Department of Computing
University of Surrey
Guildford, Surrey, GU2 7XH ([email protected])
Abstract
One key message in modern neuroscience is multimodality: the ability of uni-modal areas in the brain
such as speech and vision to interact with each other
and with hetero-modal areas, areas that are activated
by two or more input modalities, to converge the
outputs of the uni-modal systems for producing ‘higher
cognitive’ behaviour. Such behaviour includes the
ability to quantify, the ability to retrieve images when
linguistic cues are provided and vice versa. Multi-net
neural computing systems that can simulate such
behaviour are reported. Multi-net systems comprise of
modules that take unimodal input and one or more
modules that facilitate cross-modal interaction
through Hebbian connections. These systems have
achieved a modicum of success.
Keywords: Multi-net neural computing, multi-modal
systems, competitive learning, image retrieval,
information extraction.
1
Introduction
Neural computing systems were inspired by
developments in neuroscience in the 1940’s and
with few exceptions such systems work as
cellular networks and attempt to simulate learning
at the cellular level.
Developments in
neurosciences, especially during the 1980’s have
provided a unique insight into how the brain
functions, and by implication, how humans learn
and behave.
A brief review of these
developments is in order here (Section 2) before I
describe some of the work in neural computing,
which has been inspired by looking at the cooperative behaviour of two or more cellular
networks for simulating intelligent behaviour.
The title of this paper is an imitation of
Elaine Rich and Kevin Knight’s introduction to
neural computing in their book Artificial
Intelligence (1999) where they suggest if the
network can compute then it will learn to
compute. This is perhaps at some variance with
other cognitivist approaches in machine learning
where the emphasis appears to be on ‘computing
to learn’ – the insistence being on representation
schema and reasoning strategies whether it is rule
induction or rule deduction, or it is learning by
synthesis or learning by analysis. More recent
approaches to machine learning, for insistence
case-based reasoning, despite their welcome
departure from an immutable rule base, still have
the structuralist influence of early AI. My
interpretation of ‘learning to compute’ is that
learning emerges alongside changes in the
structure of the neural substrate. In neural
computing, weight changes do suggest changes in
the neural substrate. Work in neurobiology and
neuropsychology suggests that areas within the
brain get interconnected in addition to
interconnections within an area.
The more
important lesson is that some human behaviour
can only be explained on the basis of distinct and
large areas of the brain interacting in unison
(Section 3). Some neural computing systems
developed by my students over the last decade in
an attempt to train not only individual networks
but also to train networks to learn to behave cooperatively are discussed in Section 4; Section 5
comprises an afterword. Before we embark on
our discussion it may be useful to brush up on
techniques used to observe the brain ‘in action’
and elaborate on terminology used in neuroanatomy.
1.1
Seeing the Brain in Vivo
The human brain specifically, and animal brain
generally, is a complex whole which interacts
with its immediate environment, reacting here,
and through its muscular appendages, changing
the environment there. The brain at work is a
difficult place to observe. It is not possible to
measure single (neuronal) activity of neurons or
populations of neurons directly except during
surgery. One indirect measurement of neuronal
activity is the observation that active areas of the
brain invariably have increased blood flow as
compared to when these areas are not active.
Positron emission tomography (PET) technique
uses radio-labelled positron emitting isototpes and
areas of brain that may be active for durations of
1-1000 seconds and localised to within 10-100
mm can be ‘seen’ by PET scanner. Functional
magnetic resonance imaging (fMRI) is used in
visualising neuronal activity by observing flow
rates changing and affecting the magnetic signals
to which MRI images are sensitive. Many
researchers
now
confidently
report
nueroanotmoical correlates of perceptual and
cognitive behaviour. And there is speculation
about the evolution of cortical areas of the brain
as well.
These indirect observations have to be
interpreted carefully due to the ‘great variability’
in brain anatomy between individuals, which
poses technical and conceptual problems as to the
specific location of an activation in the brain.
These problems are exacerbated because the
‘positions of these [functional] areas [of the brain]
are not well predicted by anatomical landmarks in
many regions’[8].
appears to be localised in one or more of the
overlapping four lobes of the brain – frontal, or
everything in front of the so-called central fissure
of the brain; parietal lobe that is caudal to (or
towards the tail of) the frontal lobe); temporal
lobe jutting forward from the base of the brain
and ventral to (or at the base of) the frontal and
parietal lobes; and, occipital lobe which lies at the
back of very back and is caudal to the parietal and
temporal lobes. Each of the lobes is further
subdivided into posterior (back) and anterior
(front) or medial and lateral parts. The cerebral
cortex is convoluted and as much as 2/3rd of the
cortex is ‘hidden’ in small grooves (sulci, singular
sulcus), large grooves (fissures), and gyri
(singular gyrus) or the bulges between adjacent
sulci or fissures.
For instance, the fissure
between the temporal and frontal lobe is called
Sylvian fissure.
P. Somatosensory
Cortex
P. Motor Cortex
PAR IETAL LOBE
5
9
7
10
11
44
38
Visual Ass.Cortex
40
43
47
P. Auditory Cortex
Auditory Ass. Cortex
312
46
Motor Ass. Cortex
300 Words on Neuroanatomy
Our physical and mental environment appears
‘seamless’ despite the seemingly independent
modalities of vision, speech, hearing, olfaction
and touch amongst others. These modalities
interact and intermingle to manifest as unities at a
certain level of description: these unities include
objects, events, concepts and emotions. The
unities are dynamic in that they move, appear and
disappear. The system that deals with each of
these apparently independent modalities, the
human nervous system, does appear to comprise
many interacting and diffusely delineated parts.
The human brain, part of the central nervous
system, is not a uniform mass and appears to have
two interconnected hemispheres, left and right.
The cerebral cortex, the outermost layer of grey
matter of the cerebral hemispheres, is critically
involved in how we not only use the cortex, and
other areas of the nervous system, to understand,
exploit and sustain our environment, but learn to
do it as well. This interplay of sensing and
learning to make sense of the environment is most
obvious in infancy and when humans encounter
novelties in their environment. Each sensory
modality and the associated motor movements
4
6
8
FRONTAL LOBE
45
1.2
Somatosensory
Ass. Cortex
39
41
21
T EMPORAL LOBE
20
42
22
19
37
P. Visual Cortex
18
17
OCCIPITAL LOBE
Figure 1. An approximate lateral view of the various regions of the cerebral cortex. The
numbered boxes refer to Brodmann’s cyto-architectural areas originally discussed in the early
20th century. (P. refers to Primary and Ass. to Association. More accurate view of the cerebral
cortex are available on http://www.driesen.com/ or on
2
Neural Correlates of Behaviour
Literature on physiological psychology suggests
that the left hemisphere is involved in ‘controlling
serial behaviors’ including those of talking,
understanding the speech of others, reading and
writing. The right hemisphere appears to be more
active when humans engage, in laboratory
conditions, in ‘synthesis’ – drawing sketches,
reading maps, making wholes from parts and is
also involved in ‘the expression of and
recognition of emotion in the tone of voice’ [11,
29]. Right-hemisphere lesions to the parietal
cortex are possibly responsible for deficits in
spatial attention – attention being the cognitive
faculty, which enables humans to focus on certain
features of the environment to the (relative)
exclusion of others [16].
Each primary sensory area of the cerebral
cortex sends information to adjacent regions,
called the sensory association cortex also known
as memory cortex: all motor areas have some
sensory input. Association cortices, parts of the
cerebral that do not have primary motor or
sensory role, are involved in the so-called ‘higher
order processing of sensory information.’ There
are three major cortices of interest: posterior
parietal cortex (PPC), prefrontal cortex, and
temporal cortex. The temporal cortex has the
visual and audio association cortices; the
prefrontal cortex comprises ‘motor memories
[specifically] memories of muscular movement
that are needed to articulate words’ [11] and is
supposed ‘to respond to complex sensory stimuli
of behavioural relevance’ [6]; and, the PPC
comprises areas involved in combining
information on personal (somatic) awareness –
where am I? – with information on extra-personal
(visual) space.
2.1
Cortical Correlates of Perception
The visual cortex has at least 25 different
subregions arranged in a hierarchical fashion for
processing from colours (human extrastriate
cortex) to object perception (ventral stream),
from movement-, size-, and orientation of objects
(inferior temporal cortex) to movement
perception and object location (posterior parietal
cortex). Various left-hemisphere (perisylvian)
areas – temporal, parietal and frontal – have been
identified as being involved with the
‘complexities of linguistic processing, ranging
from semantic to syntactic, morphological,
phonological and phonetic analysis and synthesis’
[20]. The ‘language areas’ in the brain were the
first to be identified when the doctrine of cerebral
localization was launched in the 19th century:
Broca’s area or the ‘motor speech area’,
comprising portions of the inferior frontal gyrus,
for coordinated action of diverse muscles used in
speech, and the ‘sensory or ideational speech
area’ – distributed over the left temporal lobe and
inferior parietal lobe – responsible for
‘comprehension of language, naming of objects’
[7] amongst others. Wernicke’s area, in the
parietal lobe, contains ‘the auditory entries of
words; the meanings are contained as memories
in the sensory association areas’ [11].
Cerebral localization data for 58 word
production experiments, involving measurements
of activation levels in parts of the brain to within
10-100 mm were performed for example with
neuroimaging techniques as diverse as PET, EEG,
and fMRI, suggest that there are 28 major
regions, mainly in the cerebral cortex, which were
highly activated and a finer grain analysis
suggests that there maybe as many as 104 regions
which are involved in language production [18]
The experiments were performed by different
research teams but mainly focussed on 7 word
production tasks including picture naming and
word generation. These tasks include ‘conceptual
preparation, lexical selection, phonological coderetrieval and encoding’: the first task 150 ms, the
second is completed in the next 125 ms and the
last two within the next 125 ms. The next two
tasks take another 125 ms all told. There are
identifiable regions that are activated in word
generation that are not activated in picture naming
and vice versa. Lesion studies indicate that when
local specialist areas in the brain suffer damage
the brain cannot perform one or more functions
that it can normally perform: damage to the
various parts of parietal and temporal lobes
results in the loss of naming capability and
damage to the frontal cortex rostral (or in front of)
and to the base of the primary motor cortex leads
to non-fluent speech. (Section 4 briefly describes
two multi-net computing systems that attempt to
simulate language development and language
deficit respectively).
Vision and speech processing are good
examples of unimodal processing: it appears that
one single modality is being processed across a
network, comprising units at different locations in
the brain, and each component is more or less
specialised in processing that modality. Output of
the one unit becomes the input of the next and the
process continues until manifestations of
cognitive behaviour are finally produced:
identification of an object in an image,
understanding of a word or phrase, articulation of
linguistic output in response to a linguistic
stimulus.
2.2
Cortical Processing and Cortical
Development
The above discussion is based implicitly on the
behaviour of an adult human. It is reasonable to
expect that humans have learnt how to react to
the various modalities. Neonates, infants and
young children, almost irrespective of their
intelligence level, appear to learn to react to their
external environment at a considerable pace
whilst their nervous system is rapidly evolving as
well [5]. During the first 12-38 weeks of antenatal life the structure of the central nervous
shapes
up
accompanied
by
neuronal
differentiation, through axonal growth and
dendritic ‘ontogeny’, and by the development of
synapses (‘synapto-genesis’) and the encasing of
the synapses in the myelin sheath (myelinogenesis). Even during the genesis of the antenatal nervous system, the infant responds to
stimuli and when born appears to have capability
as diverse as being able to visually enumerate and
so appears to possess a substrate for language as
well. The first year of post-natal life involves the
establishment of connectivity in the infants brain
together with glio-genesis – the encasing of the
brain in the white matter. The child’s gaze
stabilizes earlier on in the postnatal development,
the child goes from ‘one-word’ language stage to
two words during the first 18-24 months of life.
2.3
Modality and Neuronal Correlation
The child can integrate different modalities –
Lewkowicz and Turkewitz [19] report an
experiment involving 32 infants between the
‘ages’ of 11 hours and 48 hours in which their
visual preferences for ‘light patches of different
intensity’ were examined with and without an
auditory stimulus (through a white noise
generator). The authors conclude that ‘visual
preference in human newborns can be modified
by immediately preceding exposure to sound’
[19]. Such inter-sensory interaction between
auditory and visual stimulation can be observed
when we watch television or films where TV folk
and actors appear to be talking through their lips
but actually the sound is emitted from a
loudspeaker. And ventriloquism depends on the
dominance of the visual stimulus on the auditory
stimulus. This has led to the claim that ‘illusions
remind us the our visual experience is not a
measurement of the physical stimuli, but rather a
neural computation’ [31].
There are then degrees of involvement of
other modalities, for example, naming an object
in a visual scene requires two modalities, visual
and speech, although one can segregate the two in
that after the initial visual stimulus one can argue
that speech takes over. However, attention and
numerosity studies indicate that certain behaviour
can only be explained through multi-sensory
integration. This integration has been defined as
a statistically significant difference between the
neuron’s response to a stimulus combination
compared to its response to the individual
component stimulus [21].
There is some
evidence, based on experiments on cats, that
certain areas of the cats’ nervous system comprise
unimodal neurons, at least until 12 days after
birth, but then these neurons develop capability
for integrating multi-sensory information [30]. It
has been suggested that there are ‘many areas in
the mammalian brain […] where the different
sensory streams converge onto individual neurons
responsive to stimulation in more than one
modality’ [10] and such heteromodal neurons
have been found in the prefrontal cortex, posterior
parietal cortex and the lateral temporal cortex and
in the mid-brain (specifically in the super
colliculus – the site where Meredith and Stein
[21] found multi-sensory integration in cats).
3
Learning to Compute: Cross Modal
Interaction
and
‘Numerical’
Neurons
There are two developments that indicate that
there are certain ‘higher level’ cognitive tasks that
can be understood in terms of interaction between
different sensory modalities (and motor
movement). The first development relates to the
orientation of human spatial-attention –our ability
to ‘focus selectively on sensory information from
particular locations in external space’ either
voluntarily, endogenous attention, or ‘reflexively’
by salient events, exogenous attention. In a
number of neuropsychological experiments, it has
been found that deficits in spatial attention are
correlated strongly with lesions to the parietal
cortex in the right hemisphere and lesions to the
frontal cortex. Amongst many like phenomena,
visual illusions, actors speaking on a film/TV
screen or ventriloquists’ dummy talking, indicate
that human attention has to be co-ordinated crossmodally. This makes it possible to ‘select
information from a common external source
across several modalities despite the differences’
in the initial coding of the source of each
modality [17]. Frequently, one or more senses
substitute for the (temporary) loss of one of the
senses: looking for a light source in a darkened
region results in the heightened textural and
spatial awareness is a good example for such a
substitution. This cross cueing has been exploited
in the multi-net simulation reported in Section 0
where texts help in the retrieval of images.
The second development relates to topics
variously labelled numerosity, numerons, single
neuron arithmetic and number sense in humans
and some primates. The intuitive argument here
is that judgement related to quantities, an animal
guessing how many in a herd or the extent of
‘foraging area’, or infants and monkeys making
‘accurate’ judgements about whether two
quantities are equal to each other or very different
irrespective of physical attributes, must have a
neuronal correlate. Observations on enumeration
without having been taught a number system,
subitisation
or
visual
enumeration, or
approximate calculation without rigorously
carrying out arithmetic procedures, lead to the
speculation that there may be areas in the brain
where the visuo-spatial information about the
objects, for instance, is processed such that the
number information is preserved [15]. The
development of numerosity has been simulated
using a multi-net computing system later on in
this paper (Section 4.2). We will now discuss the
two developments in turn.
3.1
Cross-Modal Interaction and Spatial
Attention
The key to spatial attention is that different
stimuli, visual and auditory, help to identify the
spatial location of the object generating the
stimuli. One argument is that there may be a
neuronal correlate of such crossmodal interaction
between two stimuli. Information related to the
location of the stimulus (where) and identifying
the stimulus (what) appears to have correlates at
the neuronal level in the so-called dorsal and
ventral streams in the brain.
In a number of neuroimaging studies of
audio-visual speech perception, researchers have
attempted to identify ‘putative’ neuroantomical
sites where multimodal integration actually takes
place [10] – these studies were inspired, in part,
by the earlier work on cats [21, 22]. Two
experiments, one dealing with subjects’ mouth
movements whilst looking at a videotape of the
lower half of a face silently mouthing numbers –
silent lip reading – and the second with subjects
listening to numbers being spoken with the
videotape switched off – auditory speech
perception – are of note here. The neuroimages
of the two experiments were compared and
contrasted. The areas activated in both the
experiments include primary auditory cortex and
the auditory association cortex. The putative
heteromodal cortex, that integrates the two
stimuli, straddles audio- and visual association
cortices and appears to include Wernicke’s area
of language idealisation (Brodmann areas 37, 39,
40, 21/22), lie in the region ‘proximal to the
superior temporal sulcus (STS). In the silent-lip
reading test the visual stimulus provided to the
brain appears to generate auditory cues from the
auditory association cortex and the convergence
takes place in the STS. Calvert and colleagues
claim that ‘Activation in of the primary auditory
cortex by visible speech cues might proceed via
back projections from heteromodal cortex’ [10].
The authors point to other regions of the brain as
well, including the prefrontal cortex, posterior
parietal cortex, and possibly the midbrain region
of superior colliculus. There is a concomitant
claim that there may exist multimodal neurons
active in ‘many areas in the mammalian brain
[…] where the different sensory streams converge
onto individual neurons responsive to stimulation
in more than one modality’ [10].
One related cross-modal phenomenon is
that of synesthesia: a condition in which a sensory
experience normally associated with one modality
occurs when another modality is stimulated. This
is a (congenital) condition when synesthetic
humans recall colour when shown letters or
numbers for example.
Such cross-modal
behaviour may be attributed to cross wiring of the
otherwise unimodal brain or more specifically
cortical regions.
The grapheme-colour
synesthesia or (involuntary cross-activation) has
been explained by arguing that the colour areas in
the brain are in the fusiform gyrus and the visualgrapheme area is also in the fusiform, especially
in the left-hemisphere [27].
3.2
Numerosity,
‘Numerons’
Number
Sense
and
The observation that neonates and monkeys have
a number sense and other mathematical skills like
estimation, trajectory computation, without being
formally educated in arithmetic, or any other
branch of mathematics, has given rise to
significant interest in this area. Neuroimaging
techniques have significantly contributed here and
some even claim to have neural correlates of our
ontogenetic numeracy. Furthermore, educational
psychologists have observed numeracy and
related skills can be acquired through training and
that there are stages in which skills are acquired –
there is a evolutionary process involved here. For
some neonates and primates only acquire
numeracy when they come in contact with their
physical environment. Whether ontogenetic or
evolutionary, our number sense involves ‘the
coding and internal manipulation of an abstract
semantic content, the meaning of number words’
[26]. Information about putative neural correlates
of numerosity has come from the so-called lesion
studies. Studies of brain damaged individuals
showing mathematical deficits in their behaviour,
when compared to their intact counterparts,
suggests that lesions to specific regions of the
parietal and temporal cortices may be the reason
for the deficits . The parietal cortex appears to
play a critical role in the representation of
magnitudes and the temporal cortex is involved in
the representation of the visual form of the
numbers [9, 13].
Number sense has played a major role in
psychology where many earlier studies were
dedicated to ‘the mathematical description of how
a continuum of sensation, such as loudness or
duration’ is represented in the brain/mind. The
19th century psychophysicist, Gustav Fechner, had
observed that ‘the intensity of subjective
sensation increases as the logarithm of the
stimulus intensity’. One of the 21st century
rendition of this ‘law’ is that the ‘external
stimulus is scaled into a logarithmic internal
representation of sensation’ [15]. Number related
behaviours ‘depend on the capacity to abstract
information from sensory inputs and to retain it in
memory’ and that in monkeys this capacity is in
the ‘prefrontal cortex’ [24] and there are reports
of activation in humans in proximate regions of
the brain. As predicted by Fechner, there is a
compressed scaling of numerical information, and
this information is stored in the prefrontal cortex
of the monkey [24] and the parietal cortex of the
human [26]. Neider et al report over a third of the
352 randomly selected neurons from the lateral
prefrontal cortex of two monkeys ‘showed
activity that varied significantly with the number
of items in the sample display’ [24]: this suggests
that certain neurons specialise as ‘number
detectors’ – the illusive numerons perhaps have
been found.
The compressed number line theory can be
used to explain the observation that neonates and
monkeys, and adults in a hurry, can accurately
enumerate quantities less than 5 without recourse
to overt counting. Higher numbers cannot be
enumerated with any accuracy through visual
enumeration or subitisation and that within the
numbers 1-5, there is a diminution in accuracy as
we approach the higher number. Subitisation is
sometimes related to the existence of ‘preverbal
numerical abilities’ [34]. Recent findings about
approximate calculations performed by healthy
volunteers shows activity in the bilateral inferior
parietal cortex, in the prefrontal cortex and in the
cerebullum [14]; perhaps, the preverbal numerical
abilites are localised in these areas. These
cortices were less active when the same
volunteers carried out tasks where they were
asked to perform exact arithmetic calculations.
During the exact arithmetic the left inferior
parietal cortex was highly activated together with
the left angular gyrus. The cortical regions active
in approximate arithmetic ‘fall outside of
traditional perisylvian language areas and are
involved in various visuo-spatial and analogical
mental transformations’ (ibid:971).
Exact
calculations, it appears, depend on languagebased representations. Dehane et al [14] recall
from previous lesion studies that lesions to the
left parietal area result in the loss of the sense of
numerical quantity but preserve rote language
based arithmetic, contrariwise the damages to the
left hemisphere resulted in the loss of language
abilities but the sense of numerical quantity were
unimpaired.
The contrasts in response times to
stimulation by numbers verbally and in numbers
presented in the Arabic notation show clear
differences; response to visual stimulation is on
average 100-200ms faster than to verbal
stimulation [26]. Studies involving fMRI and
event-related potential measurement show
indication of localisation: higher activations
reported for visual stimulation in the left and right
parietal and prefrontal cortex and for verbal
stimulation left and right occipital regions show
greater activation (ibid:1020). Simon et al [28]
have conducted fMRI experiments to examine the
organisation of the parietal cortex by looking at
fMRI images of subject performing six tasks:
pointing,
grasping,
attention,
saccades,
calculation and phoneme detection. They found
that ‘number processing tasks call for generic
processes of attention orientation and spatial
manipulation that operate upon a “spatial map” of
numerical quantities in the middle IPS (Inferior
Parietal Sulcus]’ [28]; the IPS is a ‘fine grained
anatomical specialisation’ which may have a
region for the manipulation of numerical
quantities.
4
Computing to Learn: Co-operating
Neural Networks and Competing
(inhibiting) Neurons
In this section I present multi-net neural
computing systems in an attempt to simulate
aspects of behaviour that invariably, and from
what we have discussed above inevitably, involve
interaction between two or more modalities (see
Figure 2).
.
.
.
Unimodal
Input1
.
.
.
Unimodal
Net1
Cross-modal Net:
Bi-directional
Hebbian Network
Unimodal
Net2
Unimodal
Input2
Figure 2: An architecture for learning behaviour
encoded in two modalities and for learning how to
cross-modally learn the relationships between two unimodalities
Our early work during the early 1990’s focussed
on language processing, specifically child
language development and language disorder,
both these systems have individually trained
networks for representing concepts and linguistic
description of concepts that are cross-linked
through a Hebbian network. The language
development system learnt to simulate how young
children (18-24 months) produce one-word
utterances in response to audio-visual cues and to
simulate how these children learn the word-order
of their language based on inputs from their adult
caretakers.
The word-order learning system
learnt concepts and words using two independent
SOFM’s, cross linked by a Hebbian network,
together with an additive Grossberg network that
related semantic relations (for example, agents to
possessions, objects to containers, objects in
spatial locations). A back-propagation network
was used to teach the child the two-word
organisation as found in adult language. What the
network produced was a cross-modal output
(concept-word) that guided its production of
candidate two-word collocates. The system’s
learning outcomes were compared with standard
child language productions reported in the
literature and there was good agreement between
the observations and our system [1]. The
language disorder system was similarly
constructed with conceptual and a word lexicon
using SOFM’s and the system learnt to cross-link
concepts and words through a Hebbian network.
The system was trained on normal associations
between words and concepts. The conceptual
SOFM was then systematically ablated and the
cross-modal output showed increasing semantic
errors – the so-called naming errors found in a
certain group of aphasics. The results of the
language disorder multi-net were in good
agreement with findings about aphasics in the
literature [33].
In this paper we report on our more recent
work: (i) a multi-net contents-based image
retrieval system, which can store images with
their collateral textual description, and the system
can learn to retrieve images by their collateral
linguistic features and even images for which
there is no collateral text (Section 4.1), the link
between image and linguistic features was
established through a Hebbian network, and (iii) a
multi-net system that can learn to subitise and
another that can learn to count (Section 4.2), both
these systems rely on a Hebbian connection that
is learnt whilst two individual networks learn a
single modality each to represent quantities in a
visual scene and the verbal representation of the
quantities.
The co-operative multi-net architecture we
report is essentially an extension of the original
idea of Willshaw and von der Marlsburg [32]
where two layers of neurons were connected via
Hebbian synapses; the Hebbian synapses have the
useful properties of local modification, time
variance, and have the useful feature that
correlates pre- and post-synaptic signals.
Kohonen’s self-organising feature map is also
based on a principle similar to that of Willshaw
and von der Marlsburg. The links are initially
weighted as zero or set at random and as the
training proceeds the connection strengths are
changed. Once the correlation is established
between pre- and post- signals, two uni-modal
inputs for us, we can use one modality as a cue
for the other.In all the systems we have developed
our emphasis has been on the use of the
unsupervised learning algorithms, except in cases
it was necessary to use supervised learning
algorithms.
This preferential use of an
unsupervised learning algorithm is based on the
view that ontogenesis plays a key role in learning,
and that occasions where environmental input
acts as a teacher are rather limited though
nonetheless important. We use self-organising
feature maps (SOFM) due to Teuvo Kohonen that
can transform a continuous n-dimensional input
signal onto a one or two-dimensional discrete
space of ‘neurons’. A discriminant function is
used to relate the weight vectors that connect the
input layer to the output layer: the weight vector
closest to the input is regarded as the ‘winner’ and
selected neighbourhood neurons in the output
layer form a halo and are activated when the
winner is activated.
4.1
Collateral Image and Text System
Images have been traditionally indexed with short
texts describing the objects within the image. In
some cases it needs a specialist to literally go
behind the image and discover objects not
apparent to a laypersons: radiologists, forensic
professionals, good art critics, are amongst those
specialists. The specialists excel also because
they are succinct in their description and
sometimes it’s a gift and at others this
succinctness is acquired through experience and
training. The accompanying text is sometimes
described as collateral to the image. The ability
to use the collateral texts for building computerbased image retrieval systems will help in dealing
with image collections that can now be stored
digitally. Theoretically, the manner in which we
grasp the relationship between the ‘features’ of
the image and the ‘features’ of the collateral text
relates back to cross-modality. The use of a
multi-net system comprising independent yet
interacting neural networks also contributes to the
debate on multiple classifier systems.
We have developed a multi-net system that
learns to classify images within an image
collection, where each image has a collateral text,
based on the common visual features and the
verbal features of the collateral text. The multinet can also learn to correlate images and their
collateral texts using Hebbian links – this means
that one image may be associated with more than
one collateral text and vice versa [3, 4]. The
details of the system are given below:
Module
Image Feature
Representation
Collateral Text
Feature
Representation
Image-Text Cross
Modality
Network
Architecture
Kohonen SOFM
Network
Topology
15× 15
Kohonen SOFM
15× 15
Hebbian Links
15× 15
We have had access to 66 scene of crime
images used in training scene-of-crime officers:
58 of these images were used for training and 8
for testing purposes.
Images features were
represented on a 112-dimensional vector: 21 for
colour distribution, 19 for shapes, and 72 for
texture; each of the features were automatically
extracted; the collateral texts were represented by
a 50-dimensional input vector: frequently
occurring and semantically-relevant keywords
were extracted automatically from the collateral
texts. Both the image and text vectors were
mapped onto 15X15 output layers of two
Kohonen Maps. During training, a Hebbian
network learnt to associate the most active
neurons in the two maps thereby establishing a
degree of cross-modality. Another system was
created where only one SOFM was trained to
recognise both the images and texts and was
trained using a combined 161 (121+51)
dimensional vector – we would call it a
monolithic system.
The trained system was then tested on the
8 images and their collateral texts. The multi-net
system could classify 7 out of 8 images correctly
whereas the monolithic system could only
classify 3 out of the 8 images correctly. By
correctly we mean that the images fell into a
cluster of images on the output image and text
SOFMs deemed similar by forensic experts
working with us. A closer examination of how
the test images and their collateral texts were
classified in the image and the text SOFMs
showed that the image SOFM could classify 4 out
of 8 images correctly whereas 5 out of 8 collateral
texts were in the ‘right’ clusters. The cross-modal
interaction between the two improves the
classification significantly – crudely by as much
as twice 4 compared with 7.
4.2
distance effect – larger the difference between the
two numerosities, further apart they are:
Numerosity Development
The Surrey Subitization System was developed to
study how an artificial neural can learn to
enumerate approximately (subitise) [2]. The
system comprises three interacting modules.
First, the mapping module, for responding to the
presence of objects in a visual scene irrespective
of their size and location, and then to represent
the response.
The second is a magnitude
representation module that was learns to represent
small numerosities as magnitudes along a number
line according to the Fechner’s law and the socalled distance effect. The third module is the
output module that comprises a word lexicon for
representing numbers verbally and a cross-modal
module that learns the relationship between the
magnitude representation and the verbal
representation. The details of the system are
shown below:
Module
Mapping: Scale
Invariance
Mapping: Translation
Invariance
‘Magnitude’
Representation
Verbal Representation
Magnitude-Verbal
Cross Modality
Network
Architecture
Second-order
Weight
Sharing
Kohonen
SOFM
Kohonen
SOFM
Hebbian Links
Network
Topology
648× 72
72 × 15
36× 36
16× 64
36× 64
The mapping modules transform a simple
visual scene onto a simple record of the whole
entities in the scene.
The magnitude
representation module, an SOFM, receives its
scale and translation invariant input from the
mapping module. This input is mapped onto the
output layer of the SOFM.
Initially, each
magnitude is assigned a random position on the
output layer:
1, 4
2, 3,
5
In one example, we trained the network to learn
numbers upto 5 and after a 100 training cycles,
the output layer appeared ‘compliant’ with
Fechner’s Law – numerosities organised on a
compressed number line, and reflected the
1
2
3
4
5
The results of our simulation compare well with
[12] in that the authors had used a hard-wired
network and we have trained our network. Our
subitisation system performs marginally better
than the supervised subitisation system reported
by [33]: the supervised system has difficulty in
representing intermediate numerosity, e.g ‘4’ is
not represented well when the system learns
numerosities between ‘1’-‘5’, and is dependent
upon the rather arbitrary choice of hidden layers
used typically in a supervised network [2]. In
another experiment the system was trained to
subitise numbers between 1 and 22, except for a
randomly selected set of six numbers - 2, 3, 10,
14, 15, 19 [4]. The magnitude and verbal
representation were trained for the 16 numbers in
the training set. The trained system was then
tested for the six numbers in the test set: the
results were encouraging in that each of the
numbers was recognised by the neuron in the
magnitude representation closest to its value and
the corresponding verbal output was identified as
well. In contrast a monolithic SOFM combining
both the magnitude and verbal representation
trained on the set of 16 numbers failed to
recognise most of the test set. Another advantage
of the cross-modal system was that input in one
modality (say magnitude) could cue output in
another modality (say verbal articulation) and
vice versa.
5
Afterword
Based on observations from the neuroimaging
studies, lesion studies, and neuropsychological
experiments, I can claim retrospectively that all
the systems reported below have a heteromodal
region by way of an established training
algorithm, the Hebbian learning algorithm. The
strategy myself and my colleagues adopted was to
simultaneously train two uni-modal neural
networks, say, one dealing with learning image
features and the other the linguistic description of
a visual scene.
During this training, we
interspersed a Hebbian network that learnt the
association between the image features and the
linguistic features of the accompanying
description. The Hebbian links are the putative
cross-modal links and the Hebbian networks the
heteromodal area.
It is important that neural computing
literature is aware of some fascinating
developments
in
neurobiology
and
neuropsychology as such awareness will show the
scope and limitations of neural computing
systems. There have been words of warning
about reading too much in the neuroimages [8]
and there had been numerous warnings about
reading too much into the results reported by the
neural computing community. Yes, each human
is unique and perhaps this is reflected in each
humans neuroanatomy, but we all believe, at least
most of the times, that an actor is speaking on the
TV or cinema screen, or that we do use modalities
interchangeably whilst looking up objects and
artefacts. How do we explain shared behaviour?
Neural networks reported in the literature may be
reduced a collection of switches or regression
analysis systems – both these statements are true
in that the whole purpose of a reductive argument
is to do just that. But there is, according to our
current scientific thinking and practice, electrical
activity, chemical and metabolic activity observed
in the brain, perhaps there is no harm in starting
with switches and regression algorithms.
References
[1] Abidi S.S.R. & Ahmad, K. (1997). Conglomerate
Neural Network Architectures: The Way Ahead
for Simulating Early Language Development.
Journal of Information Systems Engineering, vol.
13(2), pp. 235-266.
[2] Ahmad K., Casey, M. & Bale, T. (2002).
Connectionist Simulation of Quantification Skills.
Connection Science, vol. 14(3), pp. 165-201.
[3] Ahmad, K., Vrusias, B. & Tariq, M. (2002). Cooperative Neural Networks and 'Integrated'
Classification. Proceedings of the 2002
International Joint Conference on Neural
Networks (IJCNN'02), vol.2, pp. 1546-1551.
[4] Ahmad, K., Casey, M. Vrusias, B., & Saragiotis,
P. (2003) ‘Combining Multiple Modes of
Information
using
Unsupervised
Neural
Classifiers’, Proc. MCS 03 .
LNCS 2709.
Heidelberg: Springer-Verlag.
[5] Anderson, Vicki., Northam, Elizabeth., Hendy,
Julie., and Wrenall, Jacquie.
(2001).
Developmental Neuropsychology: A Clinical
Approach . Hove: Psychology Press. pp5 & 251.
[6] Barker, Roger A., and Barasi, Stephen. (1999)
Neuroscience at a Glance. Oxford: Blackwell
Science.
[7] Barr, Murray L., and Kiernan, John A. (1998). The
Human Nervous System: An Anatomical Point of
View (5th Edition). Philadelphia: JB Lippincotl
Company.
[8] Brett, Matthew., Johnsrude, Ingrid S., and Owen,
Adrian M. (2002). ‘The problem of functional
localization in the human brain’. Nature Vol 3
(March 2002) pp 243-249.
[9] Butterworth, B. (1999). The Mathematical Brain .
London: Macmillan.
[10] Calvert, Gemma, A, Brammer, Michael J., and
Iversen, Susan D. (1998). ‘Crossmodal
identification’. Trends in Cognitive Sciences. Vol.
2 (no. 7, July 1998). 247-253.
[11] Carlson, Neil, R. (1999). Foundations of
Physiological Psychology. Boston Allyn and
Bacon.
[12] Dehaene, S. & Changeux, J.P. (1993).
Development of Elementary Numerical Abilities:
A Neuronal Model.
Journal of Cognitive
Neuroscience, vol. 5(4), pp. 390-407.
[13] Dehaene, S. (2000). The Cognitive Neuroscience
of Numeracy: Exploring the Cerebral Substrate,
the Development, and the Pathologies of Number
Sense. In Fitzpatrick, S.M. & Bruer, J.T. (Eds),
Carving Our Destiny: Scientific Research faces a
New Millennium , pp. 41-76. Washington: Joseph
Henry Press.
[14] Dehaene, S., Spelke, E., Pinel, P., Stanesun,
Tsirkin, S., (1999) ‘Sources of mathematical
thinking: behavioral and brain-imaging evidence’.
Science. Vol 284 (number 5416, May 7 1999) pp
970-974.
[15] Dehaene, Stanislas. (2003) ‘The neural basis of
the Weber-Fechner Raw: a logarithmic mental
number line’. Trends in cognitive sciences. Vol 7
(no. 4) April 2003, pp145-157.
[16] Driver, Jon and Spence Charles (2000).
Multisensory perception: beyond modularity and
convergence’.
Current Biology. Vol. 10,
pp12731-735.
[17] Driver, Jon and Spence, Charles (1998).
‘Attention and the cross-modal construction of
space’. Trends in Cognitive Sciences. Volume 2
pp 254-262.
[18] Indefrey, Peter & Levelt, Willem J. (2000). ‘The
Neural Correlates of Language Production’. In
(Ed.) Michael S Gazzaniga. pp 845-865.
[19] Lewkowicz, David J. & Turkewitz, Gerald.
(1981). ‘Intersensory Interactions in Newborns:
Modification of Visual Preferences following
Exposure to Sound’. Child Development. Vol 52,
pp 827-832
[20] Levelt, Willem, J.M. (2000). ‘Introduction’ (to
Chapters 56-65 on Language). In (Ed.) Michael S
Gazzaniga. pp 843-844.
[21] Meredith, M A., and Stein, Berry, E. (1983).
‘Interactions among converging sensory inputs in
superior colliculus’. Science Vol. 221. pp 389391.
[22] Meredith, M A., and Stein, Berry, E. (1993). The
Merging of Senses. Boston & London: The MIT
Press.
[23] Nieder, Andreas, and Miller, Earl K. (2003).
‘Coding of cognitive Magnitude: Compressed
scaling of Numerical information in the Primate
Prefrontal Cortex’. Neuron. Vol37 (Jan 9, 2003).
Pp 149-157.
[24] Nieder, Andreas, Fredman, David J., and Miller,
Earl, K. (2002). ‘Representation of the quantity of
visual items in the Primate Prefrontal Cortex’.
Science. Vol 297 (6 Sept. 2002) pp 1708-1711.
[25] Peterson, S.A. & Simon, T.J. (2000).
Computational Evidence for the Subitizing
Phenomenon as an Emergent Property of the
Human Cognitive Architecture.
Cognitive
Science, vol. 24(1), pp. 93-122.
[26] Pinel P., Dehaene S., Riviere D., LeBihan D.
(2001) Modulation of parietal activation by
semantic distance in a number comparison task.
Neuroimage, Vol 14(No. 5) pp. 1013-26.
[27] Ramachandran, V.S., and Hubbard, E. M. (2001).
‘Synesthesia – A window into perception,
language and thought’. Journal of Consciousness
Studies. Vol. 8 (No.12) pp 3-34.
[28] Simon O, Mangin JF, Cohen L, Le Bihan D,
Dehaene S. (2002). Topographical layout of hand,
eye, calculation, and language-related areas in the
human parietal lobe. Neuron Vol. 33 (Jan 31,
2002). pp 475-487
[29] Springer, Sally, P. and Deutsch, Gregor (1998).
Left Brain, Right Brain: Perspectives from
Cognitive Neuroscience. New York: W H
Freemand and Company (5th Edition).
[30] Stein, Barry, E., Wallace, Mark T., and Stanford,
Terrence R. (2000). ‘Merging sensory signals in
the brain: The development of multi-sensory
integration in the superior colliculus’. In (Ed)
Michael S Gazzaniga. pp 55-71.
[31] Wandell, Brian, A. (2000) ‘Computational
Neuroimaging:
color
representations
and
processing’. In (Ed) Michael S. Gazzaniga. pp
291-303.
[32] Willshaw, D.J. & von der Malsburg, C. “How
Patterned Neural Connections can be set up by
Self-Organization”, Proceedings of the Royal
Society, Series B, vol. 194, pp. 431-445, 1976.
[33] Wright, J.F. & Ahmad, K. ‘The Connectionist of
Aphasic Naming’. Brain and Language. Volume
59, pp 367-389, 1997.
[34] Wynn, Karen., Bloom, Paul., & Chiang, Wen-Chi.
(2002). ‘Enumeration of collective entities by 5month old infants’. Cognition Vol 83, pp B55B62.
Acknowledgements
This work has been in part supported by the UK
Engineering and Physical Sciences Research Council
(Scene of Crime Information Systems Project, Grant
No.GR/M89041) and 5 UK Police Forces (Surrey,
Metropolitan, Kent, Hampshire and South Yorkshire).
The author is grateful to colleagues Matthew Casey
and Bogdan Vrusias for stimulating discussions and to
Mariam Tariq and Matthew for helping me to proof
read this paper. Errors and omissions are entirely
mine.