Download Multi modality

Perception and Neurorobotics
Multimodal sensory integration in human neocortex
Baran Çürüklü, Senior lecturer, <[email protected]>
Computational Perception Laboratory
Intelligent Sensor Systems, Innovation and Product Realization
School of Innovation, Design and Engineering
Mälardalen University
Contents
• Introduction
• Organization of the nervous system
• Evaluation of models using robots
What does the neocortex?
It is involved in higher functions such as
sensory perception, generation of motor
commands, spatial reasoning, conscious
thought, and in humans, language.
Multimodal sensory integration
• Senses in the multimodal sensory
integration
• Vision, hearing, taste, smell, somatic, …
• Vision: Shape, color, movement
• Just think about multimodal input
• Keyboard and mouse or other modalities such as,
speech, pen, touch, manual gestures, gaze and head
and body movements
Computational neuroscience
Computational neuroscience is an interdisciplinary
field which draws on neuroscience, computer
science and applied mathematics.
It most often uses mathematical and computational
techniques such as computer simulations and
mathematical methods to understand the
function of the nervous system.
The brain
White and gray matter
Gray matter
Neurons communicate by spiking
axon from a presynaptic neuron
dendrites
axon
soma
Classical Results by E.D. Adrian,
1926
• Spike coding is universal: Sensory neurons
produce stereotyped spikes just like the
motor neurons.
• Spike-rate coding: Larger static stimulus
generates higher spike rate.
• Adaptation: Spike rate decline over the
time for a static stimulus.
Somatic Sensory System
Receptor density on
the skin.
Receptors: Touch,
vibration, pressure,
deep pressure, pain,
temperature, muscle
length & tension
The auditory system
Early stages of vision
Representation of the distorted
retinotopic map within the LGN
(left hemisphere)
Organization of the neocortex
• Modular
– Hypercolums
(microcircuits)
– Minicolumns
– Neurons
• Laminar (6 layers)
– Long-range
connections
through the white
matter
– Horizontal longrange connections
– Local connections
Modular organization of the neocortex
Primary visual cortex
Orientation minicolumns
Hypercolumn
Modular organization of the neocortex
Primary visual cortex (V1, area 17)
Layer 2/3 patchy long-range horizontal (L-R) connections within
tree shrew primary visual cortex
Orientation maps, ocular dominance
and spatial frequency in V1
Orientation maps and spatial
frequency
Orientation maps and ocular
dominance
Modular organization of the neocortex
Higher visual areas
•
Laminar organization
Primary visual cortex
Differences between layers:
• Pyramidal cell sizes
– increase in size along layer 2 → layer 3
– large pyramidal cells in layer 5B
• Density of the inhibitory cells
– 20% on average
– 50% of the inhibitory cells are within layer 4
↔ layer 3lower
Laminar organization
Primary visual cortex
Layout of the horizontal connections
• Layer 2/3 excitatory connections, incl. L-R
–
–
–
–
Patchy (biased towards the iso-orientation domains)
Patches are found up to ~2.5 mm (in cat)
Patches vary in size and shape
Elongated along the orientation axis
• Layer 4 excitatory connections
– Up to ~1.5 mm (in cat)
– Equally distributed and isotropic
• Inhibitory connections
– Large basket cells ~1 mm. Small basket, chandelier and bipolar cells
<100 µm.
– Equally distributed and isotropic
Hierarchical organization of the areas
Communication through burst and
spike synchronization
• How: Burst or near-zero phase lag spike
• Where:
– Early stages of vision
• Retina LGN Layer 4
– Within a patch, such as the visual cortex
• Layer 4  layer 2/3
• Within layer 2/3 and between the hemispheres (mediated by the
reciprocal L-R connections)
– Between two (or more) locations in neocortex
• Each location might represent a specific modality/feature.
• Why:
– Signal enhancement, e.g. in low-contrast conditions
– Figure-ground separation
L-R connections account for
spike and burst synchronization
• Robust burst synchronization
• Zero-lag spike synchronization despite the delays
– Tighter between neurons within a minicolumn
Population activity
Cross-correlation between
minicolumns 1 and 6
-0.2
0.2
A generic network model
Long-range connections
A theoretical framework for
multimodal information processing…
… for robots!
Phase 1:
2006 2007
Phase 2:
2008 2009
Current status
• Multimodality
– Vision, speech, gestures
System overview
The video
http://www.youtube.com/watch?v=la2GOTef-1k