Download On-center off surround ganglion cells

Cognitive Neuroscience
and Embodied Intelligence
Perception - Vision
Based on book Cognition, Brain and Consciousness ed. Bernard J. Baars
courses taught by Prof. Randall O'Reilly, University of Colorado, and
Prof. Włodzisław Duch, Uniwersytet Mikołaja Kopernika
and http://wikipedia.org/
http://grey.colorado.edu/CompCogNeuro/index.php/CECN_CU_Boulder_OReilly
http://grey.colorado.edu/CompCogNeuro/index.php/Main_Page
Janusz A. Starzyk
EE141
1
Motivation
Perception is comparatively the easiest to understand although for many
specific questions there are no clear answers.
General questions:
 Why does the primary visual cortex react to oriented edges?
 Why does the visual system separate information into the dorsal
stream, connected to motion and representation of object locations, and
the ventral stream, connected to object recognition?
 Why does damage to the parietal cortex lead to spatial orientation and
attention disorders?
 In what way do we recognize objects in different places, orientations,
distances, with different projections of the image onto the retina?
2
EE141
Introduction

The purpose of vision: ‘to know what is where’
(David Marr)

The visual perception is far more complicated than simply
taking a picture with digital camera
The camera doesn’t really do anything with this image and
doesn’t have any knowledge about what is stored in the image
3

EE141
Introduction

Knowing what: perceiving features, groups and objects

Studies of human visual perception and neuroscience suggest that
there are many levels of perception.
The human brain appears to process basic visual features, such as
color, orientation, motion, texture and stereoscopic depth.
Neurons are highly tuned to specific features like a line at
particular angle, a particular color, or particular motion detection.
The activity of each neuron represents only a small part of the
visual field. How is the brain able to combine this information
across many neurons?



 The brain is able to organize basic feature elements into organized
perceptual groups.
 Psychologists proposed the Gestalt laws of perceptual grouping, such as the
laws of similarity, proximity, good continuation, common fate and so forth
4
EE141
Perceptual grouping
Grouping by similarity:
White dots grouped with
white dots, squares with
squares
Grouping by proximity:
Here we perceive two
separate groups of dots that
are near each other
Grouping by good
continuation:
On the left we perceive a
single object.
When the same lines are
separated we do not
EE141
5
Functional organization of the visual system

Visual pathway
6
EE141
Visual pathways
Visual pathways:
retina => lateral geniculate nucleus
(LGN) of the thalamus
=> visual radiation
=> area of the primary cortex V1
=> higher levels of the visual system
=> associative and multimodal areas.
V1 cells are organized in ocular dominance columns and orientation
columns, retinoscopic. Simple layer 4 cells react to bands with a specific
slant, contrasting edges, stimulus from one eye. A substantial part of the
central V1 area reacts to signals from fovea, where the density of receptors
7
is the greatest.
EE141
Functional organization of the visual system

The hierarchical organization begins in the retina, passes through the
lateral geniculate nucleus (LGN - part of the thalamus), reaching the
primary visual cortex V1, from where it's distributed further.
8
EE141
Functional organization of the visual system


Objects in environment are projected to the back of the
eye – the retina.
Retina contains millions of photoreceptors
9
EE141
The retina
Two types of light-sensitive photoreceptors
Cones
cone-shaped
less sensitive
operate in high light
color vision
Rods
rod-shaped
highly sensitive
operate at night
gray-scale vision
10
EE141
The retina
http://www.iit.edu/~npr/DrJennifer/visual/retina.html
The signals from photoreceptors are processed by a
collection of intermediary neurons, bipolar cells, horizontal
cells and amacrine cells, before they reach the ganglion cells11
EE141
Retina
 The retina is not a passive camera registering images.
 Crucial rule: enhancing contrasts underlining changes in space and
time, strengthening edges, uniformly lit areas are less important.
 Photoreceptors in rods and cones,
 3-layer network, ganglion cells =>LGN.
Receptive fields: areas, which stimulate a
given cell.
The combination of signals in the retina
gives center-surround receptive fields
(on-center) and vice versa, detects edges.
Each individual field of cells can be
modeled as a Gaussian model, so these
fields are obtained as a difference of
Gaussians (DOG).
12
EE141
On-center off surround ganglion cells
 No
stimuli:
 both fire at base rate
 Stimuli in center:
 ON-center-OFFsurround fires rapidly
 OFF-center-ONsurround doesn’t fire
 Stimuli in both regions:
 both fire slowly
 Stimuli in surround:
 OFF-center-ONsurround fires rapidly
 ON-center-OFFsurround doesn’t fire
13
EE141
On-center off surround ganglion cells
on-center cell
David Hubel & Torsten Wiesel
Received Nobel price for their
discovery of on-center off-surround
cells in retina
http://www.physiology.wisc.edu/yin/public/
EE141
14
On-center off surround ganglion cells
Retina ganglion cells receive both excitatory and inhibitory
inputs from bipolar neurons
 In the figure shown, the ganglion cell receives excitatory
inputs from cells corresponding to the on-center region,
15
and
inhibitory
inputs
from
the
off-center
region
EE141

On-center off surround ganglion cells
Original image


EE141
Image based on edges
Lateral inhibition is important in enhancing neural
representation of edges, where the light intensity changes
sharply and indicate a presence of contours, shapes, or objects.
Uniform parts of a picture are less interesting.
16
On-center off surround ganglion cells
Sometimes this later inhibition leads to a surprising visual
illusions as shown on this figure.
 Notice black dots appear on intersection of white lines.
EE141

17
Retinal ganglion cells

There are two types of ganglion cells in the retina:
 Large magnocellular ganglion cells, or M cells,
carry information about:
– Movement
– Location
– depth perception.
 Smaller parvocellular ganglion cells, or P cells,
transmit signals that pertain to:
– Colour
– Form
– texture of objects in the visual field.
18
EE141
Lateral geniculate nucleus (LGN)

From the eye, retinal ganglion cells send
their axons to a structure in the thalamus
called lateral geniculate nucleus (LGN)

The inputs from the
nasal portion of each
retina must cross at
the optic chiasm to
project to the
opposite LGN
19
EE141
LGN pathway




The M cells send their
information to layers 1 & 2
of LGN.
The P cells send their
information to layers 3-6.
So, layers 3-6 are involved
in processing information
concerning fine detail and
color.
Layers 1 & 2 process
information concerning
movement.
20
EE141
Pathways in visual system


Propagation of the visual
input from the left and right
visual fields.
Signals propagate through
eye, retina, optic nerve,
chiasm, optic tract, LGN to
visual cortex V1
21
EE141
Lateral geniculate nucleus
 Signal compression – partly already done in the retina.
 Different types of information find their way to different LGN layers.
 Intermediate station – all sensory signals (except olfactory) go
through different nuclei of the thalamus.
 Dynamic information processing: steering attention and fast largecelled pathway reacting to motion.
 Retroactive projections V1=>LGN are an order of magnitude more
numerous than LGN=>V1 (role - prediction).
The competitive dynamic
selects signals from the visual
field, especially involving
motion.
22
EE141
LGN of the Thalamus
Parvocellular
layers 3-6
Magnocellular
layers 1& 2
23
EE141
Primary visual cortex V1





Neurons in V1 are sensitive to a whole host of visual
features, not seen in the LGN, like orientation, direction of
motion, color differences, or binocular disparities.
Orientation helps to detect edges and contours.
Direction of motion is important to determine dangerous
moves of an attacker.
Color helps to differentiate and identify objects particularly
in a camouflage environment.
Binocular disparities between images in two eyes allow us
to perceive stereo-depth when we look at object with both
eyes.
24
EE141
Primary visual cortex V1
Neurons in V1 respond with different strength to orientation edges,
depending on location of their receptive fields.
 Neuron’s response is strongest if the excitation aligns with its
25
receptive
field.
EE141

Edge detectors
Contrasting signals points from the LGN are organized by the V1
cortex into edge detectors oriented at a specific angle.
Simple V1 cells combine into edge detectors, enabling the
determination of shapes, other cells react to color and texture.
Properties of edge detectors: different orientation;
high frequency = fast changes, narrow bands;
low frequency = gentle changes, wide bands;
26
polarization
=
dark-light
or
vice-versa,
dark-light-dark
or
vice-versa.
EE141
Representation in the V1 cortex
Oriented edge detectors can be created by correlational Hebbian learning
based on natural scenes.
What happens with information about color, texture, motion?
27
EE141
Retinoscopic maps in V1
The spatial position of the ganglion cells within the retina is preserved by
the spatial organization of the neurons within the LGN layers. The
posterior LGN contains neurons whose receptive field are near the fovea.
28
EE141
Area V1: The Primary Visual Cortex


V1 is made up of 6
layers (no relation to
6 layers in LGN).
LGN sends axons to
layer IV of V1.
 M and P cells are
separate.
 Right and Left eye
are separate.
29
EE141
Hierarchy of visual processing
From
retina, LGN, V1,
through V4 and to ventral
temporal cortex (VTC) neurons
gradually respond to more
complex stimuli:
Retina
and LGN extract
small dots
In V1 small dots are
combined into edges,
In V4 edges are combined
into simple shapes and color
features
In VTC simple shapes are
combined into objects
30
EE141
Extrastriate visual areas – outside of V1
Flattened map of
higher visual areas


V1 sends signals to many higher visual areas,
including areas such as V2, V3, V4 and motionsensitive area MT.
Area V4 is important for the perception of color and
some neurons in V4 respond to more complex
features or their combination (like corners or curves).
The middle-temporal area (MT), is important for motion perception.
 Almost all of the neurons in MT are direction-selective, and respond selectively
31
to
a
certain
range
of
motion
directions
or
patterns
of
motion.
EE141

The ventral and dorsal pathways:
knowing what and where where

The projections from
V2 to higher areas in
the cortex can be
roughly divided
according to two
major parallel
pathways: a ventral
pathway to temporal
lobe (what) and a
dorsal pathway to
parietal lobe (where)
what

The ‘what’ pathway includes
ventral areas like V4, LOC, and IT
32
EE141
Two streams where?/what?
“where” pathway
“Where" = large-celled pathway,
heading for the parietal lobe.
orientation
direction
depth
Recognized
Object ready
for perception
shape
color
“what” pathway
"What"= small-celled pathway
heading for the temporal lobe (IT).
33
EE141
The ventral and dorsal pathways:
knowing what and where

In the dorsal pathway,
signals from V1 travel to
dorsal areas like MT and
V3A, which then send major
projections to many regions
of the parietal lobe.

In the ventral pathway,
many signals from V1 travel
to ventral areas V2, V3 and
V4 and onward to many
areas of the temporal lobe.
EE141
34
Dorsal and ventral pathways in a monkey brain
Two streams where?/what?

Milner and Goodale
(1995): visual
pathways don't so
much determine
where and what, as
much as they enable
action and perception.

There is also the old
limbic pathway,
enabling rapid action
in dangerous
situations (after which
follows a wave of
fear).
Where?
- parietal lobe
What?
- temporal lobe
35
EE141
Two streams
Ungerleider and Mishkin (1982):
there exist two notably divided
pathways for processing visual
information, running from the
eye.
Large-grained PA retina cells, 3
types of photoreceptive cones,
large receptive fields, rapidlyconducting axons, activation for
light in a wide band.
Small-grained PB cells, 1 or 2
types of photoreceptive cones,
small receptive fields, slowly
conducting axons, recognize
color oppositions.
Large-celled pathway: runs to two large-celled LGN layers, it's characterized by a
low spatial resolution, high sensitivity to contrast, rapid signal transfer, without
information about color. The small-celled pathway has 4 small-grained layers in
the LGN, high spatial resolution, color, slower information transfer, low sensitivity
36
to
contrast.
EE141
Dorsal pathway
Large-celled pathway:
from the occipital lobe through
the dorsal pathway to the
parietal cortex. Arrives at the
4B layer in V1, from here to the
thick dark stripes of the V2
region, analyzes information
about object motion.
In V1, layer 4B => V5,
localization in the field of vision,
motion.
V5 stimulates the parietal lobe,
PPC (posterior parietal cortex),
regions 7 and 5; this enables
spacial orientation, depth and
motion perception(eye
orientation).
37
EE141
Ventral pathway
Small-celled pathway:
the ventral pathway, to the
inferior temporal cortex.
V1 => V2 interblob region,
reacts to line orientation, gives
a large visual acuity, without
color.
V1 => V3 blob region, reacts to
shapes, reaction to color in the
neurons in the dark stripes of
V3.
V2 => V4, main area of color
analysis, information arrives at
the inferior temporal cortex (IT).
The IT area in the inferior
temporal lobe has neurons
which react to complex objects.
38
EE141
Model v1rf.proj.gz, Chapter 8
How receptive fields are formed in V1?
Inputs: 12x12, signals from LGN cells
on (pos) and off (neg) center.
Input samples: randomly selected
parts of 24x24 from 4 600x800
natural pictures.
Hidden layer 14x14;
links: random excitatory connections.
Project description in chapter 8.3.2. Natural shapes and textures lead to specific
39
receptive fields: from here reactions to edges.
EE141
Areas involved in object recognition
Neurons in V1 respond to
scrambled pictures of a cat
equally well or even stronger
While neurons in lateral
occipital (LOC) respond much
less to scrambled pictures than
to a picture of a cat
Human neuroimaging studies have revealed many brain
areas involved in processing objects.
 These object-sensitive areas, which lie anterior to visual
areas V1-V4, respond more strongly to coherent shapes and
objects, as compared to scrambled, meaningless stimulus.
EE141

40
Areas involved in object recognition

Lateral occipital complex (LOC)
 The lateral occipital complex seems to have a general role in object
recognition and responds strongly to a variety of shapes and objects.
 Presumably neurons in this region respond best to different kinds of objects.

Fusiform face area (FFA)
 Human neuroimaging studies have shown that there is a region in the
fusiform gyrus, called the fusiform face area (FFA) that responds more
strongly to faces than to just about any other category of objects.
 This region responds more to human, animal and cartoon faces than to a
variety of non-face stimuli.
 Neurons in this area specialize in facial expression, particular identity or
viewpoint (e.g. profile)

Parahippocampal place area (PPA)
 The parahippocampal place area is another strongly category-selective
region that responds best to houses, landmarks, indoor and outdoor scenes.
 This area responds weakly to faces, body parts, and animals.
41
EE141
Hierarchical and interactive theories of vision

Is it possible to invoke visual
experience without seeing?
 Yes we can bypass retina and
LGN and stimulate area V1.
 However, it seems impossible
to recover full visual
experience from higher visual
areas bypassing primary visual
cortex.

Hierarchical and interactive theories of vision
 According to hierarchical theory, visual consciousness is organized in a
hierarchical fashion with increasingly higher visual areas being more
closely related to our internal conscious experience. But if this is the
case how to explain awareness of all details in the observed image?
 The interactive theory of visual consciousness emphasizes interactions
between lower and higher visual areas where higher areas send feedback
signals down to early visual area.
42
EE141
Some answers
 Why does the primary visual cortex react to oriented edges?
Because correlational learning in a natural environment leads to this
type of detector.
 Why does the visual system separate information into the dorsal
pathway and the ventral pathway?
Because signal transformations extract qualitatively different
information, strengthening some contrasts and weakening others.
 Why does damage to the parietal cortex lead to disorders of spatial
orientation and attention (neglect)?
Because attention is an emergent property of systems with
competition.
 How do we recognize objects in different locations, orientations,
distances, with different images projected on the retina?
Thanks to transformations, which create distributed representations
based on increasingly complex and spatially invariant features.
43
EE141
Model v1rf.proj.gz, Chapt. 8
How do receptive fields form? Where do these V1 properties come from?
Inputs: 12x12, signals from LGN
cells: on- and off-center.
Input images: randomly chosen
from a natural 512x512 scene.
Hidden layer 14x14; connections:
coincidental with the input,
excitatory with the surroundings.
Description of the project in Chapt. 8.3.2. Natural shapes and textures
44
lead
to
specific
receptive
fields:
from
this
come
reactions
to
edges.
EE141
Model properties
The V1 cortex receives from the LGN an on/off signal with heightened
contrast, input to V1 through layer 4, processing in this model responds
to overlapping processes mainly in layers 2 and 3.
The model includes one hypercolumn, analyzing a small sector of the
image from images of landscapes and plants => all elements see the
same thing.
Properties: spherical geometry, i.e. top = bottom, left = right;
independent inputs for on/off cells, in accordance with biology;
strong and widespread excitatory horizontal connections – like in SOM;
kWTA leaves ~10% active neurons.
Contrast for weights is small ~1, because these aren't decision-making
neurons, thresholds are large (~2) to force sparse representations,
strong correlations.
Noise helps in avoiding weak solutions.
EE141
45
Exercises from v1rf
Check the structure, connection weights (r.wt): strong activations within
the hidden layer, random connections with on/off inputs.
LoadEnv to load the 512x512 image - for the training 10 images were
used, here is one random one, processed into on/off points.
StepTrain – observe the oscillation of learning for phases – and +
Complementarity of on/off: stronger "on" activation for images which are
brighter in the middle than on the edges, dark = extra "off" activation.
Question: what can we expect if horizontal connections will dominate?
Check your predictions, temporarily changing lat_wt_scale 0.04 => 0.2.
LoadNet to load the trained network, after 100,000 presentations of
images and several days of calculations...
46
EE141
Receptive fields
How do receptive fields form? Where do these V1 properties come from?
Check the activation r.wt, change the color
scale so we can better see the field
orientation, check several hidden
elements, bi- and tri-polar fields of both
types.
Load all: View, RFIELDS
activation on=red, off=blue.
Orientation, position, size, polarity are 4
different traits of receptive fields.
Radial orientation changes (pinwheels),
singular points.
View, PROBE_ENV shows 4 different probe stimuli, StepProbe will
47
show
activation
of
hidden
units.
EE141