Download attention - CMU Graphics

ATTENTION
Akanksha Saran
Krishna Kumar Singh
How do biological beings process
visual information for a specific task
with time constraints?
VISUAL OVERLOAD!
Need to filter out some information..
Attention to the rescue?
Two types of attention
Bottom Up Attention - Visual Saliency
flashing
flashing
lights!
lights!
Top Down Attention
Times
Square
Theatre
Top-down Attention
Feature-based attention in visual cortex
John H.R. Maunsell and Stefan Treue
2006
Top-down Attention
● Task driven
● Two types:
○ Space Based
○ Feature Based
Space-based Top-down Attention
when we are
looking for traffic
light our
attention will be
at top.
Space-based top-down Attention
while
pressing
door bell,
attention
will be 3-4
foot above
ground.
Feature-based top-down Attention
Feature-based top-down Attention
Lemons
Feature-based top-down Attention
Feature Attention example
What role does attention play in
visual Information processing?
Visual Pathway
V4 and V5(MT) have RF large enough to talk about attention.
V4 and MT have large RFs deal with attention
Factors affecting neuron response while performing
visual recognition in the brain
●
●
●
Receptive Field
Feature Selectivity
Attention
Receptive Field
Receptive
Field
Neuron
Field of
View
Feature Selectivity
Stimulus
Response
Neurons are selective to orientation (features).
They have some preferred stimulus.
Feature Selectivity
Preferred
Stimulus
Stimulus
Response
Neurons are selective to orientation (features).
They have some preferred stimulus.
Feature Based Attention Experiments
Visual Search
●
●
●
Help to study feature based attention.
Ability to detect target (cue), among the distractor items.
Distractor and target differ by at least one feature.
find the circle with red color
find the vertical rectangle with green color
Visual Search
●
●
Target : Features we are searching.
Distractor : Features other than target features.
We have to find the circle
with red color.
Target is red color.
Distractor is green color.
Target can be associated with attention.
Visual Search Experiment For V4
Monkeys were trained to
search for a target with a
particular color or a
particular shape in a
crowded display
For example:Find the shape with red color
Visual Search Experiment For V4
Monkeys were trained to
search for a target with a
particular color or a
particular shape in a
crowded display
For example:
Find the shape with red color
Point at which subject is
looking
Shape visible in receptive field
of neuron
Visual Search Experiment For V4
●
There were multiple trials of the experiment.
●
In some trial preferred stimulus and target were same.
○ For example:- We are looking for red color and preferred stimulus of
neuron is also red color.
●
Whereas in some trial preferred stimulus and target differed.
○ For example:- We are looking for red color but preferred stimulus of
neuron is green color.
Feature based attention during
visual search
NEURAL RESPONSE
NEURAL RESPONSE
RESULTS
HIGH
LOW
●
Preferred Stimulus
(without attention)
●
Non-Preferred Stimulus
(without attention)
●
Preferred Stimulus +
Target ( with attention)
●
Preferred Stimulus +
Distraction
(without attention)
Preferred Stimulus + Attention gives maximum neuron response
Motion Based Attention
●
Till now, we have seen feature based attention using shape and color.
●
Feature based attention could be attending to a specific type of motion.
●
MT has neurons specific to motion.
Feature (Motion) Based Attention In MT
●
MT is sensitive to visual motion.
●
Two patches of random dots were
presented.
●
The patches always moved in the
same direction (white arrows).
●
one within the receptive field of the
neurons being record (broken white
line).
●
the attention is shown by gray
arrows.
Feature based attention in MT
●
preferred direction responses
were on average 13%
stronger.
Feature based attention in MT
Push
●
Pull
Attention has push-pull effect
(increases responses only for
neurons that prefer motion
close to the attended direction)
NEURAL RESPONSE
NEURAL RESPONSE
RESULTS (Motion)
HIGH
LOW
●
Preferred Direction
(without attention)
●
Non-Preferred Direction
(without attention)
●
Preferred Direction
(with attention)
●
Preferred Direction
( without attention)
●
Non-Preferred Direction
( without attention)
●
Non-Preferred Direction
( with attention)
Preferred Direction + Attention gives maximum neuron response
Attention amplifies the inherent response of the neuron.
Summary
● Two methods of Top-Down attention
1) Space-based
2) Feature-based
● Attention increases the sensitivity of neurons!
But what mechanism in the brain allows for this
increased sensitivity of neurons with
attention?
Interacting Roles of
Attention and Visual
Salience in V4
John H. Reynolds, and Robert Desimone
2003
Neuron Sensitivity
What can modulate the response of neurons?
●
●
●
●
Receptive Field
Feature Selectivity
Top-down Attention
Visual Saliency such as
Contrast (Bottom-up
attention)
Visual Salience
Contrast - Visual Salience
Contrast - Visual Salience
Impact of contrast on neural activity
Experiments - Stimuli
50 V4 neurons from two monkeys
Reference stimulus : preferred stimulus of the neuron population
Probe stimulus
Pair of stimuli
:
:
non-preferred stimulus of the neuron population
in the same RF
Experiments - Task
Monkeys trained to attend on cue box
Task: detect diamond shaped target
Two trials:
1. attend right (away from RF)
2. attend left (RF)
Experiments - Task
Monkeys trained to attend on cue box
Task: detect diamond shaped target
Two trials:
1. attend right (away from RF)
2. attend left (to RF)
Preferred Reference Stimulus
● high response of neuron population
for reference stimulus
Non-preferred Probe Stimulus
● Low response of neurons for probe
● As contrast increases response
higher
● Higher contrast creates bottom-up
attention
Non-preferred Probe Stimulus
● Low response of neurons for probe
● As contrast increases response
higher
● Higher contrast creates bottom-up
attention
Non-preferred Probe Stimulus
● Low response of neurons for probe
● As contrast increases response
higher
● Higher contrast creates bottom-up
attention
What happens when both stimuli are within
the same receptive field of the neuron?
What happens when both stimuli are within
the same receptive field of the neurons?
●
fires a lot for the reference stimulus (preferred)
What happens when both stimuli are within
the same receptive field of the neuron?
●
fires a lot for the reference stimulus (preferred)
●
increasing contrast of the probe stimulus creates
bottom up attention that the neuron responds to as well
What happens when both stimuli are within
the same receptive field of the neuron?
How should the neuron
respond when 2 things
are competing for the
neuron’s response
within the space of the
receptive field?
Probe and Reference in same RF
●
Initially weak probe with low contrast
has no impact
●
As contrast of probe increases,
the net effect is similar to probe
●
A non-preferred probe suppresses
the response of preferred reference
stimulus
Probe and Reference in same RF
●
Initially weak probe with low contrast
has no impact
●
As contrast of probe increases,
the net effect is similar to probe
●
A non-preferred probe suppresses
the response of preferred reference
stimulus
Probe and Reference in same RF
●
Initially weak probe with low contrast
has no impact
●
As contrast of probe increases,
the net effect is similar to probe
●
A non-preferred probe suppresses
the response of preferred reference
stimulus
What just happened?
HIGH
high contrast for weak stimuli
NEURAL RESPONSE
NEURAL RESPONSE
● For unattended stimuli V4 neurons preferential to high
contrast stimuli
● Contrast modulates neuron sensitivity!
LOW
low contrast for weak stimuli
Now lets go back to top-down attention
Neuron Sensitivity
What can modulate the response of neurons?
●
●
●
●
Receptive Field
Feature Selectivity
Top-down Attention
Visual Saliency such as
Contrast (Bottom-up
attention)
What mechanism in the brain allows for the
increased sensitivity of neurons with attention?
Attention mimics contrast
Attention increases contrast gain of
V4 neurons
●
Response to attended stimulus as though contrast increased
Attention increases contrast gain of
V4 neurons
●
Response to attended stimulus as though contrast increased
Attention increases contrast gain of
V4 neurons
●
Response to attended stimulus as though contrast increased
Attention increases contrast gain of
V4 neurons
●
Response to attended stimulus as though contrast increased
Attention increases contrast gain of
V4 neurons
●
Response to attended stimulus as though contrast increased
Attention increases contrast gain of
V4 neurons
●
Response to attended stimulus as though contrast increased
●
●
Brain hard-wired to respond “preferentially” to highest contrast stimulus
High contrast (without attention) or attention to a stimulus suppresses other
stimuli
Experiments with attention
Two trials:
1. attend left (RF)
2. attend right (away from RF)
Attention - Bias Competition Model
face : preferred
house: non-preferred
● magnified signals from
attended stimuli suppress
unattended stimuli
Attention and contrast together?
Attention and contrast are additive
Suppression is highest with high contrast and attention to probe
Suppression effect
1. contrast
2. difference in selectivity of stimuli
NEURAL RESPONSE
Three sets of neurons:
1. highly selective to reference
2. weakly selective to reference
3. selective to probe
NEURAL RESPONSE
Experiments
NEURAL RESPONSE
Three sets of neurons:
1. highly selective to reference
2. weakly selective to reference
3. selective to probe
NEURAL RESPONSE
Experiments
Experiments
NEURAL RESPONSE
NEURAL RESPONSE
Three sets of neurons:
1. highly selective to reference
2. weakly selective to reference
3. selective to probe
Attention, contrast and selectivity
As attention and contrast for probe increased, pair and probe responses
converge
Top-down attention to probe:
1. reduced pair response by 26.2 %
(highly selective)
2. reduced pair response by 0.8%
(weakly selective)
3. increased pair response by 29.1%
(reverse selective)
Attention v/s contrast
Increasing attention by 100% ~ Increasing contrast by 52-75%
Salience and Visual Search
Lesion studies show that V4 is essential to boost low
salience points with attention in presence of highly salient
stimulus distractors
Salience and Visual Search
Lesion studies show that V4 is essential to boost low
salience points with attention in presence of highly salient
stimulus distractors
Distractor
Target
Conclusion
● Sensation = Perception
● Attention : throws away irrelevant data for a task
● Top-down attention: pull
out a less salient stimulus
from more salient distractors
Discussion
● Role of attentional feedback in Computer Vision
● Task based overruling of standard visual search policies
in computer vision - too early?
In the last two papers we look at Top-down Attention
Other features apart from contrast that can create
bottom up attention?
Learning to Predict Where Humans
Look
Tilke Judd, Krista Ehinger, Frédo Durand, Antonio Torralba
Goal
eye tracking ground truth data
●
●
●
●
15 viewers
1003 images
(LabelMe + Flickr)
Passive viewing
learn a linear SVM to predict saliency maps for other users (passive
viewing)
Analysis of the dataset
Analysis of fixation locations
Common Fixations
Common Fixations
Common Fixations
On close-up, fixation at
specific part of face
Hand Labelled Data
Faces - 10% Text - 11%
Certain ROI on which user fixates
Learning the model
Combine Top Down and Bottom Up
Features
●
●
●
Low level - intensity, orientation, color contrast
mid-level - gist features
high-level - face, person, car, horizon detectors
●
center prior - 70% of fixations lie within center 25% of image
Features
Comparison of different features
Takeaways
● Center performs well over all images but not different
subsets.
● Every feature contributes, as they perform better than
chance.
● Combining features of different levels gives the best
saliency predictions.
Discussion
● Saliency for visual search in computer vision v/s passive
viewing?
● Other attributes like visual novelty, uniqueness?
● How to use attention during algorithm design to solve
computer vision problems ?
Thank you!
extras
Features and Intensity
❏ features - orientation, color, direction of motion, spatial
frequency
❏ intensity - luminance contrast
● play different roles in the bias competition model
● untuned feature - decreases poor stimulus response,
suppress pair response
● high intensity - increases poor stimulus response,
suppress pair response
Spatial Attention
● Receptive field
● Changes strength of neurons’ response without
changing underlying response properties
● Enhances synchronization of neuronal activity
● Spatial attention will increase the gain of all neurons
whose receptive field overlaps the current attentional
focus, creating an enhanced representation at that
location that is akin to a local increase in contrast
Experiments
Monkeys trained
Reference stimulus - selective to
neuron population
probe stimulus - not selective to
neuron population
Pair of stimuli - in the same RF
Feature-based attention in visual
cortex
Saliency Map
-- homogeneous, if only space attention
an attention map. Such a representation of behaviorally relevant locations might be
activated by knowledge of the environment
-- after considering feature attention, depends
on:-- Receptive field
-- Stimulus selectivity
-- Target feature
Relation between spatial and feature
attention
--feature-based and space-based attentions are
very similar.
-- space can be considered one of the feature
-- may be spatial locations identified by
the animal as behaviorally relevant based on
color luminance.
-- feature-similarity gain model responses would be
enhanced for all neurons whose sensory selectivity matched the current
Topographic organization of feature
-- For spatial location, these requirements are fulfilled by the retinotopic
organization and the well-defined spatial receptive fields in early areas of the
visual pathways.
-- topographic organization of the feature
-- issues
-- number of features
-- number of neurons required
-- lack of understanding about feature representation
-- limits feature based attention study, learn about combination of features.
Average ROC Curve For All Users
90% ground truth fixations within top 20% salient locations
Performance of
single user
compared to
remaining 14
users.
Suppression of unattended stimuli
The RF of V4 is large enough that both attended and
unattended stimuli reach there
Suppression of unattended stimuli
The RF of V4 is large enough that both attended and
unattended stimuli reach there
How does the brain deal with competing signals?
Suppression of unattended stimuli
The RF of V4 is large enough that both attended and
unattended stimuli reach there
How does the brain deal with conflicting signals?
● Attended stimulus suppresses the unattended stimuli in
the same RF by increasing contrast of attended
stimuli
Suppression by weaker probe
As contrast of probe increases, its suppressive effect
increases
Attention and contrast
No suppression when weak contrast of probe without
attention
Attention and contrast
Low suppression with weak contrast of probe with
attention
Attention and contrast
Suppression increases as contrast increases for weak
probe
Attention and contrast
Suppression increases as contrast increases for weak
probe
Attention and contrast are additive