Download Chapter 5 Learning to attend in primary visual cortex

Chapter 5
Chapter 5
Learning to attend in primary visual cortex
Liviu Stănişor, Chris van der Togt and Pieter R. Roelfsema
1
Chapter 5
ABSTRACT
The attentional selection of relevant visual objects at the expense of irrelevant ones is
associated with the modulation of neuronal activity in early visual areas. It is unknown if
these attentional selection signals are immediately present if a new task is learned. In this
study we investigated the effects of learning on neuronal activity in area V1 of monkeys
with an icon-selection task. In a two-choice paradigm, the monkeys had to learn to
differentiate between two small images (icons) and choose the relevant one by making a
saccade toward a target connected to the icon by a curve. Neural activity was recorded
from neurons with receptive fields on the curves so that the receptive field stimulation
was constant across trials. At the start of learning, the monkeys performed at chance level
and the representation of the selected curve was enhanced over the representation of the
non-selective curve at an early point during the trial. Performance improved with
learning, and this behavioral improvement was associated with in increased delay in the
appearance of the attentional selection signal. Learning the identity of the target icon
caused a paradoxical suppression of the curve in the vicinity of the icon, which caused a
delay in the attentional selection of the curve. These results provide a new and
unexpected insight in processes responsible for the learning of multi-step tasks.
INTRODUCTION
The primary visual cortex is the first cortical stage of visual information processing. Neurons
in area V1 contribute to the detection of elementary visual features like color or orientation,
but their activity is also modulated by the relevance of visual stimuli (Roelfsema et al. 1998;
Poort and Roelfsema 2009). During tasks that consist of several mental operations, neurons
in area V1 that code image elements relevant for current operation increase their activity so
that it is possible to monitor time-course of a sequence of cognitive steps in this area (Moro
et al. 2010; Roelfsema et al 2003). One example of this is the search-then-trace task that we
recently devised. In this task, subjects first carry out a visual search for a specific marker and
they then have to trace a curve that starts at that marker. In previous work, we recorded
2
Chapter 5
from area V1 in this task and we found that neurons that coded the marker that had to be
searched increase their activity at an early point in time during the trial whereas neurons
that represented the curve to be traced enhanced their activity after a delay. Thus it is
possible to monitor the precise time-course of a sequence of cognitive operations in area
V1. The modulation of neuronal activity in early visual areas during curve-tracing and visual
search is presumably responsible for the shifts of visual attention that subjects make during
these tasks (Kim and Cave 1995; Houtkamp et al. 2003).
In the previous chapter of this thesis, we investigated the neuronal activity in
area FEF in a variant of the search-then-trace task where monkeys had to learn the identity
of the icon that denoted the start of a target curve that had to be traced. In the early phase
of the learning session, the monkeys did not yet know the identity of the target icon and
they therefore had to make decisions with uncertain evidence. We found that the activity of
neurons in FEF that coded for one of the possible decisions quickly ramped up and the
monkeys made fast but random decisions. When the monkeys started to learn the meaning
of the icons, we found that the increase in FEF activity towards a saccade was delayed as if a
time-consuming visual search operation was interspersed before the monkeys reached their
decision.
Visual search and curve-tracing presumably rely on interactions between the frontal
and visual cortex. Models of visual search propose that a feature-selective signal is fed from
frontal areas back to the visual cortex to enhance the activity of neurons with a receptive
field on the target of search (van der Velde and de Kamps 2001; Hamker 2005). Similarly,
the neuronal correlates of curve-tracing emerge in visual and frontal cortex at
approximately the same time after the presentation of the stimulus (Khayat et al 2009),
which suggests that curve-tracing also relies on the reciprocal interactions between these
cortical regions. As a direct test of the coupling between frontal and visual cortex a few
studies increased the activity of neurons in the frontal cortex with microstimulation (Moore
and Armstrong 2003, Ekstrom et al 2008) or local injection of drugs (Noudoost and Moore
2001) and observed an increase in the activity of neurons in visual cortex that code the
same location in space. We therefore considered the possibility that the change in the
activity in area FEF in the icon-learning task would be associated with changes in activity in
visual cortex.
3
Chapter 5
Previous studies on the effects of learning on activity in visual cortex usually
investigated the effects of extensive training protocols where animals were trained across
many days (Kobatake et al. 1998; Schoups 2001; Schwartz et al 2002; Lee et al 2002;
Furmanski et al 2004; Sigman and Gilbert 2005; Kourtzi et al 2005; Li et al 2008). To our
knowledge, it has not yet been investigated if and how neuronal responses in the visual
cortex change as a result of learning within a single recording session. During a single
learning session, neurons in frontal cortex change their responses (Chen and Wise 1996;
Asaad et al 1998; Yotsumoto et al 2008 and 2009). Are these learning effects in frontal
cortex associated with altered attentional response modulation in visual cortex?
In the present study we recorded from area V1 in the learning version of the searchthen-trace task to investigate a number of questions. First, in previous work the modulation
of the neuronal activity by task relevance in early visual areas was investigated in welltrained tasks. It is therefore unknown if random decisions are reflected by a modulation of
neuronal responses in early visual cortex. Second, one previous study demonstrated that
the attentional response modulation required several days to develop in area V1 in a task
where the animal had to learn to integrate a path of collinear contour elements (Li et al
2008). However, in that task the performance of the animal also improved gradually over
several days. Will the attentional modulation of neuronal activity in primary visual appear
within a single learning session of the icon-selection task? Third, we wanted to investigate
the source of the increased delay in target selection in the icon-selection task where
learning caused longer response times and later response modulations in frontal cortex.
Alterations in the attentional modulation might give insight in the transition from the early
phase with fast and random decisions to the later phase where decisions are informed by a
successful visual search.
MATERIALS AND METHODS
Two monkeys (A and G) participated in this study. In a first operation, a head holder was
implanted and a gold ring was inserted under the conjunctiva of one eye for the
measurement of eye position. In a separate operation, arrays of 4x5 or 5x5 electrodes
(Blackrock Microsystems) were chronically implanted in area V1. The surgical procedures
4
Chapter 5
were performed under aseptic conditions and general anesthesia. Details of the surgical
procedures and the postoperative care have been described elsewhere (Roelfsema et al
1998, 2007). All procedures complied with the US National Institutes of Health Guidelines
for the Care and Use of Laboratory Animals and were approved by the institutional animal
care and use committee of the Royal Netherlands Academy of Arts and Sciences of the
Netherlands.
The stimuli were presented on a monitor with a diagonal of 35.5 cm (14 inch),
a resolution of 1024 by 768 pixels, a refresh rate of 100 Hz, and positioned at a distance of
75 cm from the monkey's eyes. The objects that appeared on the screen were colorful
figures that were approximately 0.8° in diameter; we will refer to these as ‘icons’ (Figure 1).
The saccade targets were red discs with a radius of 0.8° connected to the icons by a curve
that was two pixels wide (0.05°). The eccentricity of the saccade targets was 3.5° for monkey
A and 4° for monkey G.
Behavioral task
The aim of the icon selection task was to induce learning by presenting a new set of stimuli
each day. A trial started as soon as the monkey’s eye position was within a 1.2°×1.2° square
window centered on a red fixation point (FP in Fig. 1a). After an interval of 300 ms, the
stimulus appeared on the screen. It consisted of two curves starting at two different small
icons. The monkey was presented with a new pair of icons each day, one of which was
relevant and the other was to be ignored. The curve segment in the receptive fields of the
recorded neurons was always the same so that differences in activity between conditions
were caused by effects on the neuronal firing rates from outside the receptive field.
There were three different types of trials: easy, intermediate and difficult (Fig. 1b). In
the case of the easy stimuli, the relevant icon appeared on top of the fixation point whereas
the other object appeared at a distance from the fixation point. For the stimuli of
intermediate difficulty, neither icon overlapped with the fixation point, but the relevant one
was closer to the fixation point than the irrelevant one. The difficult stimuli had both icons
equally spaced from the fixation point so that the monkey could not use positional cues and
5
Chapter 5
had to rely on the identity of the relevant and irrelevant icon. Great care was taken to use
different cues (combination of shapes, contour and filling colors etc.) so that the relevant
icon was distinct from the irrelevant one and the pairs of icons differed across days. In
order to promote learning, the stimuli of the three difficulty levels were interleaved so that
the animals could use the information from trials with an easy and intermediate difficulty
while they increased their performance for the difficult ones. After the stimuli were
presented, a delay of 500 ms followed, after which the fixation point disappeared (go cue)
and the monkeys could indicate their choice by making a saccade to one of the saccade
targets and they then progressed to the next trial. We used this paradigm in order to
investigate (1) differences in neuronal activity between levels of difficulty and (2) the
changes in neuronal activity induced by learning.
We considered the learning session successful if the accuracy for the difficult stimuli
was 75% or better at the end of each recording session. An average number of 281 trials per
recording session were recorded (difficult stimuli) for monkey A and 136 trials for monkey
G. We recorded behavioral data across 22 sets for monkey A as well as monkey G (44 sets in
total) and neuronal data across 19 sets for monkey A and 22 sets for monkey G (41 sets in
total).
Recording of multiunit activity
Spiking activity was recorded from the chronically implanted multi-electrode arrays
(Cyberkinetics Neurotechnology Systems Inc.) with TDT (Tucker Davis Technologies) multichannel recording equipment. As in previous studies (Logothetis et al. 2001 Supèr and
Roelfsema 2005; Xing et al. 2009; Yeh et al 2009; Pooresmaeili et al. 2010;), the MUA signal
was amplified, band-pass filtered (300-9000Hz), full-wave rectified, low-pass filtered
(<200Hz) and sampled at a rate of 763Hz. MUA recorded in this manner gives an
instantaneous measure of spiking activity of neurons around the electrode tip. Previous
studies compared MUA to single-unit data in a curve tracing task and found that it yields a
reliable estimate of the average single-unit response (Supèr et al 2005, Cohen and Maunsell
2009). For every recording site we calculated the signal-to-noise ratio by dividing the
maximum of the average evoked response to the standard deviation of the spontaneous
6
Chapter 5
activity in a 200 ms window before stimulus onset. We included recording sites with a signal
to-noise ratio between 0.28 and 11.11, with a mean of 3.91.
The receptive field dimensions of the neurons at the recording sites were estimated
using the onset and offset of the visual response to a slowly moving white bar on a black
background, in each of eight possible directions (Supèr & Roelfsema, 2005). The median
area of the receptive fields was 0.52 deg2 (ranging from 0.1 deg2 to 2.4 deg2). Receptive field
eccentricity ranged from 1.21° to 5° with an average of 3°. The stimulus configuration was
devised in such a way that all the RFs fell onto one of the curves and they never overlapped
with the discs or with the icons.
Analysis of neuronal responses
We recorded neuronal data across 19 sets for monkey A and 22 sets for monkey G (a total
of 41 sets). We computed peri-stimulus time histograms (PSTHs) in a time window ranging
from 100 ms before stimulus onset and up to 800 ms afterwards and normalized MUA
activity to the peak response after subtraction of the spontaneous activity (average activity
in the last 100 ms before stimulus onset) and smoothing with a Gaussian kernel (standard
deviation 10 ms). The peak response was determined as the difference between the highest
and lowest activity in an interval between 0 and 150 ms after the stimulus onset.
When analyzing differences in modulation between conditions, we used a 200 ms
computational time window starting at 200 ms after stimulus onset unless otherwise
specified.
Latency analysis
Attentional response modulation reflects differences in neuronal firing rate between
relevant and irrelevant stimuli. Methods to quantify the onset of the modulation of activity
start by taking the difference between activity evoked by target and distractor stimuli. To
compute the latency of attentional modulation we used a method based on the cumulative
7
Chapter 5
sum of this response difference. This sum is a function of time and represents the integral y
of modulation from stimulus onset (t=0) up to each point in time.
τ =t
y (t ) = ∑ x(τ )
τ =0
where x(τ ) is the response modulation at time τ . One advantage of this method is that it
averages out noise which leads to smooth graphs (Bellman and Roth 1969; McGee and
Carleton 1970). The slope of the function y(t) reflects the average modulation in the time
period of interest and changes in this angle can be used to determine the onset of
modulation. We used piecewise linear fitting to estimate average modulation magnitude in
different epochs and used the joints between segments to estimate the change points. We
fitted two lines; the first had a zero slope, and the second a positive slope using the shape
prescriptive modeling toolbox implemented in MATLAB by John D’Ericco (Mathworks 2009).
When computing the cumulative sum of the modulation, we considered the neuronal
activity until the average reaction time. Channels with poor fits (which produced latencies
smaller than 50 ms after stimulus onset) were excluded from the analysis (1 site in monkey
G and 2 in monkey A).
RESULTS
We trained two monkeys to select one of two small icons using the icon-selection task with
three levels of difficulty. The monkey started a trial by direction gaze to a fixation point (FP
in Fig. 1a). After a delay of 300 ms, we presented two curves, two discs and two icons one of
which was the target icon. When monkey had identified the target icon, he had to trace the
curve connected to this icon to localize the target disc as the goal of a saccade. After an
additional delay of 500 ms, the FP disappeared which was the go cue for the monkeys to
make the saccade. Saccades to the disc connected to the target icon were rewarded with a
drop of apple juice.
8
Chapter 5
Every day we presented a new pair of icons that had not been seen by the monkey,
and the animal had to learn the identity of the target icon by trial and error. We facilitated
learning with positional cues (Fig. 1b) because the target icon was closer to the fixation
point than the distractor icon in two of the three difficulty levels. In the easy condition the
target icon was placed on top of the fixation point, in the intermediate condition it was
closer to the fixation point than the distracter icon, and in the difficult condition both icons
were at an equal distance from the fixation point. Therefore, the monkey could solve the
task by selecting the icon closest to the fixation point at two of the difficulty levels. Either
curve could be target or distractor at each of the three difficulty levels so that there were a
total of six stimuli that were randomly interleaved.
Behavioral measures of learning
We obtained behavioral data from a total of 44 recording sessions with both monkey A and
monkey G. Figure 2 shows how their accuracy and the reaction time changed during the
behavioral task. The monkeys hardly made any errors for the easy and intermediate stimuli
and we therefore did not observe changes in accuracy in these conditions. In the difficult
condition, the accuracy was initially close to 50% (chance level) and it reached a maximum
higher than 90% for monkey A and close to 100% for monkey G. The learning curve of
monkey G was somewhat steeper than the learning curve of monkey A. To investigate if the
learning was significant, we compared the accuracy between early trials (trials 1-30) and late
trials (121-150 in Monkey A; 61-90 in Monkey G). The increases in accuracy in the difficult
condition were highly significant in both animals (Monkey A:χ2=95.3, df=1, P<10-6; G:
χ2=60.6, df=1, P<10-6, Chi-Square test).
The monkeys had to maintain fixation for 500ms after stimulus onset and were
then cued to make a saccade. In spite of this fixation requirement we found that the
reaction times differed between difficulty levels and changed with learning. We analyzed
the average reaction times with a two-way ANOVA with factors ‘difficulty’ and ‘epoch’. For
the factor epoch we chose three intervals that differed between the two monkeys because
they differed in learning speed: early (trials 1-30), intermediate (61-90 in Monkey A, 31-60 in
Monkey G) and late (121- 150 in Monkey A, 61-90 in Monkey G) (grey bars in Figure 2). We
9
Chapter 5
observed a significant main effect of difficulty on reaction time because difficult trials were
associated with longer reaction times in both monkeys (Monkey A, F2,261=34.7, p<10-5;
Monkey G, F2,261=1272, p<10-5).
The effect of learning on the reaction time differed between monkeys, with reaction
time decreasing in monkey G (F2,261=20.5, p<10-5) and slightly increasing in monkey A
(F2,261=3,1, p=0.046). However, the increase in reaction time in monkey A occurred only in
the easy and intermediate condition, as the reaction times decreased in the difficult
condition (592±44ms in the early trials vs. 575±36ms in the late trials, t-test: p>0.05), just as
was observed in monkey G (833±156ms vs. 767±138ms, t-test: p>0.05). In both animals the
response times for the difficult stimuli on error trials were significantly longer than those on
correct trials (both monkeys, p<10-5, t-test).
Neuronal activity in area V1 in the icon-selection task
The stimuli were always configured in such a way that the receptive fields (RFs) of the MUA
recording sites fell either on the target curve or on the distracter curve (Fig. 1a) but not on
the icons. Figure 1c shows the neuronal response of neurons at one example V1 recording
site when the RF fell on the target curve (red response) and when it fell on the distracter
curve (blue response), for easy, intermediate and difficult stimuli. The responses were
obtained by averaging across all correct trials and across all recording sessions (N=19) with
this recording site. It can be seen that the initial visual response triggered by the appearance
of the curve in the receptive field was identical for the two conditions. However, after a
delay of 150-200 ms neuronal responses evoked by the target curve became stronger than
the activity evoked by the distracter curve. Previous studies attributed this response
modulation to the shift of visual attention to the target curve (Roelfsema et al. 1998; Poort
and Roelfsema 2009). It can also be seen that the onset of the response modulation in the
difficult condition is later than in the easy condition.
10
Chapter 5
Effects of task difficulty on neuronal activity in area V1
Figure 3 shows the population response for each monkey (monkey A: upper panels, N=26
recording sites; monkey G: lower panels, N=7 recording sites), averaged across all recording
sessions (monkey A 19 sessions; monkey G 22 sessions). The lower panels show the
response modulation computed as the difference between the activity evoked by the target
and distracter curve. It can be seen that the profile of the population response was similar
to that of the example recording site in Figure 1c. The initial responses did not discriminate
between the target and the distractor curve, and the attentional response modulation
occurred after a delay.
The temporal profile of the attentional modulation differed between the difficulty
conditions. In the easy and intermediate conditions, the modulation started relatively
abruptly, whereas the increase in modulation was more gradual in the difficult condition.
Figure 4 illustrates the modulation (upper panels) and the cumulative sum of the
modulation (lower panels, see Methods). It can be seen that the modulation is weaker in
the difficult condition than in the other two conditions, and that the latency of modulation
is longer. We analyzed the modulation in the difficult stimuli across recording sites and we
found it significant in case of monkey A (between 200 and 400 ms after stimulus onset, N =
26 sites, p<0.01, paired Wilcoxon signed rank test) as well as monkey G (between 400 and
600 ms, N = 7 sites, p=0.01, paired Wilcoxon signed rank test). Interestingly, we found a
slight inverse modulation in the difficult condition in monkey G that later reversed and then
gradually built up until the onset of the saccade. This reversed modulation might be a
consequence of the visual search. Previous studies demonstrated that search is associated
with an enhanced representation of the icon in visual cortex (Chelazzi et al 2001; Moro et al.
2010) and our result suggest that this selection process is associated with the suppression of
the representation of the adjacent curve. When the monkey starts to trace, however, this
suppression reverses and V1’s representation of the target curve is enhanced.
To investigate the reliability of the differences in the early response modulation
between conditions, we quantified the attentional effect a window from 100-300 ms after
stimulus onset. An ANOVA revealed a significant effect of task difficulty on the magnitude of
the modulation in both monkeys (one-way ANOVA, monkey A: p <10-5, F2,83 = 44.6; monkey
11
Chapter 5
G: p <10-5, F2,20 = 37.8). Early modulation in the difficult condition was significantly smaller
than in the easy and intermediate condition (p < 0.01, Tukey’s HSD test). We did not observe
a significant difference in modulation strength between the easy and intermediate condition
in either monkey (p> 0.05, Tukey’s HSD test).
We measured the latency of the response modulation with a piecewise linear fitting
method on the cumulative sums of modulatory activity (Fig. 4b). Latency is estimated as the
time of the first deflection point in the fit (see Methods for details). Across the individual
recording sites in monkey A, the average latency of the modulation was 115±6 ms (mean ±
s.e.m.) in the easy condition, 126±3 ms in the intermediate condition and 165±11 ms in the
difficult condition. In monkey G these latencies were 100±19 ms, 119±20 ms and 240±42
ms, respectively. We tested the effect of difficulty on modulation latency with a one-way
ANOVA and found a significant effect in both monkeys (monkey A: F2,83=93, p<10-6; monkey
G: F2,20=44, p<10-6). The latency of the response modulation did not differ significantly
between the easy and intermediate condition in either monkey (p>0.01, post-hoc test) but
the latency was longer in the difficult condition in both animals (difficult vs. intermediate
and difficult vs. easy both ps<0.01, Tukey’s HSD).
The effects of learning on neuronal activity in area V1
In order to investigate the effect of learning on the modulation of V1 activity, we
investigated neuronal activity in trials of the three epochs during the progress of learning
(see grey regions in Figure 2 for the selection of trials); an early epoch (random
performance), an intermediate epoch and a late epoch (maximal performance). For both
monkeys we selected 30 trials per condition (15 trials with the target curve in the RF and 15
with the distractor, in each learning phase). In this analysis we first calculated the
cumulative sum of (target–distracter) modulation (Fig. 5a) using the neuronal activity first
averaged across recording days and then across recording sites. We did not observe
consistent effects of learning on the cumulative sum of modulation in the easy and
intermediate conditions. For example, the cumulative modulation in the easy condition in
monkey A rose less steeply in the intermediate epoch than in the first set of trials (early
12
Chapter 5
phase), but it rose more steeply in the last epoch. In monkey G the cumulative modulation
tended to rise more steeply with learning in the easy and intermediate condition.
In contrast, the changes observed in the difficult condition were greater and highly
consistent across animals. The right panels of Figure 5a show the cumulative sum plots of
the response modulation averaged across all V1 recording sites. Learning induced a
suppression of activity evoked by the target curve (reverse modulation), and this effect
became maximal in the last epoch when the animals reached their maximal accuracy. These
results indicate that learning to select the target icon causes a suppression of the adjacent
curve in area V1, and this suppression changes into a response enhancement when the
monkey traces the target curve.
To further characterize these effects, we estimated the latency of (positive)
modulation of individual recording sites during the early (red symbols in Fig. 5a) and late
epoch (blue symbols) in the difficult task. The latency of the modulation in both monkeys
increased with learning (Fig. 5b). The median latency for monkey A (n = 26 recording sites)
was 220±14 ms for the first epoch (trials 1-30) and it increased to 290±9 ms in the late
epoch (trials 121-150), a difference that was significant (p < 0.01, paired t-test). V1 neurons
in monkey G showed the same effect with a median latency of 270±66 ms for the first epoch
and 570±45 ms in the late epoch (p < 0.01, paired t-test).
This increase in latency with learning was associated with an increased suppression
of the activity evoked by the target curve during the initial phase of the trials (reversed
modulation). To measure the suppression, we calculated the y-offset of the best fitting line
used to estimate the latency of the modulation (arrows on the y-axis in Fig. 5a, right panel)
across individual recording sites. Figure 5c illustrates the significant increase in suppression
with learning (both monkeys: p < 0.01, paired t-test). We next investigated if the effect of
learning on the latency on the response modulation was specific for the difficult condition
where learning was most pronounced. We therefore repeated the same analysis in the easy
and intermediate condition. In contrast to the effects in the difficult condition, there was
even a small decrease in the latency of modulation between early and late epoch. This result
indicates that the increase in modulation latency was specific to the difficult condition
where the monkey’s performance increased the most.
13
Chapter 5
Error trials
If the initial suppression of the target curve adjacent to the relevant icon is caused by the
attentional selection of one of the icons, then the neuronal responses should invert on
erroneous trials in the difficult condition where the monkey selects the wrong icon. Figure 6
compares the attentional modulation of the V1 representation of the curve and its
cumulative sum between correct trials (red traces) and trials when the monkey made an
erroneous saccade to the distracter (blue traces), in the difficult condition. On correct trials,
the representation of the target curve was initially suppressed relative to the distractor
curve and enhanced at a later point in time. On error trials, the response modulation has a
vertically mirrored profile, suggesting that the distractor curve’s representation adjacent to
the erroneously selected distractor icon is initially suppressed but that the distractor curve’s
representation is enhanced during the subsequent curve-tracing process. The difference in
the response modulation between erroneous and correct trials was significant in monkey A
(t-test, p<0.01 in a window from 200-400ms) as well as in monkey G (t-test, p<0.01 in a
window from 400-600ms).
DISCUSSION
In the present study we investigated the influence of learning on V1 activity in task where
monkeys had to learn to select a target icon with unknown identity. Initially, the animals
had to make their decisions without knowing the sensory-motor mapping but with time
they learned to select the appropriate response. In our previous study we recorded from
area FEF during this task to study the integration of neuronal activity for the optimal
selection of eye-movements and the influence of learning on this selection process. We
found that the FEF neurons initially reached fast but random decisions that while learning
proceeded changed into more gradual and accurate decisions. The delay in the buildup of
FEF activity that occurred with learning suggested that one of the curves was initially
selected at random, whereas learning the identity of the target icon caused an additional
14
Chapter 5
delay, as if an additional time-consuming visual search operation was added to the mental
routine.
The present results demonstrate that the initial fast process that randomly
selects one of the two curves is also associated with attentional modulation in area V1, so
that the selected curve is represented with stronger activity than the non-selected curve.
The implication is that attentional modulation of neuronal activity in primary visual cortex
does not only occur in well-trained tasks, but even during the initial stages of learning where
the animals explore the stimulus-response mappings. Our results also provide insight into
the source of the delay in the response modulation in area FEF that occurs during the
progression of learning. Learning to select the target icon is associated with a paradoxical
suppression of the representation of the adjacent target curve. Visual search is known to
boost the representation of the target icon in visual cortex (Chelazzi et al 2001) and we have
now observed that it also suppresses the representation of the adjacent target curve. These
results therefore suggest that the neuronal selection process during search has a Mexican
hat profile, with a central attended region surrounded by a ring of suppression, as has also
been observed in other tasks and with different techniques (Serences et al 2004, Hopf et al
2006). The strength of the suppression became stronger with learning and it thereby caused
an increasing delay in the propagation of enhanced activity along the target curve. We
suggest that this suppression process might also be responsible for the delay in the
response modulation that we observed in our previous FEF study.
Previous V1 learning studies
The present study differs from previous studies that monitored the consequences of
learning on activity in visual cortex (e.g. Kobatake et al., 1998, Schoups 2001; Lee et al 2002;
Schwartz et al 2002; Furmanski et al 2004; Kourtzi et al 2005; Sigman and Gilbert 2005; Li et
al 2008). In these previous studies, the subjects were trained in a difficult detection or
classification tasks and it was found that learning increases the neuronal sensitivity to a
specific feature or visual pattern while behavior improved over several days. In the present
study we monitored the V1 representation of the two curves that were held constant and
15
Chapter 5
the changes in activity therefore reflected the changes in the efficiency and timing of the
attentional selection process rather than altered sensitivity to specific visual features.
Comparison between easy and difficult trials
Activity in area V1 provides insight into sequential cognitive tasks because the response
enhancements index the component processes, like visual search and tracing (Roelfsema et
al., 2003; Moro et al., 2010). In the easy and intermediate conditions of the present icon
selection task, the monkeys could solve the task by tracing the curve that started nearest to
the fixation point. Under these conditions, the activity evoked by the target curve became
stronger than the activity evoked by the distractor at a relatively early point in time
(between 100 and 130 ms). In the difficult condition, the monkeys had to first carry out a
visual search and the insertion of this visual search operation caused a delay in the
enhancement of the representation of the target curve in accordance with previous studies
(Roelfsema et al. 2003; Moro et al. 2010). These previous studies did not find a suppression
of the adjacent target curve, as was observed in the present icon-selection task. One
possible difference between the present study and these previous studies was that the
monkeys had to learn the identity of the target icon during whereas the animals carried out
a highly familiar search for a specific color in the earlier studies.
The attentional modulation in the difficult condition was also weaker than that in the
easy and intermediate conditions, in accordance with a study that varied task difficulty in
the curve-tracing task (Roelfsema & Spekreijse 2001). Thus, the neuronal correlates of
attentional selection in area V1 are strongest if the selection of the target curve is easy. One
might therefore have also have expected that learning would lead to enhanced attentional
modulation in area V1. This is not what we observed, but the absence of an increase in
attentional modulation with learning in the difficult condition was presumably caused by
the inhibition caused by visual search, as was discussed above.
Source of the attentional response modulation in area V1
16
Chapter 5
It is unlikely that the primary visual cortex solves the icon-selection task by itself and the
response modulation during search and tracing presumable depends on the interaction
between area V1 and higher visual areas. Models of visual search propose that a
representation of the search target in frontal cortex provides feedback to visual areas to
enhance the activity of neurons that code matching features in the visual image (van der
Velde and de Kamps 2001; Hamker 2005; Zhou and Desimone 2011). This resulting focus of
enhanced activity in retinotopic areas could then be used as a starting point for the tracing
process that selects all contour elements of the target curve and eventually highlights one of
circles at the end of the relevant curve as target for the saccade. Models of tracing have
proposed that neurons in visual cortex propagate increased activity along the
representation of the target curve until it is entirely labeled with enhanced activity
(Grossberg and Raizada 200; Roelfsema 2006). We previously compared the neuronal
correlates of curve-tracing between areas V1 and FEF and found that the attentional
selection of the relevant curve occurs at approximately the same time in both areas (Khayat
et al. 2009), suggesting that curve tracing also requires reciprocal interactions within a
network of areas that includes areas of visual and frontal cortex.
It is unknown where the decision is taken to select one of the icons and to then
trace the curve attached to this icon. Areas in frontal cortex like the FEF and in parietal
cortex like area LIP have been implicated in eye movement decisions (Kim and Shadlen
1999; Platt and Glimcher 1999; Shadlen and Newsome 2001; Gold and Shadlen 2007).
However, it is also conceivable that there are other, perhaps even unexplored regions that
contribute to icon selection and decision making. It is even possible that these decisions
depend on an interaction between multiple areas that together act as an attractor network
that converges into one of a number of attractor states. Studies that change neuronal
activity in frontal cortex observed effects in visual cortex, thereby providing insight in how
these areas interact with each other (Moore and Armstrong 2003; Ekstrom et al 2008). The
finding that V1 also reflects the outcome of the decision process is compatible with an
active involvement of V1 in such an attractor network, but it is equally compatible with
more passive role where V1 receives feedback from higher visual areas that take the
decision.
17
Chapter 5
Conclusion
We conclude that processing in the icon-selection task proceeds from an initial stage where
the monkey takes fast and random decisions that is followed by a stage where the decision
become more accurate. The curve-tracing process becomes delayed with learning because
the monkeys insert a visual search process before tracing, which causes an initial
suppression of the target curve and it therefore takes more time before the target curve’s
representation is enhanced and can be selected as the target of an eye movement in brain
regions that select saccades, including the frontal eye fields.
18
Chapter 5
Figure 1. The icon-selection task and activity at an example V1 recording site. a, The
monkeys were required to fixate a central fixation point (FP) for 500 ms, after which two
19
Chapter 5
icons appeared that were both connected to a disc by a curve. The stimuli were configured
so that either the target curve (T) or the distractor curve (D) fell in the receptive field (RF) of
a group of neurons in area V1. After a second delay of 500 ms, the monkeys had to indicate
their choice by making a saccade to the disc connected to the relevant target icon (blue
arrow). b, Example stimuli in the easy, intermediate and difficult condition. In the easy and
intermediate conditions, the relevant icon was superimposed on the fixation point (easy
condition) or closer to the fixation point than the irrelevant icon (intermediate condition),
and these spatial cues allowed the monkeys to solve the task even if they had not yet
learned the identity of the target icon. c, Activity of an example V1 recording site in the
three conditions. Note that the activity evoked by the target curve is stronger than that
evoked by the distractor curve at each of the difficulty levels.
20
Chapter 5
Figure 2. Accuracy and reaction times of the two monkeys in the icon-selection task. a,
Accuracy (percentage correct) as a function of trial number for trials of the easy (blue),
intermediate (green) and difficult condition (red). The gray bars illustrate bins of 30 trials
(early, intermediate and late during sessions) that were selected for further analysis. b,
Reaction times (measured from stimulus appearance) for trials of the three difficulty levels.
The cyan traces represent reaction times for the error trials.
21
Chapter 5
Figure 3. Averaged activity evoked by the target and distractor curve and attentional
response modulation at three levels of difficulty. Red and blue traces show the population
response averaged over all recording sites and all trials of the easy, intermediate and
difficult stimuli for both monkeys separately. Lower panels show the attentional response
modulation (target minus distracter activity). Note the different scale in the lower right
panel for monkey G.
22
Chapter 5
Figure 4. Average profile and timing of attentional response modulation. a. Modulation for
easy (blue traces), intermediate (green traces) and difficult trials (red traces). b. Cumulative
sums of modulation. The numbers denote the onset time of the attentional response
modulation as estimated with a piecewise linear fitting procedure.
23
Chapter 5
Figure 5. Cumulative sum analysis of modulation during the learning process. a, Time
course of the cumulative sum of modulation for both monkeys. The different shades of grey
denote bins of 30 trials representing early, intermediate and late periods during sessions.
24
Chapter 5
Cyan and red symbols in the right panels show the estimated modulation latencies for
individual recording sites in the first and last epoch during learning, respectively. The dashed
lines show the best piecewise linear fits that were used to estimate the latency of the
attentional response modulation at the population level for the first (magenta) and last
epoch (cyan). b, Latency of modulation across individual recording sites in the first (abscissa)
and last epoch of trials (ordinate) in the difficult condition. c, Degree of response
suppression across individual recording sites in the first (abscissa) and last epoch of trials
(ordinate) in the difficult condition. Note the increase in latency and suppression in the last
trials.
Figure 6. Comparison of V1 activity on correct and erroneous trials in the difficult
condition. a, Attentional response modulation on correct (red traces) and erroneous trials
(blue traces) for both monkeys. The correct trials come from an intermediate phase of the
learning process (Monkey A, trials 61-75; Monkey G, trials 31-45). b, Cumulative sum of the
response modulation on correct and erroneous trials.
25
Chapter 5
REFERENCES
1. Ahissar,M., and Hochstein,S. (1993). Attentional control of early perceptual learning.
Proc. Natl. Acad. Sci. U. S. A 90, 5718-5722.
2. Asaad,W.F., Rainer,G., and Miller,E.K. (1998). Neural activity in the primate prefrontal
cortex during associative learning. Neuron 21, 1399-1407.
3. Bellman,R., and Roth,R. Curve Fitting by Segmented Straight Lines. Journal of the
American Statistical Association 64[327], 1079-1084. 1969.
4. Chelazzi,L., Miller,E.K., Duncan,J., and Desimone,R. (2001). Responses of neurons in
macaque area V4 during memory-guided visual search. Cereb. Cortex 11, 761-772.
5. Chen,L.L., and Wise,S.P. (1995). Neuronal activity in the supplementary eye field
during acquisition of conditional oculomotor associations. J. Neurophysiol. 73, 11011121.
6. Chen,L.L., and Wise,S.P. (1995). Supplementary eye field contrasted with the frontal
eye field during acquisition of conditional oculomotor associations. J. Neurophysiol.
73, 1122-1134.
7. Chen,L.L., and Wise,S.P. (1996). Evolution of directional preferences in the
supplementary eye field during acquisition of conditional oculomotor associations. J.
Neurosci. 16, 3067-3081.
8. Chen,Y., Martinez-Conde,S., Macknik,S.L., Bereshpolova,Y., Swadlow,H.A., and
Alonso,J.M. (2008). Task difficulty modulates the activity of specific neuronal
populations in primary visual cortex. Nat. Neurosci. 11, 974-982.
9. Chouinard,P.A., and Goodale,M.A. (2009). FMRI adaptation during performance of
learned arbitrary visuomotor conditional associations. NeuroImage 48, 696-706.
10. Cohen,M.R., and Maunsell,J.H. (2009). Attention improves performance primarily by
reducing interneuronal correlations. Nat. Neurosci. 12, 1594-1600.
26
Chapter 5
11. Crist,R.E., Li,W., and Gilbert,C.D. (2001). Learning to see: experience and attention in
primary visual cortex. Nat. Neurosci. 4, 519-525.
12. Ekstrom,L.B., Roelfsema,P.R., Arsenault,J.T., Bonmassar,G., and Vanduffel,W. (2008).
Bottom-up dependent gating of frontal signals in early visual cortex. Science 321, 414417.
13. Eliassen,J.C., Souza,T., and Sanes,J.N. (2003). Experience-Dependent Activation
Patterns in Human Brain during Visual-Motor Associative Learning. J. Neurosci. 23,
10540-10547.
14. Furmanski,C.S., Schluppeck,D., and Engel,S.A. (2004). Learning strengthens the
response of primary visual cortex to simple patterns. Curr. Biol. 14, 573-578.
15. Golcu,D., and Gilbert,C.D. (2009). Perceptual learning of object shape. J. Neurosci. 29,
13621-13629.
16. Gold,J.I., and Shadlen,M.N. (2007). The neural basis of decision making. Annu. Rev.
Neurosci. 30, 535-574.
17. Gold,J.I., Law,C.T., Connolly,P., and Bennur,S. (2009). Relationships between the
threshold and slope of psychometric and neurometric functions during perceptual
learning: implications for neuronal pooling. J. Neurophysiol.
18. Grossberg,S., and Raizada,R.D. (2000). Contrast-sensitive perceptual grouping and
object-based attention in the laminar circuits of primary visual cortex. Vision Res. 40,
1413-1432.
19. Hamker,F.H. (2005). The reentry hypothesis: the putative interaction of the frontal eye
field, ventrolateral prefrontal cortex, and areas V4, IT for attention and eye
movement. Cereb. Cortex 15, 431-447.
20. Hopf,J.M., Boehler,C.N., Luck,S.J., Tsotsos,J.K., Heinze,H.J., and Schoenfeld,M.A.
(2006). Direct neurophysiological evidence for spatial suppression surrounding the
focus of attention in vision. Proc. Natl. Acad. Sci. U. S. A 103, 1053-1058.
27
Chapter 5
21. Houtkamp,R., Spekreijse,H., and Roelfsema,P.R. (2003). A gradual spread of attention
during mental curve tracing. Percept. Psychophys. 65, 1136-1144.
22. Jolicoeur,P., Ullman,S., and Mackay,M. (1986). Curve tracing: a possible basic
operation in the perception of spatial relations. Mem. Cognit. 14, 129-140.
23. Khayat,P.S., Pooresmaeili,A., and Roelfsema,P.R. (2009). Time course of attentional
modulation in the frontal eye field during curve tracing. J. Neurophysiol. 101, 18131822.
24. Kim,J.N., and Shadlen,M.N. (1999). Neural correlates of a decision in the dorsolateral
prefrontal cortex of the macaque. Nat. Neurosci. 2, 176-185.
25. Kim,M.S., and Cave,K.R. (1995). Spatial Attention in Visual Search for Features and
Feature Conjunctions. Psychological Science 6, 376-380.
26. Kobatake,E., Wang,G., and Tanaka,K. (1998). Effects of shape-discrimination training
on the selectivity of inferotemporal cells in adult monkeys. J. Neurophysiol. 80, 324330.
27. Kourtzi,Z., Betts,L.R., Sarkheil,P., and Welchman,A.E. (2005). Distributed neural
plasticity for shape learning in the human visual cortex. PLoS. Biol. 3, e204.
28. Law,C.T., and Gold,J.I. (2009). Reinforcement learning can account for associative and
perceptual learning on a visual-decision task. Nat Neurosci 12, 655-663.
29. Lee,T.S., Yang,C.F., Romero,R.D., and Mumford,D. (2002). Neural activity in early visual
cortex reflects behavioral experience and higher-order perceptual saliency. Nat.
Neurosci. 5, 589-597.
30. Lewis,C.M., Baldassarre,A., Committeri,G., Romani,G.L., and Corbetta,M. (2009).
Learning sculpts the spontaneous activity of the resting human brain. Proceedings of
the National Academy of Sciences 106, 17558-17563.
31. Li,W., Piech,V., and Gilbert,C.D. (2004). Perceptual learning and top-down influences in
primary visual cortex. Nat. Neurosci. 7, 651-657.
28
Chapter 5
32. Li,W., Piech,V., and Gilbert,C.D. (2008). Learning to link visual contours. Neuron 57,
442-451.
33. Logothetis,N.K., Pauls,J., Augath,M., Trinath,T., and Oeltermann,A. (2001).
Neurophysiological investigation of the basis of the fMRI signal. Nature 412, 150-157.
34. Mazurek,M.E., Roitman,J.D., Ditterich,J., and Shadlen,M.N. (2003). A role for neural
integrators in perceptual decision making. Cereb. Cortex 13, 1257-1269.
35. McGee,V.E., and Carleton,W.T. Piecewise Regression. Journal of the American
Statistical Association 65[331], 1109-1124. 1970.
36. McMains,S., and Kastner,S. (2011). Interactions of top-down and bottom-up
mechanisms in human visual cortex. J. Neurosci. 31, 587-597.
37. Moore,T., and Armstrong,K.M. (2003). Selective gating of visual signals by
microstimulation of frontal cortex. Nature 421, 370-373.
38. Moro,S.I., Tolboom,M., Khayat,P.S., and Roelfsema,P.R. (2010). Neuronal activity in the
visual cortex reveals the temporal order of cognitive operations. J. Neurosci. 30,
16293-16303.
39. Noudoost,B., and Moore,T. (2011). Control of visual cortical signals by prefrontal
dopamine. Nature 474, 372-375.
40. Pasupathy,A., and Miller,E.K. (2005). Different time courses of learning-related activity
in the prefrontal cortex and striatum. Nature 433, 873-876.
41. Platt,M.L., and Glimcher,P.W. (1999). Neural correlates of decision variables in parietal
cortex. Nature 400, 233-238.
42. Pooresmaeili,A., Poort,J., Thiele,A., and Roelfsema,P.R. (2010). Separable codes for
attention and luminance contrast in the primary visual cortex. J. Neurosci. 30, 1270112711.
29
Chapter 5
43. Poort,J., and Roelfsema,P.R. (2009). Noise correlations have little influence on the
coding of selective attention in area V1. Cereb. Cortex 19, 543-553.
44. Roelfsema,P.R., Lamme,V.A., and Spekreijse,H. (1998). Object-based attention in the
primary visual cortex of the macaque monkey. Nature 395, 376-381.
45. Roelfsema,P.R., and Spekreijse,H. (2001). The representation of erroneously perceived
stimuli in the primary visual cortex. Neuron 31, 853-863.
46. Roelfsema,P.R., Khayat,P.S., and Spekreijse,H. (2003). Subtask sequencing in the
primary visual cortex. Proc. Natl. Acad. Sci. U. S. A 100, 5467-5472.
47. Roelfsema,P.R. (2006). Cortical algorithms for perceptual grouping. Annu. Rev.
Neurosci. 29, 203-227.
48. Roelfsema,P.R., Tolboom,M., and Khayat,P.S. (2007). Different processing phases for
features, figures, and selective attention in the primary visual cortex. Neuron 56, 785792.
49. Roitman,J.D., and Shadlen,M.N. (2002). Response of neurons in the lateral
intraparietal area during a combined visual discrimination reaction time task. J.
Neurosci. 22, 9475-9489.
50. Scholte,H.S., Spekreijse,H., and Roelfsema,P.R. (2001). The spatial profile of visual
attention in mental curve tracing. Vision Res. 41, 2569-2580.
51. Schoups,A., Vogels,R., Qian,N., and Orban,G. (2001). Practising orientation
identification improves orientation coding in V1 neurons. Nature 412, 549-553.
52. Schwartz,S., Maquet,P., and Frith,C. (2002). Neural correlates of perceptual learning: a
functional MRI study of visual texture discrimination. Proc. Natl. Acad. Sci. U. S. A 99,
17137-17142.
53. Serences,J.T., Yantis,S., Culberson,A., and Awh,E. (2004). Preparatory activity in visual
cortex indexes distractor suppression during covert spatial orienting. J. Neurophysiol.
92, 3538-3545.
30
Chapter 5
54. Shadlen,M.N., and Newsome,W.T. (2001). Neural basis of a perceptual decision in the
parietal cortex (area LIP) of the rhesus monkey. J. Neurophysiol. 86, 1916-1936.
55. Sigman,M., Pan,H., Yang,Y., Stern,E., Silbersweig,D., and Gilbert,C.D. (2005). Top-down
reorganization of activity in the visual pathway after learning a shape identification
task. Neuron 46, 823-835.
56. Spitzer,H., and Richmond,B.J. (1991). Task difficulty: ignoring, attending to, and
discriminating a visual stimulus yield progressively more activity in inferior temporal
neurons. Exp. Brain Res. 83, 340-348.
57. Super,H., and Roelfsema,P.R. (2005). Chronic multiunit recordings in behaving animals:
advantages and limitations. Prog. Brain Res. 147, 263-282.
58. Suzuki,W.A. (2008). Chapter 19 Associative learning signals in the brain. In Progress in
Brain Research, J.-C.L.V. Wayne S.Sossin, ed. Elsevier), pp. 305-320.
59. van,d., V, and de,K.M. (2001). From knowing what to knowing where: modeling objectbased attention with feedback disinhibition of activation. J. Cogn Neurosci. 13, 479491.
60. Wiener,M.C., and Richmond,B.J. (2003). Decoding spike trains instant by instant using
order statistics and the mixture-of-Poissons model. J. Neurosci. 23, 2394-2406.
61. Xing,D., Yeh,C.I., and Shapley,R.M. (2009). Spatial spread of the local field potential
and its laminar variation in visual cortex. J. Neurosci. 29, 11540-11549.
62. Yeh,C.I., Xing,D., and Shapley,R.M. (2009). "Black" responses dominate macaque
primary visual cortex v1. J. Neurosci. 29, 11753-11760.
63. Yotsumoto,Y., Watanabe,T., and Sasaki,Y. (2008). Different dynamics of performance
and brain activation in the time course of perceptual learning. Neuron 57, 827-833.
64. Yotsumoto,Y., Sasaki,Y., Chan,P., Vasios,C.E., Bonmassar,G., Ito,N., Nanez,J.E., Sr.,
Shimojo,S., and Watanabe,T. (2009). Location-specific cortical activation changes
during sleep after training for perceptual learning. Curr. Biol. 19, 1278-1282.
31
Chapter 5
65. Zhou,H., and Desimone,R. (2011). Feature-based attention in the frontal eye field and
area V4 during visual search. Neuron 70, 1205-1217.
32