Download Operant control of human eye movements

G Model
BEPROC-2328;
No. of Pages 7
ARTICLE IN PRESS
Behavioural Processes xxx (2011) xxx–xxx
Contents lists available at ScienceDirect
Behavioural Processes
journal homepage: www.elsevier.com/locate/behavproc
Operant control of human eye movements
Laurent Madelain ∗ , Céline Paeye, Jean-Claude Darcheville
Laboratoire Ureca, UFR de Psychologie, Université Lille Nord de France, Villeneuve d’Ascq, France
a r t i c l e
i n f o
Article history:
Received 9 December 2010
Received in revised form 17 February 2011
Accepted 17 February 2011
Keywords:
Eye movements
Motor control
Reinforcement
Variability
a b s t r a c t
Saccade and smooth pursuit are the eye movements used by primates to shift gaze. In this article we
review evidence for the effects of reinforcement on several dimensions of these responses such as their
latencies, velocities or amplitudes. We propose that these responses are operant behaviours controlled
by their consequences on performance of visually guided tasks. Studying the conditions under which
particular eye movement patterns might emerge from the cumulative effects of reinforcement provides
critical insights about how motor responses are attuned to environmental exigencies.
© 2011 Elsevier B.V. All rights reserved.
In this article, evidence for control by operant reinforcement
contingencies of eye movements are examined. Because many
readers of this journal, and indeed many researchers who study
operant behaviour, may be unfamiliar with some of the technical
aspects of eye movement, we will first describe some of the key
phenomena.
1. Primate eye movements
Eyes are present in a large majority of all animals species – perhaps 95% (Land and Nilsson, 2002) – revealing that vision provides
a great advantage in various environments and that eyes appeared
very early in animal evolution. However not all animal eyes are
identical. The evolutionary history of the eye, and of the visual
system, has been a challenge to biologists and Darwin (1859) himself was aware of the need to account for it. Natural selection acts
on sensory systems through the consequences of sensory guided
behaviour and one might suppose that the acquisition of visually
guided behaviours drove the progressive evolution of animal eyes
(Nilsson, 2009). Because vision depends strongly on the ability to
move the eyes, one may also assume that eye movements evolved
in relation to the evolution of both vision and visual tasks: natural selection has shaped eye movement systems in ways that are
tailored to the behavioural repertoire and visual system of each
species (Krauzlis, 2008). In primates, five types of eye movements
have been identified, each devoted to a particular visual challenge
and involving specific mechanisms.
∗ Corresponding author.
E-mail address: [email protected] (L. Madelain).
The first requirement for effective vision is to stabilize the eyes
with respect to the world. If your eyes were fixed with respect to
your head, your head would have to be as motionless as the page
you are trying to read, otherwise the image of the word would be
moving on your retina and the letters would appear blurred. This
can be experienced by rapidly shaking the page in front of you.
In reality the head is always moving and its movements would
make reading impossible unless we manage to fixate the gaze on
the words. To do so eye movements must compensate head movements. Fortunately primates eye movement systems are able to
maintain visual acuity during self-motion (i.e., to stabilize the retinal image of the world with rotations of the eyes that exactly
compensate for head and body movements). To experience it, try
to fixate a word on this page while shaking the head from left to
right. Your head movements should not impede your ability to read
because head movements are compensated by the vestibulo-ocular
reflex (VOR) which elicits eye movements in a direction opposite to
that of the head. Vestibulary-controlled reflex eye movements are
evolutionarily the oldest of all (Walls, 1962), because when animals
move with respect to their surroundings, they risk considerably
degrading their visual acuity even with the simplest form of retina:
the retinal image of the world might slip with every head movement which would cause vision to become blurred and impair the
ability to recognize and localize objects. If you now move the page
in a sinusoidal fashion at a slow speed while keeping your head
still, you will still be able to read the written words. This response,
the optokinetic reflex (OKN) allows the eyes to follow a motion of
the whole visual field, or, as in our example, a large portion of
it.
As with most animals with frontally directed eyes, primate
retinas contain a specialized central area with a high density of
photoreceptors, the fovea, the most central 1% part of our retina
0376-6357/$ – see front matter © 2011 Elsevier B.V. All rights reserved.
doi:10.1016/j.beproc.2011.02.009
Please cite this article in press as: Madelain, L., et al., Operant control of human eye movements. Behav. Process. (2011),
doi:10.1016/j.beproc.2011.02.009
G Model
BEPROC-2328;
No. of Pages 7
2
ARTICLE IN PRESS
L. Madelain et al. / Behavioural Processes xxx (2011) xxx–xxx
responsible for sharp vision. With the evolution of the fovea, a
second requirement arose for eye movements: when an object of
interest appears in the visual periphery, we need to move this central portion of the retina so that the object can be seen best. To
this end saccades, pursuit, and vergence eye movements—the gaze
shifting movements, evolved.
When reading these lines your eyes are making a series of
short rapid movements called saccades intermingled with periods
of immobility called fixations. During the time the eyes are fixating,
the image is stable on your retina, allowing accurate visual perception. You can read the word you are looking at and then move to
the next. If you attempt to read a sentence without moving the eyes
you quickly realize that you cannot distinguish the letters composing the words that surround the point of space you are fixating:
around the fixation point, only four to five letters are seen clearly
and visual acuity very rapidly drops when eccentricity increases.
Therefore the ability to move the gaze position to the next word is
necessary in order to read.
Another type of gaze shifting movement can be illustrated by a
simple experiment. While fixating a word, place your finger on the
page and move it from the leftward side to the rightward side of
the page. Your finger should appear blurred because it is moving
with respect to your eyes. If you now look at your moving finger
you can see it clearly but the words behind it are blurred because
the whole page is moving with respect to your eyes. This response,
allowing tracking a moving object against a fixed background is
called smooth pursuit.
Finally try to fixate a word while moving your head toward and
then away from the page. When doing so the two eyes are moving
in opposite direction to change the depth at which the eyes’ lines
of sight meet. These disconjugate movements are called vergence
eye movements. When we look at a near object our eyes converge
(rotate inward), and when we look at a far object our eyes diverge
(rotate outward).
Although we are usually unaware of eye movements during
daily activities such as reading, these simple experiments reveal
that stabilizing the eyes with regard to the outside world and
aiming the eyes toward moving or stationary targets are critical challenges to effective vision (Krauzlis, 2008). Gaze shifting
eye movements, and particularly pursuit and saccades, have been
extensively studied in primates and we probably know more about
these responses and their neural substrate than any other motor
responses. Interestingly, gaze shifting eye movements are often
regarded as a model for motor control because of the relative simplicity of the oculomotor system and the ease of recording and
quantitative analysis.
How reinforcement learning is involved in controlling saccade
and pursuit is the subject of this review. Our working hypothesis is
that saccades and pursuit are operant behaviours reinforced by the
ability to perform visual tasks, ensuring the required behavioural
discrepancy (Donahoe et al., 1993).
2. Smooth pursuit
Smooth pursuit is a slow movement whose trajectory is determined by a moving stimulus. This response provides crucial support
for vision by maintaining the retinal image of the object close to the
fovea and by minimizing motion blur that would otherwise undermine visual perception. Although one might inhibit tracking the
movement of an object, it is noteworthy that one cannot initiate
smooth pursuit at will: a moving stimulus is necessary but moving targets that are sensed through modalities other than vision
can, to some extent, guide pursuit. For instance one can track one’s
own outstretched finger while in darkness with mostly smooth eye
movements.
In most visual situations the perceived motion and the physical
motion of the target’s retinal image are the same, and conventional
models of pursuit typically assume that the raw retinal-image
motion is the driving stimulus for pursuit. However, experiments
using complex stimuli have demonstrated that pursuit can be
driven by perceived motion rather than physical retinal motion. For
instance, Steinbach (1976) demonstrated that the pursuit system
can respond to visual motion that has ambiguous retinal counterparts: subjects tracked a perceived horizontal motion signal –
the displacement of the invisible center of a wheel – resulting
from the cycloidal motion of targets attached to the circumference of a rolling wagon wheel. In a similar vein, monkeys may
track the invisible midpoint between two peripheral bars, such that
the visual stimuli are peripheral but the goal is foveal/parafoveal
(Hafed and Krauzlis, 2008). Recently, researchers have used linefigure objects viewed through apertures to disentangle retinal
image motion from perceived object motion: changing the coherence of the perceived motion affected the pursuit responses even
though the retinal image motion remained the same (Beutter and
Stone, 2000; Stone et al., 2000). Using a plaid stimulus that produces systematic perceptual errors, it has been demonstrated that
pursuit and perception can be biased in the same way and to the
same degree (Beutter and Stone, 1998), and that expectations about
visual motion can bias both pursuit and perception (Kowler, 1989;
Krauzlis and Adler, 2001).
To measure the temporal correlation between a perceived
change in motion and corresponding changes in pursuit Madelain
and Krauzlis (2003a) used an apparent motion stimulus consisting
of a horizontal row of evenly spaced Kanizsa illusory squares: the
illusory contours appeared at the midpoints of the illusory squares
presented in the previous frame, producing bi-directional apparent
motion of the illusory contours that could be reversed at will. Subjects were able to accurately track the perceived motion induced by
the apparent motion of illusory squares and changes in the pursued
direction and the reported perceived direction of motion were in
temporal register. However, the retinal slip (i.e., the displacement
of the retinal image of an object) is often regarded as the physical signal necessary for accurate pursuit despite experimental data
indicating that illusory motion might also drive pursuit. Contrasting with this view, one might suppose that pursuit is also controlled
by its functional perceptual consequences, not by the stabilization
of a retinal image, a hypothesis that is consistent with the ability to track the illusory motion of an object. We will turn next to
experimental evidence for the operant control over pursuit.
3. Smooth pursuit is an operant
It is noteworthy that in most natural situations the ability to
compensate for the physical motion of a target and the ability to
perform visually guided tasks involving this target are confounded:
stabilizing the retinal image of a relevant target allows its clear
vision which allows visually guided behaviours that might act as
reinforcers. However, one could create a laboratory situation in
which these two factors are disentangled. One of the most obvious ways to distinguish between the effects of stabilizing a visual
target and the effects of reinforcement on smooth pursuit is to transiently remove the pursued target. In this situation pursuit might
be reinforced in the absence of visual signals. In fact, although pursuit requires some motion to be initiated, some pursuit might be
observed when the target is temporarily occluded. For instance
Becker and Fuchs (1985) showed that humans maintain smooth
pursuit when a moving target briefly disappeared and then reappeared up to 4 s after the disappearance of the target. They found
that the velocity of the eye started to rapidly decrease about 190 ms
after the target disappearance then the eye velocity stabilized at
Please cite this article in press as: Madelain, L., et al., Operant control of human eye movements. Behav. Process. (2011),
doi:10.1016/j.beproc.2011.02.009
G Model
BEPROC-2328;
No. of Pages 7
ARTICLE IN PRESS
L. Madelain et al. / Behavioural Processes xxx (2011) xxx–xxx
approximately 40–60% of the normal pursuit velocity. Interestingly
Becker and Fuchs (1985) examined the effect of practice and concluded that the ability to predict the target’s motion was involved.
This predictive mechanism would also be at play during anticipatory smooth tracking of a highly predictable target motion (Barnes
and Asselman, 1991; Barnes and Donelan, 1999; van den Berg,
1988; Wells and Barnes, 1999).
One might suppose that in real life, smooth pursuit allows
the clear vision of the target, and might therefore be reinforced
when the eye velocity matches the target’s velocity, implying that
eye velocity is reinforced by the consequences of the movement.
Madelain and Krauzlis (2003b) Hypothesised that subjects might
learn to maintain pursuit while the visual moving target was transiently occluded if contingencies were such that a reinforcer was
given depending on the eye velocity. We conducted a series of
experiments in which the real-time delivery of a reinforcer was
based on the tracking performance of the subject estimated by the
eye velocity and occurrence of saccades (during perfect pursuit the
eye velocity matches the target’s velocity; when pursuit is not perfect the eye velocity is usually lower than the target’s velocity and
frequent saccades that catch-up with the target are observed). An
auditory tone was delivered when eye velocity remained within a
set of reinforcement criteria, and the tone was removed when eye
velocity fell outside the criteria or when a saccade was detected
(because saccades observed during pursuit compensate for imperfect pursuit). These contingencies permitted the differential real
time delivery of consequences based on the pursuit performance:
periods of pursuit with eye velocity close to the desired velocity
were reinforced while other epochs were not. If the total duration
of the auditory tone exceeded a temporal criterion in 7 out of 10
consecutive trials the subject won 25 cents. The duration of the
target disappearance was progressively increased according to the
subjects’ tracking performance: once the ability to maintain the eye
velocity close to the desired velocity was established the period of
target occlusion was progressively extended. We found that after
the learning phase, subjects were able to maintain eye velocities
close to values typically observed during steady-state tracking,
even though the target disappeared for 1 s. We also observed a
marked reduction in the rate of catch-up saccades, implying that
the eye was not lagging behind the target, and found that learning
generalized over untrained target velocities. We did not observe
significant changes in pursuit in a yoked-control group – the reinforcer was independent of the subjects’ responses – or in a control
group – no reinforcer was delivered but the task was intensively
practiced.
In a similar vein, Paeye et al. (2007) used a target whose position was continuously adjusted to match the gaze position such
that the target did not move as long as the eye was fixed. Therefore
pursuit initiation could not be controlled by the physical motion
of the target. A shaping procedure was implemented based on the
eye velocity of anticipatory movements. We found that in this situation subjects progressively learned to initiate smooth movements
in the absence of a motion signal. These results provide evidence
that smooth pursuit, even in the absence of visual target or in the
absence of external motion, is an operant behaviour.
4. Development of pursuit
Further evidence of the operant nature of pursuit is provided
by studying its development. Although some episodes of smooth
pursuit may be observed before 2 months of age, newborn infants
mostly track moving targets with a series of fast saccades (Aslin,
1981; Johnson, 1990). The poor maturational state of both the fovea
and cortical structures involved in the oculomotor control of pursuit are typically invoked to account for this immature behaviour.
3
It is interesting to point out that the occurrence of object manipulations as well as displacements of the subject also start to increase
at the same age. Therefore, one might suppose that smooth pursuit develops because of the progressive establishment of the
behavioural discrepancy required for operant learning: before 2
months, pursuit has no functional consequences whereas once
object manipulation abilities appear pursuit becomes necessary to
perform some visually guided tasks.
Darcheville et al. (1999) probed this hypothesis using an auditory reinforcer contingent on smooth eye movements in infants
aged from 1 to 7 days. Reinforcement contingencies were such that
periods of smooth movements in the direction of a moving target
triggered the onset of music in the experimental room. The proportion of slow movements elicited by the moving target markedly
increased in a reinforced group but not in control groups. Contrary
to the maturational hypothesis it may be assumed that pursuit is a
learned response driven by the ability to perform visually guided
tasks: pursuit allows the optimization of the visual perception of
the target and this optimization acts as a reinforcer.
The pursuit system has been described historically as a servosystem driven by retinal slip (Robinson, 1986). In servo-systems
the need to control the gain is mainly for calibration purposes: the
ability to adaptively modify internal parameters is required to compensate for modifications in the relationship between visual input
and motor output (Optican et al., 1985). These modifications naturally occur during development and aging as a result of changes in
the eye movement system after eye growth or muscular weakness.
However the pursuit response is modulated by attentional factors
(Khurana and Kowler, 1987), regularity or predictability of the target’s motion (Barnes and Asselman, 1991; Barnes and Donelan,
1999; Wells and Barnes, 1999; Freyberg and Ilg, 2008; Badler and
Heinen, 2006; Bennett and Barnes, 2006), movement perception
(Madelain and Krauzlis, 2003a) and learning induced by the reinforcing consequences of improved tracking (Madelain and Krauzlis,
2003b; Darcheville et al., 1999). Therefore the functional significance of operant control in the pursuit system might extend well
beyond motor calibration. We propose that, in real life, accurate
pursuit is controlled by the ability to perform visually guided tasks.
This proposition departs from the conventional view of motor control which states that the physical sensory outcome of a movement,
irrespective of functional consequences, is used to adjust the motor
command (Wolpert et al., 1995; Kording, 2007). We will now turn
to saccades and review evidence for the idea that they also have
attributes of operant behaviour.
5. Saccades
Unlike pursuit, a continuous response that can be maintained
for extended periods of time, saccades are discrete responses lasting only tens of milliseconds, allowing animals with foveal vision
to quickly move the image of a visual target from an eccentric
retinal location to the center of the retina. Because they are so
rapid—a 10◦ saccade (since the eyes are rotating, degrees of visual
angles are used to measure eye movements) lasts 40–50 ms in
humans (Becker, 1989)—no useful feedback from the visual system can guide the course of each movement, since visual signals
require 40–50 ms to reach the superior colliculus, the principal
saccade-programming brain region (Goldberg and Wurtz, 1972; Li
and Basso, 2008). Moreover, it appears that some active mechanisms of vision suppression (Thiele et al., 2002) added to the very
high speed of the retinal image (eye velocity reaches up to 500◦ /s
(Bahill et al., 1975)) which creates a strong visual blur, considerably
impairing vision during the movement. This is why, when standing
in front of a mirror, one cannot see her own eyes move although
an observer would easily observe the saccadic movements. Thus,
Please cite this article in press as: Madelain, L., et al., Operant control of human eye movements. Behav. Process. (2011),
doi:10.1016/j.beproc.2011.02.009
G Model
BEPROC-2328;
No. of Pages 7
4
ARTICLE IN PRESS
L. Madelain et al. / Behavioural Processes xxx (2011) xxx–xxx
saccades are often described as ballistic, in that the trajectory is
programmed prior to the movement.
Because the primary function of saccades is to improve the
visual perception of a target, these movements have sometimes
been described as observing responses (i.e., responses that lead to
exposure to discriminative stimuli), and some studies used saccadic eye movements to assess observing responses in various
tasks (e.g., (Dube et al., 2006; Tomanari et al., 2007)). Observing
responses have been suggested to be like any other instrumental
response subject to the principles of operant behaviour (Holland,
1957), and the pioneer works of Schroeder and Holland (1968,
1969) demonstrated that the occurrence of saccadic eye movements might be reinforced by producing a stimulus. The authors
insisted that because saccades are instrumental responses, many
of the phenomena of “attention” could be studied using eye movements in operant research. However, despite an impressive body
of research devoted to attention in relation to oculomotor control (see for instance (Posner, 1980; Deubel and Schneider, 1996;
Kowler et al., 1995)), the operant nature of saccades and the effects
of reinforcement on the saccadic topography have only rarely been
directly probed in humans.
6. Saccades are operant behaviour
The effects of reinforcement on various properties of saccadic
movements have been recently experimentally explored in monkeys. In a typical experiment, the target appears randomly at
one out of two locations, but only one location is associated
with a reinforcer. It has been observed that monkey saccades
to rewarded locations had shorter latencies (Ikeda and Hikosaka,
2003; Lauwereyns et al., 2002; Watanabe et al., 2003) and higher
peak velocities (Takikawa et al., 2002) than saccades of the same
amplitude to non-rewarded locations. These modulations might be
the result of activation of dopaminergic neurons in the basal ganglia (Nakamura and Hikosaka, 2006). Specifically, different neurons
in the caudate nucleus fire in anticipation of rewarded vs. nonrewarded saccades (Watanabe et al., 2003), midbrain dopaminergic
neurons fire in anticipation of rewarded saccades (Bayer and
Glimcher, 2005), and the basal ganglia modulate the superior colliculus, facilitating saccades to rewarded locations (Hikosaka et al.,
2006).
Furthermore, if monkeys are rewarded for making saccades to
either of two targets, with saccades to each being reinforced on different variable-interval schedules of reinforcement, monkeys learn
to match the relative frequency of saccades to each target to the relative frequency of reinforcement (Sugrue et al., 2004). In humans,
saccadic peak velocities increased when the saccade target was a
human face when compared to other visual stimuli (Xu-Wilson
et al., 2009). Montagnini and Chelazzi (2005) showed that saccadic
latencies and saccadic peak-velocities in humans might be controlled by the ability to perform a visual discrimination. Subjects
were instructed to perform a discrimination task on the saccade
target. The target was briefly displayed such that the shorter the
saccadic latency, the longer the target was viewed and the easier the discrimination task was. In this situation, saccadic latencies
decreased while saccadic peak velocities increased. One interpretation of these results is that the successful performance of the task
reinforces the short latencies. Taken together, these experimental
studies indicate that saccade properties are plastic and might be
modulated by environmental contingencies.
7. Saccadic adaptation
A more direct example of the plasticity of saccades has been
found in a phenomenon called saccadic adaptation. In the labo-
ratory, the alteration of saccadic amplitude is generally studied by
having a target step away from the fixation point and, while the eye
is in flight toward the new location, moving the target (for example, slightly back toward its original location) so that the eye lands
away from the target, and must make a second, corrective, saccade
to acquire it. After a number of repetitions, the amplitude of the
initial saccade decreases, bringing the eye closer to the displaced
target location (McLaughlin, 1967). This behavioural plasticity presumably maintains the accuracy of saccades despite changes in
the eyes, muscles and oculomotor system during development and
aging or following injuries (Abel et al., 1978; Kommerell et al., 1976;
Scudder et al., 1998).
Saccadic adaptation reliably changes the amplitude or direction
of saccades, albeit with considerable variation from subject to subject and from day to day (reviewed by Hopp and Fuchs (2004) and
Pelisson et al., 2010). Conventionally, saccadic adaptation has been
viewed as controlled by a visual position error signal, the “retinal
error” (i.e., the postsaccadic distance between target and fovea),
which is reduced over many trials (Noto and Robinson, 2001;
Wallman and Fuchs, 1998): the presence of a retinal error would
elicit a progressive correction in saccade amplitude. An argument
against retinal error being the signal guiding saccade adaptation
is the finding that saccades of a single direction and amplitude
can have two different adaptation states depending on context as
it has been shown both with a proprioceptive context (Alahyane
and Pelisson, 2004; Shelhamer and Clendaniel, 2002; Tian and Zee,
2010) and with a visual context (Herman et al., 2009). For instance,
an increase in amplitude was induced with the eyes directed up
10◦ above straight-ahead gaze and a decrease in amplitude was
induced with the eyes directed down (Tian and Zee, 2010).
To put the matter in terms of operant conditioning, saccadic
adaptation might be controlled by a discriminative stimulus. This
discriminative control may also explain why there is some maintenance of experimentally induced adaptation after 24 h in monkeys
(Noto et al., 1999) and after several days in humans (Alahyane
and Pelisson, 2005), despite thousands of unperturbed saccades
made between sessions. In recent experiments both a target and
a distractor were displayed after the saccade such that the unadapted saccades landed either on the distractor or on the target
(Madelain et al., 2010). In the first case saccadic adaptation was
induced whereas in the second case we did not observe a change
in the saccadic amplitude: in the presence of two conflicting postsaccadic visual stimuli, the target and the distractor, the saccadic
system selectively adapts its amplitude as though only the target
were present.
We hypothesized that the target-oriented adaptation we
observed might be explained by the reinforcing effect of the vision
of the target: those saccades landing near the target have been
selected whereas those landing near the distractor have not. To
probe this hypothesis we induced changes in saccade amplitude
in the absence of retinal error, using an arbitrary reinforcement –
either an auditory tone or viewing the target on the fovea (Madelain
et al., 2008). Saccadic amplitude was computed in real-time and a
percentile procedure was used to compute the reinforcement criteria on a trial to trial basis. The amplitude of saccades followed the
reinforcement contingencies such that amplitude decreased when
small saccades were reinforced while amplitude increased when
large saccades were reinforced. Instead of a servo-mechanism,
we concluded that saccadic amplitude is controlled by a general
learning process relying on the outcome of each movement given
a particular state of the environment. In other words, adjusting
the saccadic amplitude could be explained by a selection process in which those responses that produce a valuable outcome
are selected over those that do not. According to this hypothesis, viewing the target might be reinforcing because maximizing
visual perception would allow other adaptive interactions with the
Please cite this article in press as: Madelain, L., et al., Operant control of human eye movements. Behav. Process. (2011),
doi:10.1016/j.beproc.2011.02.009
G Model
BEPROC-2328;
No. of Pages 7
ARTICLE IN PRESS
L. Madelain et al. / Behavioural Processes xxx (2011) xxx–xxx
environment. This hypothesis is similar to what we proposed for
pursuit: gaze shifting movements would be controlled by the ability to perform visually guided behaviours that act as reinforcers
because of their adaptive values.
8. Reinforced variability in saccadic eye movements
Behaviour analysis views the principle of selection by reinforcement as playing the same role in the understanding of complex
behaviour that natural selection plays in understanding the evolution of complex organisms (Donahoe and Palmer, 1994). Because
variation provides the raw material upon which selection operates, the origins and functions of behavioural variability have been
extensively investigated (see Neuringer, 2002, 2004 for reviews).
It has been demonstrated that variability might be controlled by
making reinforcement contingent on variability itself: variability is
an operant dimension (Blough, 1966; Machado, 1989; Grunow and
Neuringer, 2002; Pryor et al., 1969). One might therefore suppose
that part of the variability observed in motor responses is related
to environmental contingencies such that, depending on the situation, a given range of behavioural instances of an operant response
might be reinforced. For instance, when making a saccade toward
a visual target, small variations in the landing positions might be
too small to affect the visual perception of the target and every
movement would be reinforced. On the other hand large variations
might be extinguished because landing far away from the target
has detrimental effects on vision. In other words, the natural level
of variability in saccadic control would result from operant learning
in which vision provides the functional consequences.
To probe this hypothesis we manipulated the reinforcement contingencies of saccadic amplitude variability (Paeye and
Madelain, 2011). In one group of participants reinforcement criteria were such that varying the amplitude of saccades was reinforced
while in a second yoked control group reinforcement was independent of the saccades amplitudes. Our results clearly demonstrate
that the spread of the amplitude distributions was controlled by
reinforcement. Moreover, the reinforcement-induced increases in
variability did not affect the average saccadic amplitude. In another
set of experiments we were able to shape the saccadic reaction time
distributions by direct reinforcement of variability, further confirming that variations in motor control might be learned (Madelain
et al., 2007). These results contrast with the classical view stating that variability in motor responses originates from endogenous
stochastic variations that affect each stage between a sensory event
and the motor response – sensing, information processing, movement planning and executing ((van Beers, 2007), and see Faisal et
al. (2008), for a review). In this view variability would be independent from environmental contingencies and the level of variability
would be related to the strength of the signal. In other words the
spread of a distribution would depend on the mean such that the
larger the mean the larger the variability. That saccadic amplitude variability might be increased independently from the average
saccadic amplitude would not be accounted for by this stochastic hypothesis. Similarly, computational decision models failed to
explain the reinforcement induced changes in reaction time distributions (Madelain et al., 2007) because these models predict a
mechanical relation between the mean and the variance of the distribution: changing one single parameter in these models affects
both the mean and variance of the latency distributions (Ratcliff,
2001; Reddi and Carpenter, 2000; Wagenmakers and Brown, 2007).
The ability to control variability in saccadic responses by reinforcement could be viewed as a laboratory curiosity but we
believe that these experiments might have some profound theoretical implications. It has been proposed that operant variability
is functional because it allows the shaping of new behaviours
5
by increasing the probability of emitting new approximations of
the target response and the probability of detecting changes in
environmental contingencies (Neuringer, 2002). For instance we
suggested that, in real life, saccades that lead to a clear vision of
the target are selected by their consequences. One may therefore
postulate that some variation in saccadic amplitude is necessary to
adapt to changes affecting the oculomotor system: saccadic adaptation would require some variability as a basis for differential
reinforcement. Similarly, it is possible that the ability to vary the
strategies used when looking for a particular target might be necessary to encounter new regularities in the organizations of our
visual environments. However, the direct effects of variability on
the establishment of saccadic eye movement control, and more
generally on motor control, remain to be experimentally studied.
9. Conclusion
Saccades and smooth pursuit are operant behaviours: several
dimensions of these eye movements such as their velocities, latencies or reaction times may be experimentally manipulated by
reinforcement contingencies. In real life these movements are controlled by an enhanced clarity of a visual target, allowing adapted
visually guided behaviours which act as reinforcers. In the case of
eye movements, it is therefore difficult to disentangle the visual discriminative stimulus (clarity of the visual target stimulus) from the
reinforcing consequences (the task that depends on the clear perception of the visual stimulus) and it is not surprising that the visual
properties of the target are conventionally regarded as the controlling signals for gaze shifting movements. In the laboratory however,
one may use non-visual reinforcers to induce changes in response
topography and study the operant properties of eye movements.
Because voluntary eye movements are commonly used to study
motor control, one might expect that a similar strategy might
be applied to other motor responses. It is noteworthy that little
research has been devoted to this end.
Ingvaldsen and Whiting (1997) pointed out that behaviour analysts commonly focus on why organisms behave as they do instead
on how they do so (i.e., the patterns of movement coordination)
such that the particular movements are usually not of immediate concern to the operant approach. However, we believe that
the operant approach provides a valid alternative to contemporary
computational models of motor control which, because they postulate that predictions of the physical outcome of a movement are
compared to the actual sensory outcome of a movement to produce optimal responses (Wolpert and Ghahramani, 2000), leave
no room for the functional consequences of motor behaviours. A
critical feature of these models is that motor control is viewed
as a machine-like servo-mechanism involving internal comparisons, a proposition which is difficult, if not impossible, to probe
at the behavioural level (see (Ingvaldsen and Whiting, 1997), for a
related discussion). It is our hope that more research will explore
the operant nature of motor control in the future, either by studying the development and acquisition of new motor responses or
by exploring the effects of reinforcement contingencies on existing responses. How particular configurations of motor behaviours
emerge in response to coordination constraints and particular
environmental demands remains a fascinating question for the
behaviour analyst.
Acknowledgements
This research was funded in part by Grant 1R01EY019508 (LM)
from the National Institute of Health, Agence Nationale Pour la
Recherche Grant ANR-JC09 494068 (JCD, LM, CP) and a Fulbright
Fellowship (LM).
Please cite this article in press as: Madelain, L., et al., Operant control of human eye movements. Behav. Process. (2011),
doi:10.1016/j.beproc.2011.02.009
G Model
BEPROC-2328;
6
No. of Pages 7
ARTICLE IN PRESS
L. Madelain et al. / Behavioural Processes xxx (2011) xxx–xxx
References
Abel, L.A., Schmidt, D., Dell’Osso, L.F., Daroff, R.B., 1978. Saccadic system plasticity
in humans. Ann. Neurol. 4, 313–318.
Alahyane, N., Pelisson, D., 2004. Eye position specificity of saccadic adaptation.
Invest. Ophthalmol. Vis. Sci. 45, 123–130.
Alahyane, N., Pelisson, D., 2005. Long-lasting modifications of saccadic eye movements following adaptation induced in the double-step target paradigm. Learn.
Mem. 12, 433–443.
Aslin, R., 1981. Development of smooth pursuit in human infants. In: Fischer, D.F.,
Monty, R.A., Senders, E.J. (Eds.), Eye Movements: Cognition and Vision Perception. Erlbaum, Hillsdale, NJ.
Badler, J.B., Heinen, S.J., 2006. Anticipatory movement timing using prediction and
external cues. J. Neurosci. 26, 4519–4525.
Bahill, A.T., Clark, M.R., Stark, L., 1975. The main sequence, a tool for studying human
eye movements. Math. Biosci. 24, 191–204.
Barnes, G.R., Asselman, P.T., 1991. The mechanism of prediction in human smooth
pursuit eye movements. J. Physiol. 439, 439–461.
Barnes, G.R., Donelan, S.F., 1999. The remembered pursuit task: evidence for segregation of timing and velocity storage in predictive oculomotor control. Exp.
Brain Res. 129, 57–67.
Bayer, H.M., Glimcher, P.W., 2005. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141.
Becker, W., 1989. The neurobiology of saccadic eye movements. Metrics. Rev. Oculomot. Res. 3, 13–67.
Becker, W., Fuchs, A.F., 1985. Prediction in the oculomotor system: smooth pursuit
during transient disappearance of a visual target. Exp. Brain Res. 57, 562–575.
Bennett, S.J., Barnes, G.R., 2006. Combined smooth and saccadic ocular pursuit during
the transient occlusion of a moving visual object. Exp. Brain Res. 168, 313–321.
Beutter, B.R., Stone, L.S., 1998. Human motion perception and smooth eye movements show similar directional biases for elongated apertures. Vision Res. 38,
1273–1286.
Beutter, B.R., Stone, L.S., 2000. Motion coherence affects human perception and
pursuit similarly. Vis. Neurosci. 17, 139–153.
Blough, D.S., 1966. The reinforcement of least-frequent interresponse times. J. Exp.
Anal. Behav. 9, 581–591.
Darcheville, J.-C., Madelain, L., Buquet, C., Charlier, J., Miossec, Y., 1999. Operant
conditioning of the visual smooth pursuit in young infants. Behav. Process. 46,
131–139.
Darwin, C., 1859. On the Origin of Species by Means of Natural Selection or the
Preservation of Favoured Races in the Struggle for Life. John Murray, London,
UK.
Deubel, H., Schneider, W.X., 1996. Saccade target selection and object recognition:
evidence for a common attentional mechanism. Vision Res. 36, 1827–1837.
Donahoe, J.W., Burgos, J.E., Palmer, D.C., 1993. A selectionist approach to reinforcement. J. Exp. Anal. Behav. 60, 17–40.
Donahoe, J.W., Palmer, D.C., 1994. Learning and complex Behavior. Allyn & Bacon,
Boston, MA.
Dube, W.V., Balsamo, L.M., Fowler, T.R., Dickson, C.A., Lombard, K.M., Tomanari, G.Y.,
2006. Observing behavior topography in delayed matching to multiple samples.
Psychol. Rec. 56, 233–244.
Faisal, A.A., Selen, L.P., Wolpert, D.M., 2008. Noise in the nervous system. Nat. Rev.
Neurosci. 9, 292–303.
Freyberg, S., Ilg, U.J., 2008. Anticipatory smooth-pursuit eye movements in man and
monkey. Exp. Brain Res. 186, 203–214.
Goldberg, M.E., Wurtz, R.H., 1972. Activity of superior colliculus in behaving monkey.
I. Visual receptive fields of single neurons. J. Neurophysiol. 35, 542–559.
Grunow, A., Neuringer, A., 2002. Learning to vary and varying to learn. Psychon. Bull.
Rev. 9, 250–258.
Hafed, Z.M., Krauzlis, R.J., 2008. Goal representations dominate superior colliculus
activity during extrafoveal tracking. J. Neurosci. 28, 9426–9439.
Herman, J.P., Harwood, M.R., Wallman, J., 2009. Saccade adaptation specific to visual
context. J. Neurophysiol. 101, 1713–1721.
Hikosaka, O., Nakamura, K., Nakahara, H., 2006. Basal ganglia orient eyes to reward.
J. Neurophysiol. 95, 567–584.
Holland, J.G., 1957. Technique for behavioral analysis of human observing. Science
125, 348–350.
Hopp, J.J., Fuchs, A.F., 2004. The characteristics and neuronal substrate of saccadic
eye movement plasticity. Prog. Neurobiol. 72, 27–53.
Ikeda, T., Hikosaka, O., 2003. Reward-dependent gain and bias of visual responses in
primate superior colliculus. Neuron 39, 693–700.
Ingvaldsen, R.P., Whiting, H.T.A., 1997. Modern views on motor skill learning are not
‘representative’. Hum. Mov. Sci. 16, 705–732.
Johnson, M.H., 1990. Cortical maturation and the development of visual attention in
early infancy. J. Cogn. Neurosci. 2, 81–95.
Khurana, B., Kowler, E., 1987. Shared attentional control of smooth eye movement
and perception. Vision Res. 27, 1603–1618.
Kommerell, G., Olivier, D., Theopold, H., 1976. Adaptive programming of phasic and
tonic components in saccadic eye movements. Investigations of patients with
abducens palsy. Invest. Ophthalmol. 15, 657–660.
Kording, K., 2007. Decision theory: what “should” the nervous system do? Science
318, 606–610.
Kowler, E., 1989. Cognitive expectations, not habits, control anticipatory smooth
oculomotor pursuit. Vision Res. 29, 1049–1057.
Kowler, E., Anderson, E., Dosher, B., Blaser, E., 1995. The role of attention in the
programming of saccades. Vision Res. 35, 1897–1916.
Krauzlis, R.J., 2008. Eye movements. In: Squire, L.R., Berg, D. (Eds.), Fundamental
Neuroscience. , 3rd ed. Elesevier, Amsterdam, NL.
Krauzlis, R.J., Adler, S.A., 2001. Effects of directional expectations on motion perception and pursuit eye movements. Vis. Neurosci. 18, 365–376.
Land, M.F., Nilsson, D.-E., 2002. Animal Eyes. Oxford University Press, Oxford, UK.
Lauwereyns, J., Watanabe, K., Coe, B., Hikosaka, O., 2002. A neural correlate of
response bias in monkey caudate nucleus. Nature 418, 413–417.
Li, X., Basso, M.A., 2008. Preparing to move increases the sensitivity of superior
colliculus neurons. J. Neurosci. 28, 4561–4577.
Machado, A., 1989. Operant conditioning of behavioral variability using a percentile
reinforcement schedule. J. Exp. Anal. Behav. 52, 155–166.
Madelain, L., Champrenaut, L., Chauvin, A., 2007. Control of sensorimotor variability
by consequences. J. Neurophysiol. 98, 2255–2265.
Madelain, L., Harwood, M.R., Herman, J.P., Wallman, J., 2010. Saccade adaptation is
unhampered by distractors. J. Vis. 10, 29.
Madelain, L., Krauzlis, R.J., 2003a. Pursuit of the ineffable: perceptual and motor
reversals during the tracking of apparent motion. J. Vis. 3, 642–653.
Madelain, L., Krauzlis, R.J., 2003b. Effects of learning on smooth pursuit during transient disappearance of a visual target. J. Neurophysiol. 90, 972–982.
Madelain, L., Paeye, C., Wallman, J., 2008. Saccadic adaptation: reinforcement can
drive motor adaptation. J. Vis. 8, 919.
McLaughlin, S.C., 1967. Parametric adjustment in saccadic eye movements. Percept.
Psychophys. 2, 359–362.
Montagnini, A., Chelazzi, L., 2005. The urgency to look: prompt saccades to the
benefit of perception. Vision Res. 45, 3391–3401.
Nakamura, K., Hikosaka, O., 2006. Role of dopamine in the primate caudate nucleus
in reward modulation of saccades. J. Neurosci. 26, 5360–5369.
Neuringer, A., 2002. Operant variability: evidence, functions, and theory. Psychon.
Bull. Rev. 9, 672–705.
Neuringer, A., 2004. Reinforced variability in animals and people: implications for
adaptive action. Am. Psychol. 59, 891–906.
Nilsson, D.E., 2009. The evolution of eyes and visually guided behaviour. Philos.
Trans. R. Soc. Lond. B Biol. Sci. 364, 2833–2847.
Noto, C.T., Robinson, F.R., 2001. Visual error is the stimulus for saccade gain adaptation. Brain Res. Cogn. Brain Res. 12, 301–305.
Noto, C.T., Watanabe, S., Fuchs, A.F., 1999. Characteristics of simian adaptation fields
produced by behavioral changes in saccade size and direction. J. Neurophysiol.
81, 2798–2813.
Optican, L.M., Zee, D.S., Chu, F.C., 1985. Adaptive response to ocular muscle weakness in human pursuit and saccadic eye movements. J. Neurophysiol. 54,
110–122.
Paeye, C., Darcheville, J.-C., Madelain, L., 2007. Shaping induces long-term enhancements in pursuit-like-smooth movements under image-stabilization condition.
Perception 36, ECVP Abstract Supplement.
Paeye, C., Madelain, L., 2011. Reinforcing saccadic amplitude variability. J. Exp. Anal.
Behav. 95, 149–162.
Pelisson, D., Alahyane, N., Panouilleres, M., Tilikete, C., 2010. Sensorimotor adaptation of saccadic eye movements. Neurosci. Biobehav. Rev. 34, 1103–1120.
Posner, M.I., 1980. Orienting of attention. Q. J. Exp. Psychol. 32, 3–25.
Pryor, K.W., Haag, R., O’Reilly, J., 1969. The creative porpoise: training for novel
behavior. J. Exp. Anal. Behav. 12, 653–661.
Ratcliff, R., 2001. Putting noise into neurophysiological models of simple decision
making. Nat. Neurosci. 4, 336–337.
Reddi, B.A., Carpenter, R.H., 2000. The influence of urgency on decision time. Nat.
Neurosci. 3, 827–830.
Robinson, D.A., 1986. The systems approach to the oculomotor system. Vision Res.
26, 91–99.
Schroeder, S.R., Holland, J.G., 1968. Operant control of eye movements. J. Appl. Behav.
Anal. 1, 161–166.
Schroeder, S.R., Holland, J.G., 1969. Reinforcement of eye movement with concurrent
schedules. J. Exp. Anal. Behav. 12, 897–903.
Scudder, C.A., Batourina, E.Y., Tunder, G.S., 1998. Comparison of two methods of
producing adaptation of saccade size and implications for the site of plasticity.
J. Neurophysiol. 79, 704–715.
Shelhamer, M., Clendaniel, R.A., 2002. Context-specific adaptation of saccade gain.
Exp. Brain Res. 146, 441–450.
Steinbach, M.J., 1976. Pursuing the perceptual rather than the retinal stimulus. Vision
Res. 16, 1371–1376.
Stone, L.S., Beutter, B.R., Lorenceau, J., 2000. Visual motion integration for perception
and pursuit. Perception 29, 771–787.
Sugrue, L.P., Corrado, G.S., Newsome, W.T., 2004. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787.
Takikawa, Y., Kawagoe, R., Itoh, H., Nakahara, H., Hikosaka, O., 2002. Modulation
of saccadic eye movements by predicted reward outcome. Exp. Brain Res. 142,
284–291.
Thiele, A., Henning, P., Kubischik, M., Hoffmann, K.P., 2002. Neural mechanisms of
saccadic suppression. Science 295, 2460–2462.
Tian, J., Zee, D.S., 2010. Context-specific saccadic adaptation in monkeys. Vision Res.
50, 2403–2410.
Tomanari, G.Y., Balsamo, L.M., Fowler, T.R., Lombard, K.M., Farren, K.M., Dube, W.V.,
2007. Manual and ocular observing behavior in human subjects. Eur. J. Behav.
Anal. 8, 29–40.
van Beers, R.J., 2007. The sources of variability in saccadic eye movements. J. Neurosci. 27, 8757–8770.
van den Berg, A.V., 1988. Human smooth pursuit during transient perturbations of
predictable and unpredictable target movement. Exp. Brain Res. 72, 95–108.
Please cite this article in press as: Madelain, L., et al., Operant control of human eye movements. Behav. Process. (2011),
doi:10.1016/j.beproc.2011.02.009
G Model
BEPROC-2328;
No. of Pages 7
ARTICLE IN PRESS
L. Madelain et al. / Behavioural Processes xxx (2011) xxx–xxx
Wagenmakers, E.J., Brown, S., 2007. On the linear relation between the mean and the
standard deviation of a response time distribution. Psychol. Rev. 114, 830–841.
Wallman, J., Fuchs, A.F., 1998. Saccadic gain modification: visual error drives motor
adaptation. J. Neurophysiol. 80, 2405–2416.
Walls, G.L., 1962. The evolutionary history of eye movements. Vision Res. 2, 69–80.
Watanabe, K., Lauwereyns, J., Hikosaka, O., 2003. Neural correlates of rewarded and
unrewarded eye movements in the primate caudate nucleus. J. Neurosci. 23,
10052–10057.
7
Wells, S.G., Barnes, G.R., 1999. Predictive smooth pursuit eye movements during
identification of moving acuity targets. Vision Res. 39, 2767–2775.
Wolpert, D.M., Ghahramani, Z., 2000. Computational principles of movement neuroscience. Nat. Neurosci. 3 (Suppl.), 1212–1217.
Wolpert, D.M., Ghahramani, Z., Jordan, M.I., 1995. An internal model for sensorimotor integration. Science 269, 1880–1882.
Xu-Wilson, M., Zee, D.S., Shadmehr, R., 2009. The intrinsic value of visual information
affects saccade velocities. Exp. Brain Res. 196, 475–481.
Please cite this article in press as: Madelain, L., et al., Operant control of human eye movements. Behav. Process. (2011),
doi:10.1016/j.beproc.2011.02.009