* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download The Biological Bases of Time-to
Types of artificial neural networks wikipedia , lookup
Endocannabinoid system wikipedia , lookup
Artificial general intelligence wikipedia , lookup
Response priming wikipedia , lookup
Axon guidance wikipedia , lookup
Holonomic brain theory wikipedia , lookup
Biochemistry of Alzheimer's disease wikipedia , lookup
Neurotransmitter wikipedia , lookup
Activity-dependent plasticity wikipedia , lookup
Perception of infrasound wikipedia , lookup
Binding problem wikipedia , lookup
Neuroethology wikipedia , lookup
Convolutional neural network wikipedia , lookup
Nonsynaptic plasticity wikipedia , lookup
Psychophysics wikipedia , lookup
Neural oscillation wikipedia , lookup
Molecular neuroscience wikipedia , lookup
Time perception wikipedia , lookup
Multielectrode array wikipedia , lookup
Single-unit recording wikipedia , lookup
Neuroesthetics wikipedia , lookup
Mirror neuron wikipedia , lookup
Caridoid escape reaction wikipedia , lookup
Clinical neurochemistry wikipedia , lookup
Biological motion perception wikipedia , lookup
Central pattern generator wikipedia , lookup
Metastability in the brain wikipedia , lookup
Development of the nervous system wikipedia , lookup
Neural correlates of consciousness wikipedia , lookup
Circumventricular organs wikipedia , lookup
Premovement neuronal activity wikipedia , lookup
Neuroanatomy wikipedia , lookup
Biological neuron model wikipedia , lookup
Pre-Bötzinger complex wikipedia , lookup
Neuropsychopharmacology wikipedia , lookup
Neural coding wikipedia , lookup
Optogenetics wikipedia , lookup
Stimulus (physiology) wikipedia , lookup
Efficient coding hypothesis wikipedia , lookup
Nervous system network models wikipedia , lookup
Synaptic gating wikipedia , lookup
Time-to-Contact, Advances in Psychology series Heiko Hecht & Geert J. P. Savelsbergh (Eds.) 2003 Amsterdam: Elsevier - North-Holland CHAPTER 2: The Biological Bases of Time-to-collision Computation Barrie J. Frost Queen’s University, Kingston, Ontario, Canada Hongjin Sun McMaster University, Hamilton, Ontario, Canada ABSTRACT We begin the chapter by arguing that there may be several neural mechanisms that have evolved for computing time-to-collision (TTC) information as a way of controlling different classes of action. We then focus on single unit mechanisms responsible for processing the impending collision of a moving object towards a stationary observer. After discussing TTC processing in the invertebrate visual system, we describe our own work involving neurons in the pigeon nucleus rotundus that respond exclusively to visual information relating to objects that are approaching on a direct collision course, but not to visual information simulating movement towards those same stationary objects. Based on the recorded neuronal responses to various manipulations of the stimuli, we classified these looming sensitive neurons into three different types of looming detectors based on the temporal differences in neuronal responding relative to the moment of collision. We also described quantitative models for these looming detectors as a way of explaining their physiological response properties. 14 Barrie J. Frost and Hongjin Sun 1. Introduction Information about the time-to-collision or time-to-contact (TTC) has important consequences for the survival of countless species and for their skilled interaction with both the inanimate and animate objects in their environments. As a consequence it appears very probable that there may be several neural mechanisms that have evolved to compute TTC information to control different classes of action, and even different mechanisms within the same animal for different functions. For example, it appears unlikely that mechanisms that have evolved in birds for avoidance of rapidly approaching objects such as predators, where critical and rapid evasive maneuvers are required, are the same mechanisms that control pinpoint landing on branches. In the former case the motion of the approaching predator will primarily determine TTC, whereas in the latter it is only the self-motion of the animal approaching the stationary branch that determines TTC. Of course there may be many instances where both self-motion and motion of another animal determine TTC. For an excellent review that puts TTC in a much broader context the reader is referred to a paper by Cutting, Vishton and Braren (1995). One way that may help conceptualize these factors is to subdivide the primary stimulus determinants of TTC on the one hand, and the general nature of responses controlled by the information on the other, and place these in a simple 2 x 2 table as illustrated in Table 1. Here we have divided the world simply into stationary and moving objects on the vertical axis, and Source of Looming Stimulus Behavioural Output Self-motion towards stationary objects 1 • • • Insect’s landing Bird’s landing Human or gerbil’s approaching towards target 2 • • Avoiding obstacles Avoiding cliffs and drop offs Moving objects (towards and away from the observer) 3 • • • • Pursuit – prey capture Pursuing mates Flock formation Ball catching 4 • • Predator avoidance Avoiding aggressive encounters Approach Avoidance Table 1: Situations Requiring TTC Information the behavioural responses into approach and avoidance on the other. Examples of TTC studies falling in cell 1 (Stationary objects/Approach behaviour) are the landing response of the milkweed bug, Oncopeltus fas- The Biological Bases of Time-to-collision Computation 15 ciatus (Coggshall, 1972) and the fly (Wagner, 1982). The aerodynamic folding of gannet wings just prior to their entry into water (Lee & Reddish, 1981), birds landing on stationary perches (Lee, Davies, Green & Van der Weel, 1993) or human subjects braking to avoid collision with stationary barriers (Sun & Frost, 1997) or gerbil behaviour of running towards target (Sun, Carey & Goodale, 1992) are other examples that fall into this category. Examples of behaviour falling in cell 2 (Stationary objects/Avoidance) would involve negotiating paths through a cluttered environment where obstacles have to be avoided. This might include steering around barriers, and avoiding holes or sudden drop offs. There appear to be few studies of TTC detection in this category, but Cutting et al.’s (1995) study of path interceptions may be relevant. Prey capture by predators and pursuit chasing during mating could well satisfy entry into cell 3 (Moving objects/Approach), although not all studies of this behaviour have focused on TTC information. Ball catching behaviour, and batting in sports seem also to be appropriate exemplars of this category. Escape from rapidly approaching predators or threatening rivals in territorial mating would be prime example of cell 4 (Moving objects/Avoidance). Throughout the animal kingdom the sight of a rapidly approaching object almost universally signals danger and elicits an escape or avoidance response. When confronted with such a looming stimulus, the visual system must determine precisely the 3D flight path, and compute the TTC of the object, to provide the information necessary for eliciting and controlling the appropriate evasive action (Fishman & Tallaroco, 1961; Schiff et al., 1962; Schiff, 1965; Hayes & Saiff, 1967; Tronick, 1967; Bower et al., 1970; Ball & Tronick, 1971; Dill, 1974; Ingle & Shook, 1983; Yonas & Granrud, 1985). Our own work on neurons in the pigeon nucleus rotundus of pigeons clearly fits in this category because these neurons respond only to the direct collision course of approaching objects (Wang & Frost, 1992, Sun & Frost, 1998), and not to simulation of the movement of pigeons toward the same stationary objects (Sun & Frost, submitted). Also the work on locust looming detectors would fit this category because of the demonstrated elicitation of jumping and flying by the same expanding stimulus patterns that optimally excites the Lobula Giant Movement Detector (LGMD) and the Descending Contralateral Movement detector (DCMD) neurons (Rind & Simmons, 1992, 1999; Hatsopoulos, Gabbiani, Laurent, 1995). Of course it should be remembered that the necessity to compute TTC first requires that any object or surface be indeed on a collision course if the present path of the observing or approaching animal is maintained. Gibson (1979) in his classic work on ecological optics suggests that symmetrical expansion of the images of objects specifies direct approach along a course that will ultimately result in collision with continuous motion. The advantage of using 16 Barrie J. Frost and Hongjin Sun this strategy is that it can be computed using monocular information alone. It is possible that animals with well-developed binocular stereoscopic visual systems, and subpopulations of neurons that respond to stereoscopic motion directions specifying object-motion paths directly toward the animal, might also be used for TTC computations. In this work and his other writings, Gibson also made the clear distinction between collisions with stationary objects occasioned by the motion of the observer (row 1 in Table 1) and other cases where it is the approaching object’s motion itself that will result in collision if it continues along this path (row 2 in Table 1). In this chapter we will focus primarily on research that falls in cell 4 simply because it appears that most of the empirical studies about neural mechanisms of TTC have used stimulus arrangements that simulate events that fall into this category, that is, a rapidly approaching object on a direct collision course with the observing animal, and which might therefore require some sort of evasive action or avoidance response on the part of the animal. In the other category of object motion where the observing animal is trying to arrange a collision with the moving object such as the prey capture behaviour of dragonflies (Olberg, Worthington & Venator, 2000), similar processing mechanisms may occur. 2. TTC in the invertebrate visual system Flying insects have long been used as model systems because they exhibit spectacular aerial performance and accomplish this with relatively simple neural computational mechanisms. Moreover since the same neurons can be identified from animal to animal the neural circuitry is often amenable to analysis. Two such neurons, LGMD and DCMD in the locust, that are synaptically linked, have been shown to be selectively responsive to approaching objects (Rind & Simmons, 1992; Rind, 1997; Hatsopoulos et al., 1995). Neurons that respond to changes in depth have also been found in optic lobes of the hawk moth, Manduca sexta (Wicklein & Strausfeld, 2000), but these may be examples of neurons computing approach and recession for the control of self motion, in this case controlling the hovering flight in front of flowers during nectar collection, rather than for the computation of TTC. Because the DCMD neurons can be readily recorded extracellularly, have very large receptive fields, and respond well to the movement of objects, they have been studied extensively for many years. Schlotterer (1977) was the first to use approaching stimuli to show that DCMD neurons were more responsive to approaching objects than other 2D patterns of movement. Rind and her colleagues, and Laurent and his colleagues have extensively studied these neurons using a variety of stimuli and confirmed that symmetrical expansion gener- The Biological Bases of Time-to-collision Computation 17 ated by an approaching stimulus object is the critical stimulus variable that optimally fires these cells. The allocation of the LGMD - DCMD neurons to cell 4 of our schema presented in Table 1 is justified by their connection to pre-motor interneurons and motor-neurons known to be involved in flying and jumping (Burrows & Rowell, 1973; Pearson et al., 1980; Simmons, 1980). This is further supported by the studies of Robertson and Johnson (1993a, 1993b) who have shown in tethered, flying locusts, that approaching objects elicit a steering avoidance response when the approaching object reaches a critical angular size, thus indicating that some thresholding probably occurs in this pathway. What are the critical features of a symmetrically expanding image that these locust DCMD neurons are responding to that generates their specificity to approaching objects? From an analysis of the several possible cues available in the monocular image Rind and her colleagues (Simmons & Rind, 1992; Rind & Simmons, 1992) have shown that these neurons do not register changes in overall luminance since they respond in a similar manner when light objects approach as when dark objects approach, and their responses were much smaller to sudden luminance change per se. Divergence of two lines moving in opposite directions, to partially represent the opposite contours of a symmetrically expanding object, also did not adequately stimulate DCMD neurons, but increasing the amount of edges in the Receptive Field (RF) and increasing the velocity of edges appeared to be the critical trigger features. Judge and Rind (1997; see also Rind & Simmons 1999) have shown that these locust looming sensitive neurons are very tightly tuned to the direct collision course. Stimulation of the locust retina in one area suppresses LGMD response to a second stimulus presented elsewhere in the visual field, thus indicating there are lateral inhibitory mechanisms operating. Indeed if the appropriate experiments were to be performed one might well find the RF characteristics are similar to those found in the tectofugal or collicular-pulvinar pathway of vertebrates where a directionally specific, double opponent RF organization occurs to ensure that these cells respond to moving objects, and not to the large patterns of optic flow produced by the animal's self motion (Frost 1978; Frost, Scilley & Wong, 1981; Frost, Cavanagh & Morgan, 1988, Sun, Zhao, Southall & Xu, 2002). From a functional point of view this also implies that the LGMD neurons might be interested in approaching objects that fall into cell 4 of our matrix, and not in the locust's approach toward stationary features in its environment. According to Rind and Simmons (1999) the specificity of the LGMD for approaching images is generated by a “critical race over the dendrites of the LGMD in the optic lobe”. The two competitive forces in the race are the excitation produced by the moving edges of the expanding image, and lateral inhibition mediated by neurons in the medulla also synapse onto the LGMD. Rind and Bramwell (1996) have produced a neural network model which seems to support this view and have also shown through electron microscopy that the anatomical 18 Barrie J. Frost and Hongjin Sun arrangement of presynaptic connections to LGMD are compatible with this interpretation. Hatopoulus, Gabbiani and Laurent (1995) have also investigated the LGMD of locusts, and shown that this neuron fires with an increasing rate as an object approaches, then peaks, and drops off just before collision occurs. They have shown that the responses are typically brisker for fast moving or smaller objects, but the peak firing rate does not appears to solely depend on the approach speed or object size. They describe the peak as always exhibiting a constant latency after the time at which the object reaches a fixed angular threshold size on the eye (Gabbiani, et al, 1999). These authors have suggested that the behaviour of the LGMD is best described by the following equation: f(t) = C × θ ′(t ) × e-αθ (t) (1) Here, θ is visual angular subtense, C is a proportionality constant, and is a constant for a particular neuron. In contrast to Rind and her associates view, these authors in recent papers (Gabbiani, et al, 2001; Gabbiani et al., 2002) have suggested that the LGMD post-synaptically multiplies an excitatory and inhibitory input via two different parts of LGMD neuron’s dendritic tree. In order to provide evidence in support of this model these authors (Gabbiani et al., 2002) have selectively activated and deactivated pre-and post synaptic inhibition, and have found that it is post-synaptic inhibition that plays a critical role in shaping the temporal response profile of these neurons, and this indicates that the multiplication takes place within the LGMD neuron itself. These findings are noteworthy for two reasons: in the first place they show in a detailed way how these computations which provide information about looming object are accomplished within the neural machinery of the LGMD and its synaptic connections, and secondly they provide one of the first pieces of clear evidence for how multiplication (and division) is accomplished in the nervous system. 3. Neurons that compute tau in the pigeon brain A number of behavioral studies have revealed that the tectofugal pathway in vertebrates might be involved in processing the visual information necessary for generating such escape or avoidance action. Electrical stimulation experiments indicated that the anuran optic tectum is involved in triggering both prey-catching and also various kinds of avoidance behaviours (Ewert, 1984), and ablation of the optic tectum resulted in abolition of all visually guided preycatching and visual avoidance behaviour (Bechterew, 1984, cf: GrüsserCornehls, 1984). Electrical or chemical stimulation of the superior colliculus in The Biological Bases of Time-to-collision Computation 19 rats also results in defensive-like reactions (Redgrave et al., 1981; Sahibzada et al., 1986; Dean et al., 1988) and is associated with large increases in blood pressure and heart rate (Keay et al., 1988). Pigeons with bilateral lesions of the optic tectum or/and the nucleus rotundus not only showed substantial impairment in intensity, colour, and pattern discrimination, but also exhibited severe deficits in visually guided orientation, escape or avoidance behaviour (Hodos & Karten, 1966; Hodos, 1969; Hodos & Bonbright, 1974; Jarvis, 1974; Bessette & Hodos, 1989). Wild rats with collicular lesions may ignore an approaching human (Blanchard et al., 1981) and similar results have been reported in hamsters and gerbils (Ellard & Goodale, 1986; Northmore et al., 1988). This evidence provides a vivid illustration of the importance of the tectofugal pathway in guiding orientation, detecting approaching objects, and generating escape or avoidance behaviours. Over the years several investigators have claimed that they have encountered cells that respond specifically to objects approaching the eye on a direct collision course. For example, as early as 1976 Grüsser and Grüsser-Cornehls (1976) and later Ewert (1984) reported that some frog and toad tectal neurons respond vigorously to stimuli moving on paths directly towards the eye. However from these early studies many of the appropriate controls were not performed to conclusively exclude the possibility that these neurons were not simply responding to some aspect of the lateral motion of an approaching stimulus. It should be remembered that as an approaching image expands, obviously there will be 2D motion of the edges of the object and its textures and if the expansion is placed asymmetrically over a standard 2D directionally specific motion it could artifactually stimulate the neuron to give a false impression it is responsive to approaching stimuli. We also had encountered cells we thought were responding to the direct approach path of moving objects in 1983, but it was only when we had extremely well-controlled stimuli, which we could systematically vary in their simulated 3D paths, that we could finally convince ourselves that these neurons were indeed coding some aspect of 3D motion. In 1992 Wang and Frost showed that some neurons located in the dorsal posterior regions of the nucleus rotundus of pigeons responded specifically to the direct approach direction of a soccer ball pattern. Using the 3D imaging capabilities of a Silicon Graphics computer we were able to move this soccer-ball stimulus in any trajectory in 3D space. By performing very time consuming 3D tuning curves on these cells we were able to show that this rotundal subpopulation would only respond when the soccer ball stimulus was on a direct collision course with the bird’s head. We chose a soccer ball because the spaceaverage mean luminance did not change as this stimulus expanded and contracted in size (especially when the object moved against a stationary background with the same texture pattern), and it provided many moving and expanding/contracting elements that might be necessary for these neurons to re- 20 Barrie J. Frost and Hongjin Sun spond. Earlier studies, and often some more recent ones, use a simple expanding square or circle where changes in luminance obviously occur concurrently with the expansion/contraction of stimuli, and this necessitates several other controls to rule out this variable as the major contributor to the responsiveness of the neuron. Also other studies have not specifically performed 3D tuning curves to quantify the true directional tuning of neurons of this type. Figure 1 shows the typical 3D tuning curves of one of these rotundal neurons. Figure 1: A. A soccer-ball-like stimulus pattern consisting of black and white panels, was moved along simulated 3D trajectories 45° apart in spherical coordinates. The diagram illustrates the 4 planes along which stimuli were moved. B. A typical single neuron from the nucleus rotundus of pigeons exhibiting clear selectivity for a looming visual stimulus. Firing rate is plotted for the different directions of motion of the soccer-ball stimulus in 3D space. Each direction of motion was presented 5 times in a randomly interleaved sequence, and the values plotted represent the mean number of spikes for each 3D direction. Note that in the standard X-Y (tangent screen plane, or Azimuth = 90°) plot, there is no indication of directional preference, and firing rate is quite low. However, for the 0°azimuthal plane (Z-axis) there is a strong preference for stimuli directly approaching the bird (0°). Polar tuning plots for directions specifying the azimuthal 45° and 135° planes likewise show no strong preference for any direction. Thus it is only the direct collision course or looming direction that produces an increased response in these neurons, and this pattern of activity was typical of the 27 neurons studied in the dorsal posterior area of the nucleus. The Biological Bases of Time-to-collision Computation 21 Even with 26 directions, 45 degrees apart these are relatively crudetuning curves, so in a few cases we have used a much narrower range of directions after having first performed the broad 3D tuning and found them to be very tightly tuned indeed (Sun & Frost, submitted). In fact the half-width and half-height of detailed tuning curves like those illustrated in Figure 2 is about 3 degrees, where the rotation is around a point halfway along the Figure 2: Fine tuning of two cells located in the pigeon dorsal nucleus rotundus. First these cells were presented with a soccer ball stimulus that moved in 26 directions 45 degrees apart in 3D space, and they only responded to the direct collision course direction. The graphs shown here are the fine grained tuning curves and show that when the soccer ball, which traveled along a simulated path of 15 meters, was rotated by small amounts each time passing through the center of the path, the cells reduced their firing substantially. The few degrees of rotation of the path indicate that now the soccer ball would travel in a “near miss” and not collide with the bird. simulated 15 meter path taken by the approaching stimulus. This means in simple terms that a stimulus that depicts a very “near miss” of the bird’s head will only fire the neurons minimally, and one that is a clear “miss” will not evoke any response at all. Perhaps the most important defining character of these neurons’ responses, in addition to their sensitivity to the direct collision course direction, was the constant time they fired before collision, irrespective of the size of the simulated approaching soccer-ball, or of its approach velocity (Wang & Frost, 1992). This indicated to us that these neurons might well be computing the opti- 22 Barrie J. Frost and Hongjin Sun cal variable tau that had been suggested by Lee (1976) to provide important information about the TTC with approaching objects. In our original paper (Wang & Frost, 1992) we reported that there were a variety of times before collision that the population of neurons exhibiting these characteristics showed. This can be seen in Figure 3A. Figure 3: A. Distribution of different response onset time for 27 looming cells from the dorsal posterior zone of nucleus rotundus of pigeons. Although different cells exhibit different values of TTC, individual cells (B) show remarkable consistency even when velocity or size of stimulus is varied. But for a particular neuron, the variation in its response onset was remarkably constant (see Figure 3B) on repeated trials and with stimuli of different sizes and velocities. This variation in the population is precisely what is needed if other factors, such as recognition of what the incoming object is, can jointly influence the time selected to perform escape responses of different sorts, each of which may have characteristic time requirements for their optimal deployment. The Biological Bases of Time-to-collision Computation 23 Clearly the characteristics of these rotundal neurons suggested to us that they might be involved among other things in predator avoidance and thus fall in to cell 4 of our classification system shown in Table 1. To provide some evidence for this we tested a few birds under a lightened anesthesia at the conclusion of their recording session. Under deep anesthesia no electromyelograhic signals (EMGs) are obtained from animals, but as the anesthetic is lightened and clearly before any pain stimuli can be experienced, it is possible to obtain good clear muscle responses. These responses do not result in any overt movement of the animal, but can be very useful in indicating what major muscle groups might be involved in a response system normally associated with a stimulus. In this case we recorded from the large pectoralis muscles that power the wings for flight. When we presented the approaching soccer-ball stimulus under these conditions we found that first the tau cells responded with their characteristic maintained burst of firing, then the pectoralis EMGs occurred some 200 milliseconds later, and then finally the heart rate went more slowly up to levels near 300% of the resting rate. These responses again were incredibly specific, and only occurred when the soccer-ball was on a direct collision course with the bird. Near misses and directions 180 degrees away showed no increased EMGs or increases in heart rate. Data typical of these experimental findings can be seen in Figure 4. Although only correlative, we feel this constellation of activity in these tau neurons, and the increased wing EMG and heart rate are indicative of a flight response elicited by the rapidly approaching ball. In a more detailed recent study Sun and Frost (1998) have again confirmed the presence of a population of neurons in nucleus rotundus of pigeons that only respond when the soccer-ball stimulus is on a direct collision course with the bird’s head. Additionally, we found that these neurons only responded when our computer simulated the approach of a moving object towards the bird (stimuli falling into cell 4 of the matrix), and not when the complex stimulus pattern was configured to simulate the bird moving towards the same stationary soccer-ball (stimuli falling into cell 1, or possibly 2 of the matrix) (Sun & Frost, submitted). 24 Barrie J. Frost and Hongjin Sun Figure 4: Heart rate and pectoralis muscle EMGs recorded concurrently with single cell response rate from a looming selective rotundal neuron. Note that the "looming cell" begins firing first, then the muscle response occurs, and then heart rate increases dramatically when the soccer-ball looms toward the bird (A). B. No responses occur when the ball moves along the same path but in the opposite direction directly away from the bird. Bars under the visual response histograms indicate the duration of the visual stimulus. Data collection for the looming-selective neuron was terminated with stimulus offset. The neuronal data and EMGs represent the summed activity over 5 sweeps of the stimulus whereas the heart rate data represent the means and standard errors for the same 5 sweeps. Simulated size of stimulus was 30 cm, path length 15 m, and velocity 375 m/s. To do this we placed the soccer ball against a stationary background, which consisted of a checkerboard pattern. When the soccer-ball was moved in a trajectory towards the bird (symmetrical expansion) while the background remained stationary, the neurons responded in the typical way and identically to the case where no background was present. However, when the background was moved along the same trajectory as the soccer-ball, so as to show a similar but delayed expanding pattern, the neurons did not respond. This latter configuration formed the precise simulation of the bird approaching a stationary soccer-ball that remained a constant distance in front of the background “brick wall”. The stimulus conditions simulating a soccer ball approaching the bird, and the bird approaching a stationary soccer ball are shown in Figure 5. The Biological Bases of Time-to-collision Computation 25 Figure 5: Schematic diagram of stimulus conditions presented to pigeons. The left portion of the figure illustrates the kind of object movement and its background in the simulated display, while the right portion represents the image change on the screen for the corresponding simulation illustrated on the left. In A, the soccer ball object is presented against a blank background, and the “path” simulates direct approach toward the bird. In B, the same looming object is presented against a stationary textured checkerboard background. In C, both object and textured background move (at the same speed) toward the bird, which simulates the bird’s self-motion toward a stationary soccer ball. Note that the expansion pattern of motion of the object is identical in the three conditions. It must be emphasized that the expansion pattern of the soccer-ball was identical in these to two cases of moving object and moving bird simulations, yet in the former the cells responded vigorously, while in the latter they were essentially silent. Figure 6 shows the responses of a tau neuron to an approaching soccerball, and also a simulation of the bird approaching a stationary soccer-ball where the rate of expansion is identical in both cases. 26 Barrie J. Frost and Hongjin Sun Figure 6: Comparison of the responses Peri-Stimulus Time Histograms (PSTHs) for a single tau neuron to a series of stimuli (soccer-ball) of varying sizes swept along the direct collision course path toward the bird. Responses are the sum of 5 sweeps and are referenced to time zero, which is the time when the stimulus would have contacted the bird. The looming object was presented against a white non-textured background (A), a stationary textured checkerboard background (object-motion) (B), and a looming background moving at the same speed behind the object (C), as shown in Figure 5. The latter condition simulated the approach of the animal toward the stationary ball and background (selfinduced motion). Responses were similar to those produced by the looming object against a blank background and a stationary checkerboard background. The magnitude of responses (maximal firing rate) were similar across different object sizes. Note that the neuron did not fire to the self-motion display, even though the soccer ball’s image was expanding in the same way in B and C. This implies this tau neuron is exclusively selective for “object motion in depth”. The simulated path for the object was 15 m in length and the simulated object size varied from a diameter of 10 cm to 50 cm. Velocity was 375 cm/s. The Biological Bases of Time-to-collision Computation 27 We have also found that not all of the neurons in nucleus rotundus appear to be computing the tau function (Sun & Frost, 1998). Histological examination indicated that those neurons were distributed in a larger anatomical region (dorsal rotundus) as opposed to dorsal posterior rotundus in our earlier discovery by Wang and Frost (1992). In fact, half of the neurons seem to be clearly responding in this fashion, that is, they suddenly start firing at a particular and constant time before the collision event and maintain this high firing rate throughout the remainder of the approach sequence. Roughly a quarter of the neurons that show selectivity to an approaching object show a response that begins earlier for larger objects, or soccer-ball stimuli approaching at slower velocities. In detailed mathematical arguments and quantification of the timing of the response, Sun and Frost (1998) show that these neurons are computing the rate of expansion, rho, of the approaching object. Finally, the remaining quarter of the looming neurons appear to be computing the very same function which best describes the locust looming detector (Hatsopoulos et al., 1995; Gabbiani et al., 1999). An example of the response patterns of each of these three classes of neuron is shown in Figure 7. Sun and Frost (1998) show that on several multidimensional plots these three classes of neurons form very distinct and tight clusters which indicate that there is not some simple underlying continuum that we have arbitrarily divided into three separate groups, but that these are genuine types of neurons each computing the following three functions. (1) Rho ρ (t ) = θ ′(t) (2) (2) Tau τ (t) ≈ θ (t) θ ′(t ) (3) (3) Eta η(t) = C × θ ′(t ) × e-αθ (t) (4) 28 Barrie J. Frost and Hongjin Sun Neuron a (τ) Neuron b (ρ) Neuron c (η) Figure 7: Based on the differences in the time course of the neuronal responses relative to the moment of collision, the looming sensitive neurons in nucleus rotundus have been classified into three distinct classes. This figure shows the response pattern (PSTHs) for a typical neuron in each of the three classes (neuron a, b, and c for tau, rho, and etc respectively) to a series of stimuli (a simulated moving sphere with a soccer-ball pattern) of varying sizes (A) and of varying velocities (B), moving along the direct collision course path toward the bird. Responses are the sum of 5 trials and are referenced to time zero, which is the time when the stimulus would have collided with the bird. The simulated path was 15 m in length. In (A) velocity for neuron b was 375 cm/s and for neurons a and c was 500 cm/s. In (B), object size was 30 cm for all three neurons. Note that for the neuron a, the timing of the response remains invariant despite substantial changes in size and velocity, whereas for neuron b and neuron c, the timing depends on object size and velocity, with larger or slower objects evoking an earlier response. The Biological Bases of Time-to-collision Computation 29 An example of such clustering is shown in Figure 8 taken from Sun and Frost (1998). Figure 8: Quantitative examination of the timing of the response for the population of nucleus rotundus looming-sensitive neurons when presented with approaching objects that varied in size or velocity. The variances (standard deviation) of Tconset were plotted along the x axis, and the average drop-off in firing rates at the time of collision, relative to the response peak (%), were plotted along y axis. The data points are clustered in three separate regions, therefore this population of neurons can be classified into three distinct groups ( ○ tau neuron, ▲ rho neuron, ● etc neuron). B B What is rather amazing about these findings is that all looming cells are accounted for. Each rotundal neuron that responds specifically to an object approaching on a collision course with the bird is either a rho neuron, a tau neuron or an eta neuron. Usually in single unit recording studies there are “junk” or “intermediate” categories required for neurons that don’t seem to fit the major classifications that apply to the other cells. But here we have no residual cells to account for! 30 Barrie J. Frost and Hongjin Sun The function of the tau neurons is quite clear; they can provide the animal with useful information about the TTC of the object that is approaching on a direct collision course. The rho cells obviously compute rate of expansion information that is required to compute tau, that is, the denominator in the tau equation. Also the eta function also contains a rate of expansion term and rho neurons could provide essential input into this computation also. Indeed the multiplication and division required in these three equations that describe the total set of looming detectors we have found in the pigeon nucleus rotundus may well be performed by biophysical operations similar to those described in the recent papers by Peña and Konishi (2001) and Gabbiani et al. (2002). In this latter paper Gabbiani et al. (2002) suggest that the eta operation is implemented within single LGMD neurons by exponentiation of the sum of a positive excitatory postsynaptic potential representing the logarithm of angular velocity, and a negative postsynaptic potential representing angular size. 4. Model We have developed neuronal models to explain the physiological response properties of our pigeon looming sensitive neurons. These models were created based on the physiological responses (both qualitative and quantitative data) recorded to various stimulus conditions (including direct manipulation of various optical variables that could be specified by a looming object). The models also take into account the physiological response properties and anatomical connection of the optic tectum that sends a major input to nucleus rotundus. For these rotundal looming detectors, the RF could be composed of a radial arrangement of concentric arrays of RFs of simple local motion detectors (possibly tectal neurons), with the centre of expansion overlapping with the centre of RF radial layout. These tectal neurons would respond to movements that are oriented radially from the centre of the concentric array and they then converge onto rotundal looming detectors. This arrangement is shown in Figure 9. The Biological Bases of Time-to-collision Computation 31 Figure 9: General model of receptive field (RF) organization of rotundal looming sensitive neurons. The RF is composed of a number RF subunits, each of which corresponds to the RF of a tectal local motion sensitive neuron. These tectal RFs are arranged on the circumference of a series of concentric circles (or rings) with different radii. The converging input from each concentric ring of RF subunits enable both spatial and temporal summation to signal symmetrical image expansion. Note that this general model explains qualitatively the vigorous firing of rotundal looming neurons to symmetrical image expansion. The spatial and temporal summation of the activity of these small RFs could provide a basis for the rotundal neurons’ large receptive field size, the strong directional selectivity for motion along the direct collision course (indicated by symmetrical expansion from a stationary centre). The opponentdirection centre-surround organization of these tectal units (Frost & Nakayama, 1983) could contribute to the silence of these rotundal neurons to stimulation with a self-motion display, in which local segments of the background move in the same direction as that of the object in the adjacent region across the object boundary. A series of quantitative models were also formulated to account for the specific properties of the time course of rotundal neurons response to impending collision. When an object approaches the eye, the visual angular subtense θ(t) and rate of change of visual angle θ'(t) form the basic building blocks for the 32 Barrie J. Frost and Hongjin Sun calculation of those optical variables that could signal impending collision. For example, TTC can be signaled by the optic variable tau, which can be specified by the ratio of θ' (t) over θ(t) (equal to 1/tau). One way to generate this ratio is through response to the instantaneous values of θ'(t) and θ(t) individually by two sets of neurons, and then through neuronal interaction to generate the ratio. Alternatively, if the instantaneous value of θ'(t) can be encoded (perhaps through the velocity selectivity of local motion detectors, e.g. tectal neurons), then visual angle θ(t) could be registered through the spatial location of the RF of the responding local motion detector relative to the centre of the concentric array of RFs of these local motion detectors. With the centre of image expansion overlapping with the centre of such a concentric array of RFs, a fixed ratio of θ'(t) over θ(t) could be hardwired in the brain to create a threshold for the optical variable 1/tau, so that Rts neurons would only start to fire when the predefined ratio increase to a fixed level. Consequently, such neurons would only fire when TTC decrease to a threshold level as the object approaches (see Figure 10). A B A' preferred velocity A radius of the circle of subunits Figure 10: Computation of tau. In this model, rate of expansion can be signalled by the spatial summation of the ring of subunits, each with a certain local velocity selectivity (indicated by the size of arrow in A). The instantaneous visual angle could be registered by the distance of this ring of RFs from the centre of the circle at the same time (radius of expansion). To calculate tau, the preferred velocity of each local tectal unit should be linearly related to the radius of the concentric ring of RF subunits located at the same distance (shown in B). A similar quantitative model of the other two types of rotundal cells has been provided in a recently submitted paper (Sun & Frost, submitted). Our models include alternative one and two component versions where size encoding is realized either from intrinsic pattern of RF organization (one component model) The Biological Bases of Time-to-collision Computation 33 or from the simultaneous coding of visual angle in spatial overlapping neurons, and then converges onto the rotundal neuron. These neuronal models provide explanations for the various physiological responses found in different classes of rotundal looming detectors. Not only are they potentially informative for understanding the neuronal coding of motion in depth, but these models could provide important insights for robotics and machine vision. A neural net model In an interesting recent paper Karanka-Ahonen, Luque-Ruiz and LópezZamora (2002) present the results of a neural network employing backpropagation, implemented in T learn, that learned to predict collision of objects of different sizes that moved towards it. The network consisted of 3 layers with 47 nodes, 40 of which were input nodes, three were hidden nodes, one was the output node and there were three context nodes. The nodes of each layer were completely connected to those of the preceding layer and connections from the context nodes to the context nodes were one to one and reciprocal. After training the network was tested with new objects that differed in size and distance from the original training set and it was found that 75% of collisions were correctly predicted. Interestingly, the behaviour of the hidden units appeared to be very similar to some of the units we found in the pigeon nucleus rotundus, in that the output units predictive response came earlier for larger objects, like our eta cells. 5. Conclusions In this chapter we suggest there may be several different classes of neurons that are each specialized to compute TTC information for different subsets of tasks critical for the survival of animals in their natural environments. We then examine one class of these neurons in detail and find there may be several subroutines required before the tau or eta computations can be performed. What is interesting is that there appears to be convergence in solutions between looming sensitive neurons found in invertebrates and vertebrates in that the eta function describes well some of the neurons in both locust and pigeon visual systems. The functional significance of having both tau and eta neurons is still not obvious, although it is provocative that learning networks appear to give rise to both types of hidden neurons. 34 Barrie J. Frost and Hongjin Sun REFERENCES Ball, W. & Tronick, E. (1971). Infant responses to impending collision. Science, 171, 818-820. Bechterew, W. (1884). Über die Function der Vierhügel. Pflügers Arch, 33, 413. Bessette, B. B. & Hodos, W. (1989). Intensity, colour, and pattern discrimination deficits after lesions of the core and belt regions of the ectostriatum. Visual Neuroscience, 2, 27-34. Blanchard, D. C., William, G., Lee, E. M. C. & Blanchard, R. J. (1981). Taming of wild rattus norvegicus by lesions of the mesencephalis central gray. Physiological Psychology, 9, 157-163. Bower, T. G. R., Broughton, J. M. & Moore, M. K. (1970). Infant responses to approaching objects: an indicator of response to distal variables. Perception and Psychophysiology, 9, 193-196. Burrows, M. & Rowell, C. H. F. (1973). Connections between descending visual interneurons and metathoracic motoneurons in the locust. Journal of Comparative Physiology, 85, 221234. Coggshall, J. C. (1972). The landing response and visual processing in the milkweed bug, Oncopeltus fasciatus. Journal of Experimental Biology, 57, 401-413. Cutting, J. E., Vishton, P. M. & Braren, P. A. (1995). How we avoid collisions with stationary and moving obstacles. Psychological Review, 102, 627-651. Dean, P., Mitchell, I. J. & Redgrave, P. (1988). Response resembling defensive behaviour produced by microinjection of glutamate into superior colliculus of rats. Neuroscience, 24, 501-510. Dill, L. M. (1974). The escape response of the zebra danio (Brachydanio rerio). I. The stimulus for escape. Animal Behaviour, 22, 771-722. Ellard, C. G. & Goodale, M. A. (1986). The role of the predorsal bundle in head and body movements elicited by electrical stimulation of the superior colliculus in the Mongolian gerbil. Experimental Brain Research, 71, 307-319. Ewert, J.-P. (1984) Tectal mechanisms that underlie prey-catching and avoidance behaviours in toads. In H. Vanegas (Ed.), Comparative neurology of the optic tectum (pp. 247-416). New York, NY:Plenum Press. Fishman, R. & Tallaroco, R. B. (1961). Studies of visual depth perception. II. Avoidance reaction as an indicator response in chicks. Perceptual and Motor Skills, 12, 251-257. Frost, B. J. (1978). Moving background patterns alter directionally specific responses of pigeon tectal neurons. Brain Research, 151, 599-603. Frost, B. J., Cavanagh, P. & Morgan, B. (1988). Deep tectal cells in pigeons respond to kinematograms. Journal of Comparative Physiology, 162, 639-647. Frost, B. J. & Nakayama, K. (1983). Single visual neurons code opposing motion independent of direction. Science, 220, 744-745. Frost, B. J., Scilley, P. L. & Wong, S. C. P. (1981). Moving background patterns reveal double opponency of directionally specific pigeon tectal neurons. Experimental Brain Research, 43, 173-185. The Biological Bases of Time-to-collision Computation 35 Gabbiani, F., Krapp, H. G., Koch, C. & Laurent, G. (2002). Multiplicative computation in a looming-sensitive neuron. Nature, 420, 320-324. Gabbiani, F., Krapp, H. G., & Laurent, G. (1999). Computation of object approach by a widefield, motion- sensitive neuron. Journal of Neuroscience, 19, 1122-1141. Gabbiani, F., Mo, C. H., & Laurent, G. (2001). Invariance of angular threshold computation in a wide-field looming-sensitive neuron. Journal of Neuroscience, 21, 314-329. Gibson, J. J. (1979). The ecological approach to visual perception (Houghton Mifflin, Boston). Grüsser, O. J. & Grüsser-Cornehls, U. (1976). Neurophysiology of the anuran visual system. In R. Llinas & W. Precht (eds.), Frog neurobiology. (pp. 297-385). New York, NY: Plenum Press. Hatsopoulos, N., Gabbiani, F. & Laurent, G. (1995) Elementary computation of object approach by a wide-field visual neuron. Science 270, 1000-1003. Hayes, W. N. & Saiff, E. I. (1967). Visual alarm reactions in turtles. Animal Behaviour, 15, 102108. Hodos, W. (1969). Color discrimination deficits after lesions of the nucleus rotundus in pigeons. Brain, Behavior and Evolution, 2, 185-200. Hodos, W. and Bonbright, J. C. (1974). Intensity difference thresholds in pigeons after lesions of the tectofugal and thalamofugal visual pathways. Journal of Comparative and Physiological Psychology, 87, 1013-1031. Hodos, W. & Karten, H. J. (1966). Brightness and pattern discrimination deficits in the pigeon after lesions of nucleus rotundus. Experimental Brain Research, 2, 151-167. Ingle, D. J. & Shook, B. L. (1983). Action-oriented approaches to visuo-spatial brain functions. In D. Ingle, M. Jeannerod & D. Lee (Eds.). Brain mechanisms of spatial vision. (pp. 229258). Dordrecht:Matinus Nijhoft. Jarvis, C. D. (1974). Visual discrimination and spatial location deficits after lesions of the tectofugal pathway in pigeons. Brain, Behaviour and Evolution, 9, 195-228. Judge, S. & Rind, F. C. (1997). The Locust DCMD, a movement-detecting neurone tightly tuned to collision trajectories. Journal of Experimental Biology, 200, 2209-2216. Karanka-Ahonen, J. T., Luque-Ruiz, D. & López-Zamora, M. (2002). A neural network that acquires the capacity of predicting collision from the increase in the retinal size of objects. XIV Congreso de la Sociedad Espaňola de Psicologia Comparada. Keay, K. A., Redgrave, P. & Dean, P. (1988). Cardiovascular and respiratory changes elicited by stimulation of rat superior colliculus. Brain Research Bulletin, 20, 13-26. Lee, D. N. (1976). A theory of visual control of braking based on information about time-tocollision. Perception 5, 437-459. Lee, D. N., Davies, M. N. O., Green, P. R. & Van der Weel, F. R. (1993). Visual control of velocity of approach by pigeon when landing. Journal of Experimental Biology, 180, 85-104. Lee, D. N. & Reddish, P. E. (1981). Plummeting gannets: a paradigm of ecological optics. Nature, 293, 293-294. Northmore, D. P. M., Levine, E. S. & Schneider, G. E. (1988). Behaviour evoked by electrical stimulation of the hamster superior colliculus. Experimental Brain Research, 73, 595605. 36 Barrie J. Frost and Hongjin Sun Olberg, R. M., Worthington, A. H. & Venator, K. R. (2000). Prey pursuit and interception in dragonflies. Journal of Comparative Physiology A, 186, 155-162. Pearson, K. G., Heitler, W. J. & Steeves, J. D. (1980). Triggering of locust jump by mutimodal inhibitory interneurons. Journal of Neurophysiology, 43, 257-278. Peña, J. L. & Konishi, M. (2001). Auditory spatial receptive field s created by multiplication. Science, 292, 249-252. Redgrave, P., Dean, P., Souki, W. & Lewis, G. (1981). Gnawing and changes in reactivity produced by microinjections of picrotoxin into the superior colliculus of rats. Psychopharmacology, 75, 198-203. Rind, F. C. (1997). Collision avoidance: from the locust eye to a seeing machine. In M.V. Srinivasan & S. Venkatesh (Eds), From Living Eyes to Seeing Machines (pp 105-125). Oxford:Oxford University Press. Rind, F. C. & Bramwell, D. I. (1996). Neural network based on the input organization of an identified neuron signaling impeding collision. Journal of Neurophysiology, 75, 967-985. Rind, F. C. & Simmons, P. J. (1992). Orthopteran DCMD neuron: A reevaluation of responses to moving objects. I. Selective responses to approaching objects. Journal of Neurophysiology, 68, 1654-1666. Rind, F. C. & Simmons, P. J. (1999). Seeing what is coming: building collision-sensitive neurones. Trends in Neurosciences, 22, 215-220. Robertson, R. M. & Johnson, A. G. (1993a). Retinal image size triggers obstacle avoidance in flying locusts. Naturwissenschaften, 80, 176-178. Robertson, R. M. & Johnson, A. G. (1993b). Collision avoidance of flying locusts: Steering torques and behaviour. Journal of Experimental Biology, 183, 35-60. Sahibzada, N., Dean, P. & Redgrave, P. (1986). Movements resembling orientation or avoidance elicited by electrical stimulation of superior colliculus in rats. Journal of Neuroscience, 6, 723-733. Schiff, W. (1965). Perception of impeding collision: A study of visually directed avoidant behaviour. Psychological Monographs: General and Applied, 79, 1-26. Schiff, W., Caviness, J. A. & Gibson, J. J. (1962). Persistent fear responses in rhesus monkeys to the optical stimulus of ‘looming’. Science, 136, 982-983. Schlotterer, G. R. (1977). Response of the locust descending movement detector neuron to rapidly approaching and withdrawing visual stimuli. Canadian Journal of Zoology, 55, 13721376. Simmons, P. (1980). Connexions between a movement-detecting interneurone and flight motoneurones of a locust. Journal of Experimental Biology, 86, 87-97. Simmons, P. J. & Rind, F. C. (1992). Orthopteran DCMD neuron: A reevaluation of responses to moving objects. II. Critical cues for detecting approaching objects. Journal of Neurophysiology, 68, 1667-1682. Sun, H.-J., Carey, D. P. & Goodale, M. A. (1992). A mammalian model of optic-flow utilization in the control of locomotion. Experimental Brain Research, 91, 171-175. Sun, H.-J. & Frost, B. J. (1997). The effect of image expansion on a human target-directed locomotion task tested in virtual reality. Society for Neuroscience Abstracts,23 The Biological Bases of Time-to-collision Computation 37 Sun, H.-J. & Frost, B. J. (1998). Computation of different optical variables of looming objects in pigeon nucleus rotundus neurons. Nature Neuroscience, 1, 296-263. Sun, H.-J., Zhao, J., Southall, T. L., & Xu, B. (2002). Contextual influences on the directional responses of tectal cells in pigeons. Visual Neuroscience, 19, 133-144. Tronick, E. (1967). Approach response of domestic chicks to an optical display. Journal of Comparative and Physiological Psychology, 64, 529-531. Wang, Y. & Frost, B. J. (1992) Time to collision is signalled by neurons in the nucleus rotundus of pigeons. Nature, 356, 236-238. Wagner, H. (1982). Flow-field variables trigger landing in flies. Nature, 297, 147-148. Wicklein, M. & Strausfeld, N. J. (2000). Organization and significance of neurons that detect change of visual depth in the hawk moth Manduca sexta. Journal of Comparative Neurology, 424, 356-376. Yonas, A. & Granrud, C. (1985). The development of sensitivity to kinetic binocular and pictorial depth information in human infants. In D. Ingle, M. Jeannerod, & D. Lee (Eds.), Brain Mechanisms and Spatial Vision (pp. 113-145). Amsterdam: Martinus Nijhoff Press. 38 Barrie J. Frost and Hongjin Sun SUBJECT INDEX Action.......................................................36 Auditory ...................................................37 motion 14, 16, 17, 19, 20, 25, 26, 30, 31, 32, 33, 35, 36 tau 22, 23, 25, 26, 27, 28, 29, 30, 32, 33 Flow .........................................................38 optic flow ............................................17 Information...............................................14 Looming ...................................................14 Tau ..................................................... 18, 27 local ........................................ 30, 31, 32 Time to collision ......................................38 trajectory ............................................ 19, 24 Velocity....................................................26