Perhaps the most fundamental question in Visual-Motor Neuroscience is when, where, and how visual signals are transformed into motor signals. We will consider more complex aspects of this in the following sessions, but right now we just want to differentiate between visual and motor signals in the brain. Does this difference occur between different areas of the brain? Between different neurons? Within the same neurons at different times? Approaching the brain from a global view, one starts with the impression that vision is encoded in occipital cortex, movement in frontal cortex, and parietal cortex is involved in the transformation from vision to action. However, things are not that simple. For example, frontal cortex neurons often carry visual signals, and some occipital areas may code the direction of movement rather than the direction of the stimulus when these are dissociated (although this is likely a matter of imagery rather than control). To answer this seemingly basic question, one needs a variety of techniques. Computational models can help us understand how visual signals are transformed into motor signals within artificial networks that are designed to emulate some part(s) of the brain. Studies of patients with damage to specific brain areas can help identify behavioral deficits that are more linked globally to vision vs. action. fMRI can help identify different areas of the brain that show activity correlated to visual direction vs. movement direction. Neurophysiological recordings from neurons can further show whether different cells within the same or different brain areas code different aspects of vision and movement, or both of these within the same cell. This generally needs to be done in association with some behavioral paradigm that dissociates vision from action in time and/or space. A simple way to dissociate vision from action in time is to require a delay (usually in the order of a second for neurophysiology which records neurons in real time, in the order of about 10 seconds in fMRI which measures much slower responses). We visited this ‘memory delay’ paradigm before when we talked about working memory, but here we will focus on the visual and motor signals. Using this paradigm, one often finds cells in the same area (for example in the cortical gaze control system and superior colliculus) that have only visual responses (just after the visual stimulus), only motor responses (starting just before the movement), or both (visuomotor). This allows us to link neural responses to events, and category different neurons, but does not yet tell us what the cells are coding in space. A simple way to dissociate visual and motor direction is the anti-saccade (or anti-reach) paradigm. Again, in this paradigm humans are asked (or monkeys are trained) to saccade toward (pro) or away (anti) from the visual stimulus, based on some kind of cue. In this paradigm, most visuomotor areas show tuning toward the visual stimulus in their visual responses, and tuning toward the movement direction in their motor responses, even in the same visuomotor cells. When combined with fMRI, one sees a similar switch at the level of entire brain areas, although there is a tendency for the visual response to dominate (or sometimes be the only response) in earlier occipital areas and the motor direction response to dominate in later areas, especially primary motor cortex. Parietal areas are intermediate. A variant of the anti-reach task is to train people to make reaches while looking through prisms that reverse everything you see left-to-right. You will even see your hand reversed, but of course the real objects are still in the same place so you have to learn to reach opposite to what you see. We have found that this occurs fairly automatically within about 10 trials if the task is kept simple. It is also action-specific (for example, training on reach direction does not affect grasp orientation, and vice versa). When we did this in conjunction with fMRI, we found that most parietal areas (except AG) continued to code visual direction rather than movement direction. This seems to contradict the antipointing results, but these could be reconciled if we say that parietal cortex is coding the goal of the movement in visual space (which is the direction you see with reversing prisms, but is imagined to be opposite in the ant-reach experiments). However, a more complicated results occurred when people recorded from individual ‘parietal reach’ neurons in monkeys trained on both the prism and anti-reach task. Here, some neurons coded visual goal direction, whereas others coded movement direction. This discrepancy could be because of different species (I think unlikely), or more likely either from differences in what fMRI vs. neurophysiology measures, or from extensive training in monkeys. Finally, another way to dissociate vision from movement is by looking at trials with errors (e.g., saccades that miss the visual stimulus). This has been done in situations where saccades made errors as a function of initial eye position. It has also been done by looking to see if neural activity reflects variable errors in saccades (motor coding), or not (target coding). We have recently exploited this in headunrestrained gaze shifts in a memory delay paradigm while recording from both the superior colliculus and frontal eye fields. In both places, we found that visual responses remained tightly linked to stimulus location whereas motor responses showed variations related to errors in future gaze position. (It appears that the source of these errors accumulates in neurons during the memory delay, and in the final step from memory to the motor response). Moreover, visuomotor cells showed both of these responses, showing again that the same cell can code different things at different times. The overall conclusion of these studies is that the visuomotor transformation is a network phenomenon that can be observed at both the global and very local levels in the brian. Much of occipital-parietalfrontal cortex is active in such transformations, and there is a general transition from vision-tomovement coding along this axis. However, when one looks within these areas, one sees that different cells contribute differently to this process, and that it can even be observed within cells. In the upcoming sessions, we will look more closely at the specific computations that these transformations must perform in order to create action from vision.