* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download The computational and neural basis of voluntary motor control and
Neurocomputational speech processing wikipedia , lookup
Neural engineering wikipedia , lookup
Neuroscience in space wikipedia , lookup
Time perception wikipedia , lookup
Electromyography wikipedia , lookup
Microneurography wikipedia , lookup
Brain–computer interface wikipedia , lookup
Proprioception wikipedia , lookup
Response priming wikipedia , lookup
Metastability in the brain wikipedia , lookup
Development of the nervous system wikipedia , lookup
Feature detection (nervous system) wikipedia , lookup
Neuroplasticity wikipedia , lookup
Cognitive neuroscience of music wikipedia , lookup
Visual servoing wikipedia , lookup
Executive functions wikipedia , lookup
Muscle memory wikipedia , lookup
Central pattern generator wikipedia , lookup
Evoked potential wikipedia , lookup
Embodied language processing wikipedia , lookup
Perceptual control theory wikipedia , lookup
Review The computational and neural basis of voluntary motor control and planning Stephen H. Scott Centre for Neuroscience Studies, Department of Biomedical and Molecular Sciences, Department of Medicine, Queen’s University, Kingston, ON K7L 3N6, Canada Optimal feedback control (OFC) provides a powerful tool to interpret voluntary motor control, highlighting the importance of sensory feedback in the control and planning of movement. Recent studies in the context of OFC have increasingly used mechanical perturbations and visual shifts to probe voluntary control processes. These studies reveal the surprising sophistication of corrective responses, which are goal-directed and exhibit knowledge of the physical properties of the limb and the environment. These complex feedback processes appear to be generated through transcortical feedback pathways. The research reviewed here opens and enhances several lines of discovery, including testing whether feedback corrections share all of the attributes associated with voluntary control, identifying how prediction influences optimal state estimation, and importantly, how these voluntary control processes are generated by the highly distributed circuitry within the brain. Changing views on the use of sensory feedback for voluntary control It is easy to recognize that sensory signals, such as from our eyes, skin or muscles, help us perceive our body and the world around us. It is harder to understand how these same sensory signals help guide and correct our body movements. Sensory feedback is clearly involved when someone accidentally bumps your arm at a cocktail party leading to rapid motor corrections to avoid spilling a drink held in the hand. Less obvious is whether and how sensory feedback is used continuously to adjust small errors when we reach out to grab an object. The importance of sensory feedback for motor control was demonstrated over a century ago in the classic study by Sherrington on the influence of afferent feedback on spinal level reflexes in the cat hindlimb [1]. The development of feedback control theory in the mid 20th century led to its application to biological control [2] and studies in the 1970s began to explore spinal and cortical feedback during voluntary motor actions. However, several observations in the 1980s suggested that feedback may not be very important for voluntary control [3]. In particular, it was noted that transmission delays between the central nervous system and the limb, on the order of 10s of milliseconds, makes traditional servo-control unstable. As a result, voluntary control was commonly assumed to use motor programs, with motor cortex specifying feedforward Corresponding author: Scott, S.H. ([email protected]). commands to the spinal cord. Feedback was relegated to a modest contribution at only the spinal level. Recognition of the importance of sensory feedback for voluntary control is in a renaissance, due to the introduction of optimal feedback control (OFC) as a model of biological control. Here I review the use of OFC to interpret voluntary control and the value of using perturbations (once again) to probe the properties of the voluntary motor system. Finally, I provide a brief sketch of how OFC-like control may be implemented by distributed circuits in the brain. Optimal Feedback Control (OFC) as a theory of voluntary control Optimality principles have been common in motor control with various possible objectives, such as minimizing jerk, torque change, or the influence of noise [4]. In 2002, Todorov and Jordan [5] proposed OFC as a theory of voluntary control (Figure 1a). Delayed sensory feedback that makes servo-control unstable is overcome by using optimal state estimation, a Kalman filter integrating efference copy signals with delayed sensory feedback. The derivation of the optimal control policy (i.e., feedback gains) uses knowledge of the system dynamics, such as properties of the musculoskeletal system, to achieve an optimal balance between behavioural performance and associated motor costs [6]. What is most impressive about these controllers is that they capture many aspects of biological control. In particular, biological movements are highly variable and yet always quite successful. This variability is created by noise (neural, mechanical, and other), a known feature of sensorimotor systems [7]. Importantly, these types of controllers correct errors/noise if they influence the goal of the task and ignore them if they do not interfere with the goal. This selective control of movement leads to characteristic patterns of motor variability during behavior [8–13]. Although other models of motor control exist [14–16], they cannot capture this behavioural-level feature of voluntary control. One criticism of OFC as a model for biological control is that it cannot be disproved; observed behaviour can be predicted by modifying the cost function (this is true to a degree). There are studies that highlight how humans do not appear to behave optimally [17–19]. However, OFC is best viewed as a normative model that identifies the best possible solution for a given control problem (like Bayesian inference for decisional processes). OFC crystalizes what good control ought to look like, generating a wealth of 1364-6613/$ – see front matter ß 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.tics.2012.09.008 Trends in Cognitive Sciences, November 2012, Vol. 16, No. 11 541 Review [(Figure_1)TD$IG] Trends in Cognitive Sciences November 2012, Vol. 16, No. 11 (a) Opmal feedback control policy Task selecon Motor commands Efference copy Opmal state esmaon Sensory feedback (b) Controlled plant Basal ganglia Spinal cord PF PP PP dPM vPM SMA A5 dPM vPM SMA A5 S1 A5 S1 dPM M1 Cerebellum TRENDS in Cognitive Sciences Figure 1. Optimal Feedback Control and neural implementation. (a) Optimal Feedback Control Policy converts state variables to motor commands. Optimal State Estimation uses efference copy of motor commands and sensory feedback to estimate state variables. Task Selection specifies the goal of the behaviour which then defines the corresponding control policy. (b) Neural implementation of OFC-like voluntary control. The controlled plant is assumed to include the limb and spinal cord. The three basic processes, task selection, control policy, and state estimation, are each generated by highly distributed circuits. Dashed boxes denote individual cortical regions and some cortical regions are assumed to participate in two or even all three processes. Cortical connections are assumed amongst cortical regions involved in a given process. PF, PP and S1 include multiple subdivisions not displayed for simplicity. Cortical abbreviations: PF, prefrontal; PP, posterior parietal; dPM, dorsal premotor; vPM, ventral premotor; SMA, supplementary motor area; A5, area 5; S1, primary somatosensory; M1, primary motor. Images of spinal cord and limb adapted from [21]. experimental predictions on the general properties that biological controllers could or should possess. In fact, observed deviations from ‘optimal behaviour’ provide extremely useful insight into the tricks and tradeoffs inherent in biological control. The mathematical tools used to solve optimal control problems are limited largely to linear control problems with some extensions for nonlinear systems [8]. Clearly, the brain does not use the formalisms of OFC to solve control problems. This is often used as a criticism of OFC [20], although these alternatives inevitably have the same problem. The assumption is that genetics and learning provide good solutions that are OFC-like. Perturbations as a probe of voluntary control Because feedback is an essential feature of OFC, a key approach to probe control is through perturbations, either visual or mechanical. The use of mechanical perturbations is not new – this approach has been used for many decades to observe muscle stretch responses (Box 1). What OFC clarifies is that control is highly specific to the ongoing behavioural goal. Perturbations do not just elicit stereotypical motor reflexes, but also voluntary control processes 542 directly [21]. Thus, perturbations provide an important window into voluntary control. Recent studies highlight several important features of these rapid corrective responses (see [22], for a detailed review). Behavioural goal One key feature of OFC is that errors are only corrected if they affect the behavioural goal and are ignored if they do not. This principle has been shown in a recent study that compared corrective responses when subjects reached to a circular target versus a rectangular bar (Figure 2, [23]). Corrective responses were directed back to the circular target, whereas responses for the rectangular bar were redirected to a new location along the bar. Subjects took advantage of the rectangular shape by making smaller corrective responses closer to the perturbed position of the hand. Differences in electromyographic (EMG) activity for corrective responses to the circular and rectangular targets were observed in as little as 70 ms. Further, obstacles in the environment altered these corrective responses. Long latency responses can also be altered to maximize overall success [24]. Blocks of predominantly resistive perturbations resulted in an increase in perturbation Review Trends in Cognitive Sciences November 2012, Vol. 16, No. 11 Box 1. Electromyographic responses to a perturbation A mechanical perturbation applied to a limb (or joint) elicits activity in muscle afferents (and likely some cutaneous afferents), generating a complex pattern of electromyographic (EMG) activity in muscles that resist the perturbation (Figure I; for a review, see [22]). This pattern of activity is generally divided into a number of phases. The earliest response occurs from 20 to 40 ms (varies somewhat dependent on distance of muscle from spinal cord), is called the short latency reflex, and is generated entirely by the spinal cord. Activity after 100ms has traditionally been viewed as ‘voluntary’, because movement-related EMG can be generated at this time from visual or somatosensory stimuli [40]. The time from 50 to 100 ms is termed the long latency response and involves both spinal and supraspinal feedback. Stretch responses to mechanical perturbations provide a simple paradigm to study feedback corrections during voluntary control. The location of muscle spindles (in muscles!) creates an inherent link between their activity and motor errors, whereas other sensory systems are only conditionally involved (which is also interesting for this very reason). The large diameter of primary muscle-spindle afferents makes transmission to the spinal cord relatively fast and suggests that the motor system has put a premium on their role in rapid feedback processing. There are several factors that need to be considered when exploring corrective responses: (i) OFC highlights how feedback depends on the behavioural goal, thus it is important to define a clear behavioural goal to the subject, both before and after the perturbation. (ii) The task-dependent flexibility of stretch responses impacts the types of perturbations one should apply. Motors are commonly used to apply perturbations but can be used in two different modes. In force control, the motor specifies a force and subjects can counter these perturbations in order to continue to perform a behavioural task. In motion control, the motor specifies a position (or velocity), making it impossible to attain a behavioural goal. Subjects may attempt to compensate, but this is difficult to measure experimentally. (iii) The stretch response can be quickly quenched 30 ms following the offset of a perturbation [69]. Thus, step perturbations that remain on to the end of the trial allow the full multi-phasic EMG response to be observed. (iv) Perturbation magnitude is a balance between large enough to elicit clear EMG responses, but small enough so that subjects can continue to perform the behavioural goal. This magnitude effect may explain why homonymous short-latency responses were observed in [44], whereas heteronymous responses were found for large perturbations in [31]. responses in order to maintain task success as compared to when such perturbations were rare. Shifts in visual feedback (visual perturbations) provide a parallel approach to study the use of vision for control. The influence of target shape was highlighted in a recent study that used visual shifts in hand position during reaching [25]. Corrections for rectangular targets oriented parallel to the movement path elicited larger corrective responses than corrections for targets oriented perpendicular to the movement path. Subjects can also ignore visual shifts in hand feedback, if required. Franklin and Wolpert [9] found that subjects made larger corrective responses when visual shifts remained to the end of the trial, whereas smaller corrective responses were elicited if the visual shift was only transiently present. Everyday activities require us to use each limb independently (e.g., use a spoon to eat cereal) or to coordinate the limbs together (e.g., carry a large bowl). As neural circuitry for controlling each limb is largely confined to separate halves of the brain, bimanual motor tasks provide an interesting model to study task-dependent feedback [26]. (v) Background loads prior to the perturbation allow excitatory and inhibitory responses to be observed. Short latency responses are also difficult to observe without some background muscle activity. [(Box_1)TD$FIG] Applied step-torque perturbaon 0 50 100 Joint moon 0 100 50 EMG of stretched muscle Spinal propriocepve feedback Corcal propriocepve feedback Visual feedback Perturbaon Short onset latency epoch Voluntary epoch Long latency epoch R3 R1 R2 epoch epoch epoch 0 100 50 Time (ms) TRENDS in Cognitive Sciences Figure I. Mechanical perturbation applied to a joint causes joint motion and a multiphasic electromyographic response in stretched muscles. When the two limbs are used in a bimanual motor task, perturbations applied to one limb result in corrective responses in the other limb [27–29]. Impressively, EMG changes to the perturbation can be observed in less than 100 ms for these inter-limb corrections (Figure 3, [29–31]. The presence of goal-directed corrections touches on an age-old debate on whether there is a desired trajectory during tasks such as reaching. Alternative models to OFC commonly assume that the motor system computes a desired trajectory [20]. If the goal is to put the hand at a spatial goal, OFC suggests that there is no need for a desired trajectory. Although there appears to be some importance placed on initiating relatively straight movements [18], even in congenitally blind subjects [32], corrective responses do not return to a desired trajectory. Movements performed in a curl force field that tends to push the hand sideways result in over-compensation early in the movement [33]. Why should the brain produce a greater force than is necessary to overcome a perturbation at the start of a reaching movement? Optimal control predicts this pattern as early 543 Review [(Figure_2)TD$IG] Trends in Cognitive Sciences November 2012, Vol. 16, No. 11 Hand paths (a) y x 2 cm (b) (c) X posion 0.35 Brachioradialis 15 R1 R2 R3 0.3 EMG (au) Posion (m) 0.25 0.2 0.15 0.1 10 5 0.05 0 −0.05 0 200 400 600 800 1000 0 -50 0 50 100 150 Time (ms) Time (ms) TRENDS in Cognitive Sciences Figure 2. Corrective responses to mechanical perturbations applied when reaching to a circle or rectangular bar. (a) Black lines are unperturbed reaching movements to each target. Display of rectangular bar in diagram is clipped on the left, because unperturbed reaches were made roughly to the centre of the bar. Extensor perturbations were applied to each joint just after movement onset. Corrective responses were directed back to the circular goal (blue lines), whereas corrective responses were redirected along the bar near the perturbed hand (red lines). (b), (c) Differences in hand corrections between target shapes can be observed in just over 200 ms, with EMG differences in an elbow flexor observable in 60 ms post-perturbation. Vertical lines denote perturbation onset and dashed lines denote separation of stretch response epochs. Adapted from [23]. compensation is advantageous for producing movements with minimum effort [23,33]. One could say that there is a nominal trajectory that is generated by the voluntary motor system if there were no noise or errors. However, any noise or error generates slight deviations from the nominal trajectory that leads to goal-directed corrections, creating a unique trajectory on each trial. In contrast, if the goal is to draw the letter h, [(Figure_3)TD$IG]then the trajectory is the goal. Each individual has a (a) 0 Task selection The studies described above required subjects to counter a perturbation and continue to perform a motor task. However, many studies ask subjects to generate another task after the perturbation, such as giving instructions to Triceps longus Brachioradialis 50 100 0 150 Key: (b) 0 unique way of writing the letters of the alphabet and this transfers into related control policies to generate that shape whether drawing using the hand, whole arm, or foot. 50 100 150 Contralateral flexion Contralateral extension 101 50 100 96 150 Time from perturbaon onset (ms) 0 50 100 150 Time from perturbaon onset (ms) TRENDS in Cognitive Sciences Figure 3. Inter-limb corrective responses elicited by perturbations to a single limb. Schematics show EMG recorded from the unperturbed right arm during perturbations of the left arm. (a) Group averages of normalized EMG in the uncoupled control task when the contralateral arm was perturbed either downward (blue traces) or upward (orange trace). No statistical change in the either muscle was observed in this task. (b) Response in the coupled single-object task when the two hands controlled a single bar and the subject was instructed to maintain it in a spatial target that was parallel to the body. Red vertical line indicates the earliest time at which differences in EMG were observed between the two conditions. Adapted from [29]. 544 Review resist/let go after the perturbation. In effect, the perturbation elicits a change in the control policy, from maintaining a fixed posture to generating a rapid movement to oppose the load for ‘resist’ or relaxation for ‘let go’. In order to remove the ambiguity of the behavioural goal inherent in these verbal instructions, Pruszynski et al. [34] introduced a spatial version of this task in which a perturbation pushed the hand either into or away from a spatial goal. Subjects can easily launch goal-directed motor actions throughout the workspace with corresponding EMG responses present in 60 ms. Such motor responses are not entirely preplanned (triggered), given that the response scales with the size of the perturbation [35]. Motor reflexes, such as the stumble corrective response or flexion/crossed-extension reflexes, are triggered responses that interrupt an ongoing motor pattern, such as walking [36]. These corrective responses are only observed for large perturbations or stimuli. Perturbations in the studies described above were all relatively large in order to generate clear EMG responses. Although subjects were still able to attain the behavioural goal, the question is whether these corrective responses are distinct from voluntary control, like the stumble corrective response. This issue was addressed by observing EMG responses to multi-joint perturbations of varying magnitude [37]. Significant muscle responses were observed, even when perturbations generated extremely small changes in angular motion that approached the natural range of variability observed during unperturbed trials. The use of linear regressions found that the scaling between perturbation magnitude and EMG activity passed through the origin. This suggests that the perturbations used in most studies cause corrective responses that reflect feedback processing that is also present during unperturbed voluntary motor actions. The ability of mechanical perturbations to launch impending motor plans provides an interesting way to probe decisional processes. For example, a recent study examined stretch responses as subjects judged the direction of motion in a random dot motion display [38]. The authors probed the state of the motor system by perturbing the arm at random times during decision formation and found that long-latency responses were modulated by the strength and duration of the motion pattern prior to their decision to respond to the stimuli. Long-latency responses can also be quickly updated to new spatial goals [39,40]. For example, long latency responses can be modified in as little as 100 ms following the presentation of a spatial goal [40]. Interestingly, responses to mechanical perturbations applied 100 ms after the shift in the target during reaching also result in a corresponding change in the corrective response, including modulation of short latency spinal responses [39]. Thus, shifts in the behavioural goal lead to a shift in movement direction and corresponding changes in feedback corrections, both now directed towards the new spatial goal. These studies highlight the intimate link between decisional processes and motor control [41,42]. Limb and environment The physics of limb movement are complex as the activity of a muscle spanning a joint can lead to motion at multiple Trends in Cognitive Sciences November 2012, Vol. 16, No. 11 joints. The voluntary motor system considers these interactions [43], but what about feedback corrections? This question was addressed by applying loads to the shoulder and elbow, yielding particular motion patterns at these two joints [44]. In one experiment, pure shoulder torque or pure elbow torque perturbations were applied, resulting in the same shoulder motion but different elbow motion. Perturbation-related activity of mono-articular shoulder muscles was examined to see whether they reflected the shoulder motion or the underlying shoulder torque. The short-latency response was the same in both conditions, indicating that it responded only to local joint motion. In contrast, the long-latency response considered the motion at both joints, generating a larger response for the shoulder torque perturbation than the elbow torque perturbation. In a second experiment, combined shoulder and elbow torque perturbations were applied generating no motion at the shoulder and either flexion or extension motion at the elbow (Figure 4a). If the stretch response of shoulder muscles account for the limb’s mechanical properties, they should still respond to this perturbation, even though the joint is not moving. Again, short-latency response of the monoarticular shoulder extensor muscle (posterior deltoids) mirrored the local joint motion. Importantly, there was a robust long-latency response that appropriately countered the underlying torque perturbation (Figure 4b). Knowledge of the physics of the limb is also used for online corrections during reaching for mechanical perturbations [45], visual jumps in target location [46], and even motor planning [47]. The physical environment alters the forces necessary to move our bodies, whether it is an object we are holding or the viscous forces generated when moving underwater. There is a large literature demonstrating that the voluntary motor system can quickly adapt to changes in the visual or mechanical environment [4,48]. Evidence is now mounting that corrective responses are also altered during these adaptations. Burdet et al. [49] and others [50,51] imposed a diverging force field on reaching movements and found that, with practice, corrective responses during the reach increased along an axis that was parallel to the diverging field. When people reached in a curl field in which the forces pushed the hand in only one direction, perturbation responses increased and reflected the properties of the load [52–55]. If a perturbing force is present only for a portion of the movement, long latency responses are modified even before the hand enters that region [56,57]. Adaptation to novel loads result in changes in long latency responses prior to movement during the preparatory period [58]. If the force field perturbs the hand, but this perturbation is irrelevant to task success (i.e., the cursor moves to the target in any case), corrective responses are decreased during the reach [59]. Finally, visual feedback corrections are also modified when adapting to novel mechanical loads [60]. Neural implementation Spinal and transcortical feedback As a theory to describe behaviour, OFC is agnostic as to how voluntary motor control is generated by the spinal cord and brain. The spinal cord provides the first level of 545 Review [(Figure_4)TD$IG] Trends in Cognitive Sciences November 2012, Vol. 16, No. 11 (c) (a) te 80 te Excitatory shoulder torque ts 40 ts No shoulder Moon Inhibitory shoulder Torque 0 x091007a Perturbaon onset (d) (b) 60 Perturbaon onset Post. deltoid EMG R1 R2/R3 1au Firing rate (Hz) 50 40 Responding to underlying torque 30 20 10 0 50ms ∼50 100 Time (ms) 200 TRENDS in Cognitive Sciences Figure 4. Long-latency response accounts for limb dynamics. (a) Subjects were instructed to maintain their hand at a central target and step-torque perturbations were randomly applied to both the shoulder and elbow (flexor torques at both joints, red, or extensor torques at both joints, blue). Shoulder and elbow torque magnitudes and arm geometry were chosen so that they caused substantial elbow motion but minimal shoulder motion. (b) Muscle activity aligned on perturbation onset. Note that even though the shoulder muscle was neither stretched nor shortened by the mechanical perturbation, there is still a robust long-latency response (both excitatory and inhibitory) in the shoulder extensor muscle. Adapted from [44]. (c), (d) Response of an exemplar shoulder-like M1 neuron (c) and the population of shoulder-like neurons (d) to a mechanical perturbation that causes pure elbow motion. Same color scheme as in (a). Note that the shoulder-like neurons respond to the underlying shoulder torque, even though the local motion information from the shoulder is ambiguous. Adapted from [71]. feedback processing. In many species, spinal circuits support sophisticated control, such as scratching, wiping, and basic locomotor patterns [36,61,62]. Analyses of these movements highlight many of the key characteristics predicted by OFC, notably success with variability. The spinal cord also provides phase-dependent feedback in humans during cyclical tasks such as locomotion and even hand cycling [63,64]. Recent work also highlights the importance of spinal circuits for transmitting descending commands during voluntary motor actions [65], although the specific computational processes provided at this level remains unknown. The short-latency stretch response is generated only by spinal feedback. Its size is relatively small and can only be modulated over days or weeks with training [66]. Shortlatency spinal responses are modified by background load due to the effects of motoneuron recruitment, but this is not necessarily a beneficial quality [67]. Sensory gating also occurs at the spinal cord prior to and during movement [68]. Stretch responses can be quickly halted 30 ms after perturbations are removed, demonstrating a powerful inhibitory capability of the spinal cord to quench stretch responses [69]. Although spinal feedback certainly contributes to longlatency responses, a transcortical feedback pathway starts to contribute at this time. Several studies have demonstrated that TMS used on primary motor cortex influences 546 feedback corrections during the long-latency time period [54,56,70,71]. In non-human primates, many studies in the 1970s and early 1980s demonstrated how primary motor cortex (M1) neurons respond to mechanical perturbations or passive limb motion (for reviews, see [3,72]). A classic study by Evarts and Tanji [73] examined neural processing in M1 in a variant of the intervene/do not intervene task by training monkeys to respond to a mechanical perturbation by either pulling or pushing the perturbing handle. Early activity in M1 (20-50 ms post-perturbation) reflected the applied perturbation, but was quickly followed by the appropriate motor response at 50 ms. M1 neurons are directionally tuned to mechanical perturbations [74] in the same manner that activity is broadly tuned to various parameters of movement during reaching [75] or load-related activity during posture [76]. The impact of sensory feedback on M1 activity has also recently been demonstrated by comparing how encoding models of M1 activity depend on the ongoing movements of the limb [77]. In this study, an encoding model was generated from M1 activity when the monkey made reaching movements to spatial targets. Subsequent performance of the monkey on the use of the model to move a cursor to spatial targets was best when the monkey was also moving its hand to the spatial targets, but degraded somewhat when the limb remained stationary and was worst when the limb was randomly moved around the workspace. Thus, changes in Review afferent signals influenced ongoing activity in M1. The impact of sensory feedback on M1 de-emphasizes ideas of neural coding and emphasizes neural dynamics when interpreting M1 function [78]. A recent study examined whether the transcortical pathway through M1 considers the mechanical properties of the limb [71] using the paradigm developed by Kurtzer and colleagues [44]. Unlike muscles that have clear anatomical actions, the study identified shoulder and elbowrelated M1 neurons based on their response to steady-state motor outputs as the monkey countered various combinations of shoulder and elbow loads [74]. The earliest response, 20 to 50 ms after perturbation onset, did not distinguish between the various loading conditions, similar to the default response observed in Evarts and Tanji [73]. However, at 50ms the shoulder-related neurons rapidly increased their firing for the appropriate load at the shoulder, approximately 20 ms before shoulder muscles appropriately responded to the applied load (Figure 4c and d). Thus, in M1 (or elsewhere in this pathway), elbow and shoulder motion are integrated to identify the underlying muscular torque that will counter the applied load. Some models of voluntary control focus attention on spinal feedback [14,15]. However, the observation that virtually all sophisticated stretch responses occur during the long- and not the short-latency time period suggests that supraspinal feedback must be important for voluntary control. A shift to higher motor centres may have occurred as the complexity and breadth of motor skills increased through evolution. Spinal circuits may provide good control, but only for a limited range of behaviours, such as quadrupedal locomotion, scratching, and other stereotyped behaviours. As animals, notably primates, began to increase their breadth of motor skills, such as goal-directed reaching for objects in the environment and social grooming, and increased the use of vision for control, the voluntary motor system may have required a more general, highly-flexible approach to control, increasing the size and importance of corticocerebellar circuits. One reason for this shift to supraspinal control may reflect the importance of prediction in optimal state estimation, because the use of efference copies for control requires knowledge of the properties of the limb, contact forces, and prior experience (but see, [79]). Mapping OFC onto brain circuits OFC-like control is speculated as being generated by cortical and sub-cortical brain circuits [21,80]. Figure 1b provides an overview of the putative contributions of various brain regions to voluntary control. It is at best a sketch that will evolve as experimentation helps to delineate further the contribution of different brain regions to control. The three basic processes, task selection, control policy, and state estimation, are each generated by highly distributed cortical circuits. The basal ganglia participate in task selection through connections with associated cortical regions and the cerebellum participates in optimal state estimation through associated connections with cortex [81–83]. The diagram highlights that many cortical regions support each process and a given cortical region likely participates in two or even all three processes. Interactions Trends in Cognitive Sciences November 2012, Vol. 16, No. 11 between the three processes can occur between connected cortical regions within a given process or within a cortical region that is involved in multiple processes. There are also direct projections from basal ganglia and cerebellum to M1. From the perspective of supraspinal circuits, the spinal cord may be viewed as part of the peripheral motor apparatus modifying the force-length and force-velocity relationships of the limb, simplifying the properties of the musculoskeletal system [84]. More complex spinal processes certainly exist. However, it seems unlikely that all the complexities observed for spinal processing during cyclical behaviours of the hindlimb of non-primates are engaged during voluntary motor actions in the upper limb of humans. Regardless, the prediction is that supraspinal circuits recognize and compensate for spinal complexities much like they recognize and compensate for properties of the musculoskeletal system. The model emphasizes the distributed nature of feedback control much like decisional processes are presumed to involve both frontal and parietal circuits [42,85]. Transcortical feedback for stretch responses has been assumed to reflect a relatively simple pathway of sensory transmission from primary somatosensory cortex to primary motor cortex. However, state estimation is complex, involving the conversion of efferent copy signals into estimates of the present position of the limb, as well as integration of delayed sensory feedback from visual, muscle, and cutaneous afferents. These processes likely involve both cerebellar and parietal cortical regions. A key challenge for research is to understand how different feedback pathways contribute to state estimation during online control. Sensorimotor integration has been examined predominantly during motor planning before the initiation of movement. Does sensory integration observed prior to movement reflect how such processes occur during motor action? Are all signals integrated in a single representation that represents the present state of the system or is each signal processed individually and signals only converge in motor areas such as M1? Present models emphasize a serial organization across cortical regions with descending control principally generated by M1. However, M1 provides only 40% of the axons in the corticospinal tract [86]. Premotor and somatosensory cortex also directly influence spinal processing. Given that the spinal cord is viewed in the present model as part of the plant, one can consider operations such as modifying sensory processing in the spinal cord and modifications of gamma motoneuron activity as essential features of control. We need a broader view of control beyond the generation of EMG patterns. Concluding remarks The use of OFC to interpret voluntary movement has emphasized the importance of sensory feedback for control. Visual and mechanical perturbations during motor behaviour constitute an important scientific approach to probe the sophistication of feedback processing, providing a window into voluntary control processes. Recent studies reveal that corrective responses are highly adaptable based on the behavioural goal and consider the many complexities inherent in multi-joint movements. The basic computational 547 Review Box 2. Questions for future research How is OFC-like control generated without the mathematical formalisms? The brain clearly does not use the mathematical formalisms of OFC and a major challenge is to understand how learning and evolution can generate OFC-like behavior. How is OFC-like control generated by the brain? Figure 1b identifies which brain regions may contribute to a specific process, but not how. State estimation is a complex process involving forward models to estimate the present state from efference copy and delayed sensory signals, and integration across these multiple sources of information. How do the many feedback pathways in the brain generate state estimation? Even a single sensory modality has multiple feedback pathways or processes [87]. What is the role of spinal feedback during voluntary control? Spinal feedback is clearly important in cyclical behaviours, but appears to be more limited during voluntary control. The ability to record spinal neurons in non-human primates during voluntary behaviour provides an important method to address this question [88,89]. Also important to consider is evidence of predictive signals in the firing patterns of muscle spindle afferents during human movement [90], which suggests that this sensory system provides more than just muscle length and velocity feedback. processes in OFC – task selection, control policy and state estimation – provide an interesting way to interpret the contributions of brain regions to voluntary control, guiding future research on the neural basis of voluntary motor action (Box 2). Acknowledgements This work was supported by grants from the National Science and Engineering Research Council of Canada, Canadian Institutes of Health Research (CIHR), and a GlaxoSmithKlein-CIHR chair in Neurosciences. The author would like to thank members of the Limb Lab for helpful feedback on this article. References 1 Sherrington, C.S. (1910) Flexion-reflex of the limb, crossed extensionreflex, and reflex stepping and standing. J. Physiol. 40, 28–121 2 Powers, W.T. (2008) Living Control Systems III: The Fact of Control, Benchmark Publications 3 Scott, S.H. (2008) Inconvenient truths about neural processing in primary motor cortex. J. Physiol. 586, 1217–1224 4 Franklin, D.W. and Wolpert, D.M. (2011) Computational mechanisms of sensorimotor control. Neuron 72, 425–442 5 Todorov, E. and Jordan, M.I. (2002) Optimal feedback control as a theory of motor coordination. Nat. Neurosci. 5, 1226–1235 6 Todorov, E. (2006) Optimal control theory. In Bayesian Brain: Probabilistic Approaches to Neural Coding (Doya, K. et al., eds), pp. 269–298, MIT Press 7 Faisal, A.A. et al. (2008) Noise in the nervous system. Nat. Rev. Neurosci. 9, 292–303 8 Li, W. and Todorov, E. (2007) Iterative linearization methods for approximately optimal control and estimation of non-linear stochastic systems. Int. J. Control 80, 1439–1453 9 Franklin, D.W. and Wolpert, D.M. (2008) Specificity of reflex adaptation for task-relevant variability. J. Neurosci. 28, 14165–14175 10 Nagengast, A.J. et al. (2009) Optimal control predicts human performance on objects with internal degrees of freedom. PLoS Comput. Biol. 5, e1000419 11 Valero-Cuevas, F.J. et al. (2009) Structured variability of muscle activations supports the minimal intervention principle of motor control. J. Neurophysiol. 102, 59–68 12 Stevenson, I.H. et al. (2009) Bayesian integration and non-linear feedback control in a full-body motor task. PLoS Comput. Biol. 5, e1000629 13 Dingwell, J.B. et al. (2010) Do humans optimally exploit redundancy to control step variability in walking? PLoS Comput. Biol. 6, e1000856 548 Trends in Cognitive Sciences November 2012, Vol. 16, No. 11 14 Feldman, A.G. and Levin, M.F. (2009) The equilibrium-point hypothesis–past, present and future. Adv. Exp. Med. Biol. 629, 699– 726 15 Loeb, G.E. (2012) Optimal isn’t good enough. Biol. Cybern. http:// dx.doi.org/10.1007/s00422-012-0514-6 16 Friston, K. (2011) What is optimal about motor control? Neuron 72, 488–498 17 Hudson, T.E. et al. (2010) Compensation for changing motor uncertainty. PLoS Comput. Biol. 4, e1000982 18 Kistemaker, D.A. et al. (2010) The central nervous system does not minimize energy cost in arm movements. J. Neurophysiol. 104, 2985– 2994 19 De Rugy, A. et al. (2012) Muscle coordination is habitual rather than optimal. J. Neurosci. 23, 7384–7391 20 Mohan, V. and Morasso, P. (2011) Passive motion paradigm: an alternative to optimal control. Front. Neurobot. 5, 4 21 Scott, S.H. (2004) Optimal feedback control and the neural basis of volitional motor control. Nat. Rev. Neurosci. 5, 534–546 22 Pruszynski, J.A. and Scott, S.H. (2012) Optimal feedback control and the long-latency stretch response. Exp. Brain Res. 218, 341–359 23 Nashed, J.Y. et al. (2012) Influence of the behavioural goal and environmental obstacles on rapid feedback responses. J. Neurophysiol. 108, 999–1009 24 Cheng, E.J. and Loeb, G.E. (2008) On the use of musculoskeletal models to interpret motor control strategies from performance data. J. Neural Eng. 5, 232–253 25 Knill, D.C. et al. (2011) Flexible, task-dependent use of sensory feedback to control hand movements. J. Neurosci. 31, 1219–1237 26 Diedrichsen, J. et al. (2010) The coordination of movement: optimal feedback control and beyond. Trends Cogn. Sci. 14, 31–39 27 Diedrichsen, J. (2007) Optimal task-dependent changes of bimanual feedback control and adaptation. Curr. Biol. 17, 1675–1679 28 Diedrichsen, J. and Gush, S. (2009) Reversal of bimanual feedback responses with changes in task goal. J. Neurophysiol. 101, 283–288 29 Dimitriou, M. et al. (2012) Task-dependent coordination of rapid bimanual motor responses. J. Neurophysiol. 107, 890–901 30 Mutha, P.K. and Sainburg, R.L. (2009) Shared bimanual tasks elicit bimanual reflexes during movement. J. Neurophysiol. 102, 3142–3155 31 Manning, C.D. et al. (2012) Proprioceptive reaction times and longlatency reflexes in humans. Exp. Brain Res. 221, 155–166 32 Sergio, L.H. and Scott, S.H. (1998) Hand and joint paths during reaching movements with and without vision. Exp. Brain Res. 122, 157–164 33 Izawa, J. et al. (2008) Motor adaptation as a process of reoptimization. J. Neurosci. 28, 2883–2891 34 Pruszynski, J.A. et al. (2008) Rapid motor responses are appropriately tuned to the metrics of a visuospatial task. J. Neurophysiol. 100, 224– 238 35 Pruszynski, J.A. et al. (2011) The long-latency reflex is composed of at least two functionally independent processes. J. Neurophysiol. 106, 449–459 36 Rossignol, S. et al. (2006) Dynamic sensorimotor interactions in locomotion. Physiol. Rev. 86, 89–154 37 Crevecoeur, F. et al. (2012) Fast corrective responses are evoked by perturbations approaching the natural variability of posture and movement tasks. J. Neurosphysiol. 107, 2821–2832 38 Selen, L.P.J. et al. (2012) Deliberation in the motor system: reflex gains track evolving evidence leading to a decision. J. Neurosci. 32, 2276– 2286 39 Mutha, P.K. et al. (2008) Visual modulation of proprioceptive reflexes during movement. Brain Res. 1246, 4–69 40 Yang, L. et al. (2011) Rapid motor responses quickly integrate visuospatial task constraints. Exp. Brain Res. 211, 231–242 41 Wolpert, D.M. and Landy, M.S. (2012) Motor control is decision-making. Curr. Opin. Neurobiol. http://dx.doi.org/10.1016/j.conb.2012.05.003 42 Cisek, P. (2012) Making decisions through a distributed consensus. Curr. Opin. Neurobiol. http://dx.doi.org/10.1016/j.conb.2012.05.007 43 Gribble, P.L. and Ostry, D.J. (1999) Compensation for interaction torques during single- and multijoint limb movements. J. Neurophysiol. 82, 2310–2326 44 Kurtzer, I.L. et al. (2008) Long-latency reflexes of the human arm reflect an internal model of limb dynamics. Curr. Biol. 18, 449–453 Review 45 Kurtzer, I. et al. (2009) Long-latency responses during reaching account for the mechanical interaction between the shoulder and elbow joints. J. Neurophysiol. 102, 3004–3015 46 Gritsenko, V. et al. (2009) Integration of predictive feedforward and sensory feedback signals for online control of visually guided movement. J. Neurophysiol. 102, 914–930 47 Cos, I. et al. (2011) The influence of predicted arm biomechanics on decision making. J. Neurophysiol. 105, 3022–3033 48 Shmeulof, L. and Krakauer, J.W. (2011) Are we ready for a natural history of motor learning? Neuron 72, 469–476 49 Burdet, E. et al. (2001) The central nervous system stabilizes unstable dynamics by learning optimal impedance. Nature 414, 446–449 50 Franklin, D.W. et al. (2007) Endpoint stiffness of the arm is directionally tuned to instability in the environment. J. Neurosci. 27, 7705–7716 51 Wong, J. et al. (2009) The influence of visual perturbations on the neural control of limb stiffness. J. Neurophysiol. 101, 246–257 52 Wang, T. et al. (2001) Learning the dynamics of reaching movements results in the modification of arm impedance and long-latency perturbation responses. Biol. Cybern. 85, 437–448 53 Wagner, M.J. and Smith, M.A. (2008) Shared internal models for feedforward and feedback control. J. Neurosci. 28, 10663–10673 54 Shemmell, J. et al. (2009) The differential role of motor cortex in stretch reflex modulation induced by changes in environmental mechanics and verbal instruction. J. Neurosci. 29, 13255–13263 55 Krutky, M.A. et al. (2010) Interactions between limb and environmental mechanics influence stretch reflex sensitivity in the human arm. J. Neurophysiol. 103, 429–440 56 Kimura, T. et al. (2006) Transcranial magnetic stimulation over sensorimotor cortex disrupts anticipatory reflex gain modulation for skilled action. J. Neurosci. 26, 9272–9281 57 Kimura, T. and Gomi, H. (2009) Temporal development of anticipatory reflex modulation to dynamical interactions during arm movement. J. Neurophysiol. 102, 2220–2231 58 Ahmadi-Pajouh, M.A. et al. (2012) Preparing to reach: selecting an adaptive long-latency feedback controller. J. Neurosci. 32, 9537–9545 59 Scheidt, R.A. et al. (2005) Interaction of visual and proprioceptive feedback during adaptation of human reaching movements. J. Neurophysiol. 93, 3200–3213 60 Franklin, S. et al. (2012) Visuomotor feedback gains upregulate during the learning of novel dynamics. J. Neurophysiol. 108, 467–478 61 Stein, P.S. (2005) Neuronal control of turtle hindlimb motor rhythms. J. Comp. Physiol. A 191, 213–229 62 Frigon, A. (2012) Central pattern generators of the mammalian spinal cord. Neuroscientist 18, 56–69 63 Zehr, E.P. and Duysens, J. (2004) Regulation of arm and leg movement during human locomotion. Neuroscientist 10, 347–361 64 Zehr, E.P. et al. (2012) Neural control of rhythmic arm cycling after stroke. J. Neurophysiol. 108, 891–905 65 Alstermark, B. and Isa, T. (2012) Circuits for skilled reaching and grasping. Annu. Rev. Neurosci. 35, 559–5788 66 Wolpaw, J.R. (2010) What can the spinal cord teach us about learning and memory? Neuroscientist 16, 532–549 67 Pruszynski, J.A. et al. (2009) Temporal evolution of ‘Automatic GainScaling’. J. Neurophysiol. 102, 992–1003 68 Seki, K. and Fetz, E.E. (2012) Gating of sensory input at spinal and cortical levels during preparation and execution of voluntary movement. J. Neurosci. 32, 890–902 Trends in Cognitive Sciences November 2012, Vol. 16, No. 11 69 Kurtzer, I. et al. (2010) Long-latency and voluntary responses to an arm displacement can be rapidly attenuated by perturbation offset. J. Neurophysiol. 103, 3195–3204 70 Perreault, E.J. et al. (2008) Interactions with compliant loads alter stretch reflex gains but not intermuscular coordination. J. Neurophysiol. 99, 2101–2113 71 Pruszynski, J.A. et al. (2011) Primary motor cortex underlies multijoint integration for fast feedback control. Nature 478, 387–390 72 Hatsopoulos, N.G. and Suminski, A.J. (2011) Sensing with the motor cortex. Neuron 72, 477–487 73 Evarts, E.V. and Tanji, J. (1976) Reflex and intended responses in motor cortex pyramidal tract neurons of monkey. J. Neurophysiol. 39, 1069–1080 74 Herter, T.M. et al. (2009) Comparison of neural responses in primary motor cortex to transient and continuous loads during posture. J. Neurophysiol. 101, 150–163 75 Kalaska, J.F. (2009) From intention to action: motor cortex and the control of reaching movements. Adv. Exp. Med. Biol. 629, 139–178 76 Herter, T.M. et al. (2007) Characterization of torque-related activity in primary motor cortex during a multi-joint postural task. J. Neurophysiol. 97, 2887–2899 77 Suminski, A.J. et al. (2010) Incorporating feedback from multiple sensory modalities enhances brainmachine interface control. J. Neurosci. 30, 16777–16787 78 Churchland, M.M. et al. (2012) Neural population dynamics during reaching. Nature 487, 51–56 79 Hantman, A.W. and Jessell, T.M. (2010) Clarke’s column neurons as the focus of a corticospinal corollary circuit. Nat. Neurosci. 13, 1233– 1239 80 Shadmehr, R. and Krakauer, J.W. (2008) A computational neuroanatomy for motor control. Exp. Brain Res. 185, 359–381 81 Strick, P.L. et al. (2009) Cerebellum and nonmotor function. Annu. Rev. Neurosci. 32, 413–434 82 Bostan, A.C. and Strick, P.L. (2010) The cerebellum and basal ganglia are interconnected. Neuropsychol. Rev. 20, 261–270 83 Prevosto, V. et al. (2010) Cerebellar inputs to intraparietal cortex areas LIP and MIP: functional frameworks for adaptive control of eye movements, reaching, and arm/eye/head movement coordination. Cereb. Cortex 20, 214–228 84 Nichols, T.R. and Houk, J.C. (1976) Improvement in linearity and regulation of stiffness that results from actions of stretch reflexes. J. Neurophysiol. 39, 119–142 85 Cisek, P. and Kalaska, J.F. (2010) Neural mechanisms for interacting with a world full of action choices. Annu. Rev. Neurosci. 33, 269–298 86 Porter, R. and Lemon, R.N. (1993) Corticospinal Function and Voluntary Movement, Oxford University Press 87 Gritsenko, V. and Kalaska, J.F. (2010) Rapid online correction is selectively suppressed during movement with a visuomotor transformation. J. Neurophysiol. 104, 3084–3104 88 Takei, T. and Seki, K. (2010) Spinal interneurons facilitate coactivation of hand muscles during a precision grip task in monkeys. J. Neurosci. 30, 17041–17050 89 Shalit, U. et al. (2012) Descending systems translate transient cortical commands into a sustained muscle activation signal. Cereb. Cortex 22, 1904–1914 90 Dimitriou, M. and Edin, B.B. (2010) Human muscle spindles act as forward sensory models. Curr. Biol. 20, 1763–1767 549