* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download No Slide Title
Brain Rules wikipedia , lookup
Neuroplasticity wikipedia , lookup
Broca's area wikipedia , lookup
Premovement neuronal activity wikipedia , lookup
Computer vision wikipedia , lookup
Synaptic gating wikipedia , lookup
Metastability in the brain wikipedia , lookup
Feature detection (nervous system) wikipedia , lookup
Cognitive neuroscience wikipedia , lookup
Holonomic brain theory wikipedia , lookup
Human brain wikipedia , lookup
Neuroanatomy of memory wikipedia , lookup
Neural correlates of consciousness wikipedia , lookup
Aging brain wikipedia , lookup
Neurophilosophy wikipedia , lookup
Neuropsychopharmacology wikipedia , lookup
Nervous system network models wikipedia , lookup
Mirror neuron wikipedia , lookup
Neuroesthetics wikipedia , lookup
Neurolinguistics wikipedia , lookup
Cognitive neuroscience of music wikipedia , lookup
Time perception wikipedia , lookup
Embodied cognitive science wikipedia , lookup
Cognitive and Semantic Forms (CF & SF) Grammaticization: From bag of tricks to systematic syntax Karine Megerdoomian: Unlocking the CF of verbs Unpacking syntactic categories: e.g., mass nouns versus count nouns. From scene components to constituents Heine: in front of/behind PF Language-Specific SF CF “Almost” Language-Independent I will use the term SF for Semantic Form (not San Francisco!) The idea is that this occupies the same place as LF in the approach of many linguists, but emphasizes that Logic is more likely to be a useful descriptive tool rather than a strict match for neural representations. Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 1 Performance Revisited Observations: Each form is distributed across multiple brain regions. Binding of subrepresentations is required both within and across Forms. The Problem of Serial Order: Linking hierarchical constituents to ordered expressive gestures. Hypothesis: Linkages of the Basal Ganglia to multiple levels play a crucial role. Cognitive Structures (Schema Assemblages) CF Semantic Structures (Hierarchical Constituents SF expressing objects, actions and relationships) “Phonological” Structures PF (Ordered Expressive Gestures) Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 2 Extending the Mirror System A subtle issue: We start with a clear distinction between the representation of the grasp (the action/proto-verb) and the raisin (the thing/proto-noun) in the brain, but in the spoken language both noun and verb are uttered by actions. Given this, how are we to maintain the distinction between verb and noun? Review the imaging data in this light. Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 3 A Key to Distinguishing CF from Linguistic Forms Distinguishing the sign from the affordance or the schema Recall: Two Roles for Imitation in the Evolution of Manual-Based Communication 1. Extending imitation from imitation of hand movements by hand movements to pantomime which uses the degrees of freedom of the hand (and arm and body) to imitate degrees of freedom of objects and actions other than hand movements. Distinguishing the neural representation of the action or object per se (CF) from the gesture which represents it (PF) 2. Extending these pantomime movements to to provide ad hoc gestures that may convey to the observer information which is hared to pantomime in an “obvious” manner. This requires extending the mirror system from the grasping repertoire to mediate imitation of gestures to support the transition from ad hoc gestures to conventional signs which can reduce ambiguity and extend semantic range. The beginning of morphology - modifying a gesture to provide shadings of meaning. Such modifications may be ad hoc, yet become more systematic as historical evolution regularizes certain of these constructions. Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 4 Activity of F5 canonical neurons is part of the code for Command: Grasp-A(Object) Object features Object affordance extraction cIPS 7b: PF/PG Object affordance -hand state association Hand shape recognition Hand motion detection STS AIP Integrate temporal association Mirror Action Feedback recognition Hand-Object spatial relation analysis 7a (Mirror Neurons) F5mirror The Mirror Neuron System (MNS) Model F5canonical Motor program (Grasp) Motor program (Reach) F4 Motor execution M1 Object location The full neural representation of the “Cognitive Form” (CF): Grasp-A(Object) requires not only the regions AIP and F5canonical shown in the MNS diagram, but also inferotemporal cortex (IT) which holds the identity of the object. How are these representations bound together? NOTE: This is only the Cognitive Form. There are no “Linguistic Forms” in the monkey. How are these in humans linked to the CF (assumed homologous to monkey’s)? Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 5 Activity of F5 mirror neurons is part of the code for Declarative: Grasp-A(Agent, Object) Object features Object affordance extraction cIPS 7b: PF/PG Object affordance -hand state association Hand shape recognition Hand motion detection STS Mirror Feedback Hand-Object spatial relation analysis 7a AIP Integrate temporal association Action recognition F5mirror The Mirror Neuron System (MNS) Model F5canonical Motor program (Grasp) Motor program (Reach) F4 Motor execution M1 Object location The full neural representation of the “Cognitive Form” (CF): Grasp-A(Agent, Object) requires not only the regions AIP, STS, 7a, 7b and F5miirror shown in the MNS diagram, but also inferotemporal cortex (IT) which holds the identity of the object and regions of STS (?) not included in MNS which hold the identity of the agent. How are these representations bound together? NOTE: This is only the Cognitive Form. There are no “Linguistic Forms” in the monkey. How are these in humans linked to the CF (assumed homologous to monkey’s)? Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 6 Beyond the Mirror to Neurolinguistics Object features cIPS Object affordance extraction 7b: PF/PG Object affordance -hand state association Hand shape recognition Hand motion detection STS AIP Integrate temporal association Mirror Action Feedback recognition Hand-Object spatial relation analysis 7a (Mirror Neurons) F5mirror The Mirror Neuron System (MNS) Model F5canonical Motor program (Grasp) Motor program (Reach) F4 Motor execution M1 Object location If the monkey needs so many brain regions for the mirror system for grasping, how many more brain regions will we need for an account of the language-ready brain that goes beyond the mirror that goes far beyond the F5 Broca’s area homology to develop a full neurolinguistic model linking CF, SF and PF ?? Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 7 Towards a Computational Neurolinguistics Cooperative computation in the brain: to make sense of data relating different brain regions to different aspects of language. Do these data reflect the brain's genetic prespecification and/or the results of the self-organization of the infant brain when the infant develops within a particular language community? Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 8 Cooperative Computation The HEARSAY Paradigm for Speech Understanding Woodja … ? Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 9 HEARSAY II (1976) A Serial Implementation of a Distributed Architecture: Consider how it might relate to the interaction of multiple brain regions Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 10 A Simplistic View of Perceptual Schemas: Constraint Satisfaction Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 11 Cooperative Computation The VISIONS Paradigm for Visual Scene Analysis Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 12 A 20-year-old Overview of Neurolinguistics:Luria (Arbib & Caplan) A B A summary of diagrams developed by Arbib and Caplan (1979) based on Luria's (1973) analyses of Select ive Naming ____ Vis ual Perception ____ Tertiary L. Pariet oOccipital Zone L. Temporo-Occipital Zones D C Switching Control ____ Articulatory Sys tem ____ Inferior Zone of L. Premotor Cortex Inferior Zone of L. Postcentral Cortex Vis ual Input Auditory Input Naming of objects; Verbal expression of motives; Speech understanding; Speech repetition E F" Updating the Plan of the Expres s ion ____ Frontal Lobes F ... H Secondary Zone of L. Temporal Cortex G Plan Formation ____ Frontal Lobes Lexical Analys is ____ Phonemic Analysis ____ Posterior Zone of L. Temporo-Occipital Region I Speech Memory ____ Middle Zones of L. Temporal Region Deep Zones of L. Temporal Lobe Formation of t he Linear Scheme ____ Inferior Zone of L. Fronto-Temporal Cortex F' Active Analys is of Most Significant Elements ____ Frontal Lobes Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language J Logical Scheme ____ L. Parieto-TemporoOccipital Zones 13 Naming of objects A B C Switching Control ____ Selective Naming ____ Visual Perception ____ Tertiary L. ParietoOccipital Zone L. Temporo-Occipital Zones D Articulatory System ____ Visual Input Inferior Zone of L. Premotor Cortex Inferior Zone of L. Postcentral Cortex Arbib and Caplan (1979) based on Luria's (1973) analysis Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 14 Speech repetition E F" Updating the Plan of the Expression ____ Frontal Lobes Phonemic Analysis ... ____ Secondary Zone of L. Temporal Cortex Arbib and Caplan (1979) based on Luria's (1973) analysis Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 15 Speech understanding H Lexical Analysis ____ Posterior Zone of L. Temporo-Occipital Region I Speech Memory ____ Middle Zones of L. Temporal Region Deep Zones of L. Temporal Lobe F' Active Analysis of Most Significant Elements ____ Frontal Lobes J Logical Scheme ____ Arbib and Caplan (1979) based on Luria's (1973) analysis L. Parieto-TemporoOccipital Zones Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 16 Verbal expression of motives F G Plan Formation ____ Frontal Lobes Formation of the Linear Scheme ____ Inferior Zone of L. Fronto-Temporal Cortex Arbib and Caplan (1979) based on Luria's (1973) analysis Arbib and Itti: CS 664 (University of Southern California, Spring 2002) Integrating Vision, Action and Language 17 Arbib and Caplan (1979) B based on Luria's (1973) analysis C D Switching Control Selective Naming A Visual Perception Articulatory System Visual Input Auditory Input E F" Updating the Plan of the Expr’n ... Phonemic Analysis G F F' Plan Formation Analysis of Significant Elements Formation of the Linear Scheme H Lexical Analysis I J Speech Memory Logical Scheme