* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download TalkHumaine_grandjean
Perception of infrasound wikipedia , lookup
Convolutional neural network wikipedia , lookup
Synaptic gating wikipedia , lookup
Premovement neuronal activity wikipedia , lookup
Affective neuroscience wikipedia , lookup
Executive functions wikipedia , lookup
Neural modeling fields wikipedia , lookup
Visual search wikipedia , lookup
Sensory cue wikipedia , lookup
Cognitive neuroscience of music wikipedia , lookup
Nervous system network models wikipedia , lookup
Neural coding wikipedia , lookup
Emotion and memory wikipedia , lookup
Visual selective attention in dementia wikipedia , lookup
Holonomic brain theory wikipedia , lookup
Central pattern generator wikipedia , lookup
Neuroesthetics wikipedia , lookup
Metastability in the brain wikipedia , lookup
Visual extinction wikipedia , lookup
Channelrhodopsin wikipedia , lookup
Emotional lateralization wikipedia , lookup
C1 and P1 (neuroscience) wikipedia , lookup
Evoked potential wikipedia , lookup
Stimulus (physiology) wikipedia , lookup
Time perception wikipedia , lookup
Sensory substitution wikipedia , lookup
Embodied cognitive science wikipedia , lookup
Inferior temporal gyrus wikipedia , lookup
Emotion perception wikipedia , lookup
Multimodal sensory integration: questions and suggestions GERG • Santorini, 18-21 June, 2004 Outline • Models of multimodal integration (MI) • Functions and factors of MI • A cognitive neuroscience approach: The superior colliculus • Recognition of emotion: How many levels of analysis are needed in one modality? The case of visual integration of low spatial frequencies of emotional facial expression. • What about the time in multimodal integration models? • Conclusions GERG • Santorini, 18-21 June, 2004 1 Some models of integration (1) Direct Identification Model (DI) The input signals are directly transmitted to the bimodal classifier (Klatt, 1979). (2) Separated Identification Model (SI) The visual and the auditory input are separately identified through two parallel identification processes (McGurk & McDonald, 1976). GERG • Santorini, 18-21 June, 2004 2 Some models of integration (cont’d) (3) Dominant Modality Recoding Model (RD) A dominant modality drives the perception of other modalities, this dominant modality would be different according to the context or the task (identification or localization). (4) Motor Space Recoding Model (MR) Basics of the model: The two modalities are projected upon a common motor space where they are integrated before final categorization. (4bis) Model of Motor Dynamics GERG • Santorini, 18-21 June, 2004 3 Factors of integration Two major factors are relevant for multimodal integration whether it is spatially (spatial occurrence in event localization, e.g. ventriloquy) or through recognition (McGurk effect). Two functions: Two major processes: RecognitionIdentification (what system) Temporal aspects Localization (where system) GERG • Santorini, 18-21 June, 2004 Spatial aspects 4 Boundaries of multimodal integration Localization: Recognition: Thurlow et al. 1973 - 10° maximum of divergence - 200 ms of asynchrony Illusion of collision: - 40° of divergence - ~ 400 ms of asynchrony -Lewald et al. 2001 - 3° - 100 ms - similar to limits of the McGurk effect (Watanabe, 2001) GERG • Santorini, 18-21 June, 2004 5 The superior colliculus: An example of spatial integration Receptive field: 55% of neurons are a receptive field related to multimodal integration (visual-auditory-somesthesic) in this nucleus: the responses of these neurons are higher when a multimodal stimulation occurs than when there is a unimodal stimulation. GERG • Santorini, 18-21 June, 2004 6 Adapted from Emanuelle Reynaud Propreties of multimodal neurons in the superior colliculus (Stein et al., 1993;1998) The spatial rule: The responses of neurons are higher when multimodal spatial stimuli occur compared to unimodal stimulus or the sum of unimodal stimuli. When the spatial occurrence of stimuli are disparate these neurons do not discharge or show a decrease of spontaneous activity. The temporal rule: Apparently time is less important in the generation of responses of the multimodal neurons than spatial occurrences. The amplitude of the increase of response decreases with the increase of asynchrony. The maximum of responses is related to the overlap of pattern activity through the time (binding problem). The rule of inverse efficacity: When two stimuli are spatially near and temporally synchronized the response of multimodal neurons is superior to the maximum of the unimodal response. The less the response is high for the unimodal stimuli, the higher the multimodal response is (% of gain). These rules are likely to be also useful in the identification of multimodal integrative areas at the cortical level (e.g., the Superior Temporal Sulcus (STS) or the parietal lobe). GERG • Santorini, 18-21 June, 2004 7 Functions of multimodal integration related to emotion and attention The process of multimodal integration could maximize the detection of events and their identification. These two processes are relevant to modulate attentional processes and could thus orient the ressources of organism (or ECAs …) on specific events or objects. People better recognize the stimuli when two or more sensory channels are excited (visual+auditory) and their reaction time decreases with the increase of channels excited (when the stimuli are congruent). Multimodal level (e.g. STS, parietal lobe) Sensory information (e.g. auditory unimodal cortex) GERG • Santorini, 18-21 June, 2004 Sensory information (e.g. visual unimodal cortex) 8 Recognition of emotion In a virtual environment it is necessary to detect the emotional signals at more than one level for a unimodal stimulation, for example: spatial frequency in the visual domain: analyses of low frequencies and high frequencies with two different and partially independent processes (cf Vuilleumier et al., 2003)? Some brain areas (e.g. amygdala) respond differentially to emotional faces at low frequencies than high frequencies. GERG • Santorini, 18-21 June, 2004 9 Recognition of emotion Concept of weighting of channel in relationship to emotional state: • For example disgust is more recognized in the visual channel than the auditory channel; • Fear could be more recognized in the auditory channel than the visual channel (often confused with surprise in the visual channel) • How can we weight the channels before recognizing the emotional states? Different levels of analyses (cf example of low and high frequencies in the visual channel). GERG • Santorini, 18-21 June, 2004 10 Recognition of emotion Integration: which model could help to detect more precisely and more rapidly the emotional state of people? (two simplified examples) Sensory information (e.g. auditory modality) Time 2 Feature extraction Local Recognition Decision - identification Sensory information Feature extraction Sensory information Feature extraction (e.g. visual modality) (e.g. auditive modality) Local Recognition Fusion: Pattern recognition Sensory information (e.g. visual modality Dynamic aspects: temporal channel and analysis (?) weighting from time 1 to time 2 Decision - identification Feature extraction GERG • Santorini, 18-21 June, 2004 Time 2 11 Conclusions - questions • To what extent can the functional analysis of the brain help us to enrich the models of multimodal integration as much in recognition as in production? Collaborations? • At which moment should integration be realized? On several occasions? To what extent should successive integrations mutually influence each other? • Are categorization and emotional labelling relevant to determine the recognition and the production algorithms? Should there not be an attempt to categorize (detect) the cognitive processes underlying the emotional processes (unfolding with time) in the different modalities (cf emotional facial expressions described in the appraisal processes rather than the discrete or dimensional models). • What is the status of time in the multimodal integration process? GERG • Santorini, 18-21 June, 2004 12